Skip to content

[WIP] docker inspection fs (/dev/docker in a container), for exposing the Swarm task slot#24893

Closed
AkihiroSuda wants to merge 2 commits intomoby:masterfrom
AkihiroSuda:inspectionfs
Closed

[WIP] docker inspection fs (/dev/docker in a container), for exposing the Swarm task slot#24893
AkihiroSuda wants to merge 2 commits intomoby:masterfrom
AkihiroSuda:inspectionfs

Conversation

@AkihiroSuda
Copy link
Member

- What I did

Provides the following files for exposing task information (especially, the task slot) to containers:

  • /dev/docker/container/json
  • /dev/docker/swarm/task/json

/dev/docker is mounted by default for tasks, but not mounted for non-task containers. (you can enable it by docker run --inspectionfs)

For the story so far, please look into #24110 (comment)

ping @stevvooe @vbatts @SvenDowideit

Fix #24110
Update #23843
Update #24113

- How I did it
Used FUSE.

- How to verify it

  • Create a service: docker service create --replicas 3 --name s1 alpine tail -f /dev/null
  • Do cat /dev/docker/container/json in each of task containers
  • Do cat /dev/docker/swarm/task/json in each of task containers
  • Remove the service: docker service rm s1
  • Make sure the FUSE filesystem are unmounted: cat /proc/mounts

Discussion:

TODOs:

  • Hide /dev/docker/swarm if the container is not a swarm task
  • Separete modifications under vendor
  • Fix compilation errors for Windows

Provides the following files for exposing task information to containers:

 - /dev/docker/container/json
 - /dev/docker/swarm/task/json

/dev/docker is mounted by default for tasks, but not mounted for non-task containers.
(you can enable it by `docker run --inspectionfs`)

Signed-off-by: Akihiro Suda <[email protected]>
@stevvooe
Copy link
Contributor

@justincormack
Copy link
Contributor

I don't like use of FUSE. There is no equivalent solution for non Linux OS in general. Nor do I like mounting on /dev.

@SvenDowideit
Copy link
Contributor

/dev/, /sys, or even /docker :)

I really do like the idea that we have a standard place for container introspection - maybe it could be implemented as a volume plugin :)

@AkihiroSuda
Copy link
Member Author

I'm 👍 for removing FUSE and just using a real filesystem #24110 (comment)

I'm not sure we can mount a directory in /sys or /proc, maybe /run is one of options.

I'll update PR when we reach consensus about the design

@thaJeztah
Copy link
Member

maybe it could be implemented as a volume plugin :)

I like that thought :)

@dweomer
Copy link

dweomer commented Jul 22, 2016

Seems like all of this information could be gotten from the docker.sock no? Why not inject a heavily access-controlled-by-default docker.sock into all containers, with CLI switches or some other mechanism for opening that up in parts? Alternatively one could repackage only the parts of the Docker API that are appropriate for a meta-data service a la Digital Ocean ( https://developers.digitalocean.com/documentation/metadata/). This could live it it's own socket (simple filesystem-based discovery) or at a pre-defined link-local IP or on the Docker bridge.

P.S. I apologize if any of this has been previously covered and/or decided against. If that is the case please point me to the relevant discussion(s)!

@AkihiroSuda
Copy link
Member Author

@SvenDowideit @thaJeztah

I once considered making that a volume plugin.
But I think it is worth supporting that as a standard feature rather than as a 3rd party plugin.
Maybe 👍 for built-in volume plugin.

@dweomer

I once considered the socket, but

  • you need to install a rich REST client to containers
  • there is no equivalent of AF_UNIX for Windows, AFAIK (correct me if wrong). It is possible to use pre-defined IP instead, but it doesn't work when --net=none.

@AkihiroSuda
Copy link
Member Author

AkihiroSuda commented Jul 25, 2016

Comparison (correct me if wrong 😄, last update Jul 26, 1:42 UTC)

FS
(bind-mount)
FS
(FUSE)
Env var, with template syntax Socket
(UNIX)
Socket
(TCP)
Dynamic update
(e.g. for lables #21721)
Notification of dynamic update
but inotify has some limitations
MS Windows ✅ (named pipes instead)
Low impact to the stability of the daemon (higher than env var, lower than others)
No extra tool needed in containers

Currently I'm +1 for FS (bind-mount), but not strongly sticking to it.

I'm planning to modify the PR so as to:

I'm still not sure about:

  • should it be mounted under /dev/docker /run/docker? /run/docker-context?
  • should it be a built-in volume plugin? (what about default standard mount point?, what about the volume name?)

@thaJeztah
Copy link
Member

Thanks for starting the work on this @AkihiroSuda - note that most maintainers are still very busy on the 1.12 release, but we should definitely give this attention once we have some time on our hands. Having an introspection API has been on the list of a long time, but still needed more discussion in what form; hopefully this PR (and related issue) gets the discussion going again.

FWIW; we should give proper thought if this should be available by default or as an opt-in (the API can expose information that you may not want to have available in just "any" container). Also, it should be discussed if the API can be "scoped", so that only a limited subset of information is available to a container (think of multi-tenant situations). The last obviously depends on what we want to expose (e.g., "docker events" are currently used by various containers, and currently implemented by bind-mounting /var/run/docker.sock inside a container; we should really have a better alternative for that)

@justincormack
Copy link
Contributor

Windows does have the equivalent of sockets, "named pipes" which is what the Docker now uses on WIndows (it used to always use a tcp socket).

@dweomer
Copy link

dweomer commented Jul 25, 2016

@AkihiroSuda

I'm still not sure about:

  • should it be mounted under /dev/docker?
  • should it be a built-in volume plugin? (what about default mount point?)
  • I like /run/docker/ better than /dev/docker
  • If you do it as a volume plugin isn't the concept of a default mount point moot? Because the user is now enlisting the meta-data/introspection service via volume plugin they will also have to specify where to mount it at, thereby offloading the service-discovery pre-allocation/bootstrap/set-aside to the end-user.

@SvenDowideit
Copy link
Contributor

+1 to built in volume plugin :)

@AkihiroSuda
Copy link
Member Author

I'll try to rework on this as a built-in volume plugin, sorry for long delay

@AkihiroSuda
Copy link
Member Author

Opened a new PR (built-in volume driver): #26331

It looks like this:

$ docker service create --name nginx --replicas 3 --mount type=volume,src=docker_introspection,dst=/foo nginx
$ docker exec -it $SOME_NGINX_CONTAINER bash -c 'for f in $(find /foo -type f);do echo ===$f===; cat $f; done'
===/foo/v1/container/name===
/nginx.2.76rasygemgtwq9akwzmurtuze
===/foo/v1/container/labels===
com.docker.swarm.task.name=nginx.2
com.docker.swarm.node.id=7l4osfysnqpu347o5o88lyjqy
com.docker.swarm.service.id=cbnmi8ecj9buk9375bl342q1s
com.docker.swarm.service.name=nginx
com.docker.swarm.task=
com.docker.swarm.task.id=76rasygemgtwq9akwzmurtuze
===/foo/v1/container/id===
5f1e9ead122451486395c6afe2ce0975dd94ace730496a7c9c6474ec0b8bb672
===/foo/v1/task/name===
===/foo/v1/task/slot===
2
===/foo/v1/task/labels===
===/foo/v1/task/id===
76rasygemgtwq9akwzmurtuze
===/foo/v1/service/name===
nginx
===/foo/v1/service/labels===
===/foo/v1/service/id===
cbnmi8ecj9buk9375bl342q1s
===/foo/v1/node/name===
===/foo/v1/node/labels===
===/foo/v1/node/id===
7l4osfysnqpu347o5o88lyjqy

@AkihiroSuda AkihiroSuda closed this Sep 6, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants