Allow running dockerd as a non-root user (Rootless mode) by AkihiroSuda · Pull Request #38050 · moby/moby

AkihiroSuda · 2018-10-16T08:03:35Z

- What I did

Allow running dockerd in an unprivileged user namespace (rootless mode).
Close #37375

No SETUID/SETCAP binary is required, except newuidmap and newgidmap.

For Kubernetes integration, please refer to https://github.com/rootless-containers/usernetes .

This PR contains two commits, but the first one is same as #38038 (overlayfs in userns for Ubuntu).
I'll rebase this PR when #38038 gets merged. (Updated: #38083 is merged now)

- How I did it

By using user_namespaces(7), mount_namespaces(7), network_namespaces(7), and slirp4netns.

Please refer to docs/rootless.md for the details.

- How to verify it

Make sure /etc/subuid and /etc/subgid contain the entry for you

$ id -u
1001
$ whoami
penguin
$ grep ^$(whoami): /etc/subuid
penguin:231072:65536
$ grep ^$(whoami): /etc/subgid
penguin:231072:65536

Start daemon: dockerd-rootless.sh --experimental
Start client: docker -H unix://$XDG_RUNTIME_DIR/docker.sock run ...

Remarks:

Some distros such as Debian (excluding Ubuntu) and Arch Linux require sudo sh -c "echo 1 > /proc/sys/kernel/unprivileged_userns_clone".
Some distros require sudo modprobe ip_tables iptable_mangle iptable_nat iptable_filter.

Restrictions:

Only vfs graphdriver is supported. However, on Ubuntu and a few distros, overlay2 and overlay are also supported. Starting with Linux 4.18, we will be also able to implement FUSE snapshotters.
Cgroups (including docker top) and AppArmor are disabled at the moment. In future, Cgroups will be optionally available when delegation permission is confi
gured on the host.
Checkpoint is not supported at the moment.
Running rootless dockerd in rootless/rootful dockerd is also possible, but not fully tested.

- Description for the changelog

Allow running dockerd in an unprivileged user namespace (rootless mode)

- A picture of a cute animal (not mandatory but encouraged)

https://en.wikipedia.org/wiki/Little_penguin#/media/File:Eudyptula_minor_Bruny_1.jpg

Screenshot:

penguin0@suda-ws01:~$ id
uid=1002(penguin0) gid=1006(penguin0) groups=1006(penguin0)
penguin0@suda-ws01:~$ ps u
USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
penguin0 122952  0.0  0.0  21484  5156 pts/3    Ss   16:58   0:00 /bin/bash -l
penguin0 123093  0.0  0.0  21484  5200 pts/4    Ss   16:58   0:00 /bin/bash -l
penguin0 123094  0.0  0.0 134792  2860 pts/4    S    16:58   0:00 (sd-pam)
penguin0 123252  0.0  0.0   4628   784 pts/4    S+   16:58   0:00 /bin/sh /usr/local/bin/dockerd-rootless.sh --experimental
penguin0 123253  0.0  0.0 105772  3696 pts/4    Sl+  16:58   0:00 rootlesskit --net=slirp4netns --mtu=65520 --copy-up=/etc --copy-up=/run /usr/local/bin/dockerd-rootless.sh --experimental
penguin0 123257  0.0  0.0 105516  4024 pts/4    Sl+  16:58   0:00 /proc/self/exe --net=slirp4netns --mtu=65520 --copy-up=/etc --copy-up=/run /usr/local/bin/dockerd-rootless.sh --experimental
penguin0 123265  0.0  0.0   2980  1072 pts/4    S+   16:58   0:00 slirp4netns --mtu 65520 123257 tap0
penguin0 123281  0.0  0.0   4628   828 pts/4    S+   16:58   0:00 /bin/sh /usr/local/bin/dockerd-rootless.sh --experimental
penguin0 123283  0.6  0.8 583536 65728 pts/4    Sl+  16:58   0:00 dockerd --experimental
penguin0 125126  0.0  0.0  38372  3688 pts/3    R+   17:00   0:00 ps u
penguin0@suda-ws01:~$ docker -H unix:///run/user/1002/docker.sock run --rm hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

AkihiroSuda · 2018-10-16T08:07:02Z

cc @tonistiigi @tiborvass

codecov · 2018-10-16T19:56:05Z

Codecov Report

❗ No coverage uploaded for pull request base (master@50e63ad). Click here to learn what that means.
The diff coverage is 19.23%.

@@            Coverage Diff            @@
##             master   #38050   +/-   ##
=========================================
  Coverage          ?   36.54%           
=========================================
  Files             ?      610           
  Lines             ?    45368           
  Branches          ?        0           
=========================================
  Hits              ?    16581           
  Misses            ?    26497           
  Partials          ?     2290

sargun · 2018-10-17T06:20:30Z

How can you delegate cgroups? A piece of work prior to this might be supporting cgroup namespace?

AkihiroSuda · 2018-10-17T06:46:08Z

How can you delegate cgroups? A piece of work prior to this might be supporting cgroup namespace?

Cgroups delegation is disabled on this PR and it is likely to be a separate PR in future.

Until we can get full cgroups v2 support in runc (blocked due to lack of freezer and device subsystems, see opencontainers/runc#654), we would need to use pam_cgfs, although it is unlikely to be available on Red Hat distros: containers/podman#1429

thaJeztah

Not too familiar with all the requirements to make this work, but had a quick glance over, and left some comments/suggestions 🤗

AkihiroSuda · 2018-10-19T06:58:52Z

addressed comments

cpuguy83 · 2019-01-17T04:08:52Z

Ah I was looking at libproxy in vpnkit and assumed that was what was being used :(

Well, IF these are tcp conns being proxied there, io.Copy will use splice(2) on Linux instead of a user space copy.

AkihiroSuda · 2019-01-18T17:01:06Z

Updated to use prebuilt djs55/vpnkit binary.

Support for non-amd64 and slirp4netns can be discussed in follow-up PR series.

thaJeztah · 2019-01-18T18:03:42Z

Wonder if we can build from source; there's a Dockerfile in the repo to build vpnkit https://github.com/moby/vpnkit/blob/master/Dockerfile, but not sure we should copy that (perhaps the steps from the Dockerfile could be move into the Makefile? @djs55 - think that would work?

It requires more than 10 minutes...

Any chance to get non-amd64 prebuilt binaries?

@djs55 Is it possible to cross-compile vpnkit for non-amd64 targets?

I think cross-compilation is actively being worked on in OCaml, see https://discuss.ocaml.org/t/ocaml-cross-compiler/1494 . I don't think it works 100% yet :(

I think the only reliable way to build vpnkit for other targets would be to build on those targets. The current vpnkit CI builds binaries for macOS and Windows (for use in Docker Desktop). Which other targets do you need?

Linux for armhf, arm64, s390x, and ppc64le

@AkihiroSuda Btw, we don't have s390x and ppc64le releases for 18.09 anymore.

@tonistiigi @thaJeztah does that mean that we can actually consider to drop s390x and ppc64le CI builds from Moby?

I haven't been part of this discussion, but I don't see why moby would need to drop them from CI as moby is a source project to Docker 18.xx and future 19.xx products; whether Docker has official releases seems unrelated to whether those architectures are still validated in the moby upstream project during CI.

Yes, no need to drop CI but if it becomes troublesome to support them for new features it is a data point to consider. For example, I think if we can't have these new binaries available on these platforms it shouldn't block this PR.

AkihiroSuda · 2019-01-25T20:56:09Z

Alternative plan: can we just remove vpnkit/slirp4netns from make install, and ask users to install it separately?

thaJeztah · 2019-01-26T01:07:34Z

Was about to suggest "dummy" aliases for the djs55/vpnkit image as a buildstage to make it pass on power/z/arm but then realized we're not using BuildKit yet :/

…

On 25 Jan 2019, at 23:56, Tõnis Tiigi ***@***.***> wrote: @tonistiigi commented on this pull request. In Dockerfile: > @@ -233,6 +238,10 @@ RUN cd /docker-py \ && pip install paramiko==2.4.2 \ && pip install yamllint==1.5.0 \ && pip install -r test-requirements.txt +COPY --from=rootlesskit /build/ /usr/local/bin/ +# VPNKit git b4c8b69e68f74c69a6e2fff696a3a196b061dde6 (1/5/2019) +# FIXME: currently, this always install amd64 binary Yes, no need to drop CI but if it becomes troublesome to support them for new features it is a data point to consider. For example, I think if we can't have these new binaries available on these platforms it shouldn't block this PR. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

thaJeztah · 2019-01-26T01:11:12Z

We could still push dummy images to Docker Hub perhaps for these architectures (or conditionally download a tar.gz)? 🤔

…

On 25 Jan 2019, at 23:56, Tõnis Tiigi ***@***.***> wrote: @tonistiigi commented on this pull request. In Dockerfile: > @@ -233,6 +238,10 @@ RUN cd /docker-py \ && pip install paramiko==2.4.2 \ && pip install yamllint==1.5.0 \ && pip install -r test-requirements.txt +COPY --from=rootlesskit /build/ /usr/local/bin/ +# VPNKit git b4c8b69e68f74c69a6e2fff696a3a196b061dde6 (1/5/2019) +# FIXME: currently, this always install amd64 binary Yes, no need to drop CI but if it becomes troublesome to support them for new features it is a data point to consider. For example, I think if we can't have these new binaries available on these platforms it shouldn't block this PR. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

AkihiroSuda · 2019-01-31T13:45:43Z

Updated PR. Now user needs to be install either slirp4netns or vpnkit separately.

We should still include vpnkit in make install bundle, but let's discuss it separately. (And also slirp4netns in some "contrib" binary tgz maybe)

cpuguy83

LGTM

cpuguy83 · 2019-02-01T03:55:24Z

@thaJeztah You good?

thaJeztah

Reviewing from my phone, so just from looking over the changes; left some comments/questions

I'm good to move this forward if those were errors on my side (and this is really cool to see arrive 👌😍🥳)

thaJeztah · 2019-02-03T01:14:31Z

@AkihiroSuda are any packaging changes needed for this?

/cc @seemethere

…ode) Please refer to `docs/rootless.md`. TLDR: * Make sure `/etc/subuid` and `/etc/subgid` contain the entry for you * `dockerd-rootless.sh --experimental` * `docker -H unix://$XDG_RUNTIME_DIR/docker.sock run ...` Signed-off-by: Akihiro Suda <[email protected]>

AkihiroSuda · 2019-02-03T15:26:42Z

Updated PR

cc @estesp @icecrime 😃

@AkihiroSuda are any packaging changes needed for this?

I suggest adding dockerd-rootless.sh, rootlesskit and vpnkit to official RPM/DEB and binary tar archive, but not sure how to build VPNKit for non-amd64.
Also, I suggest providing slirp4netns binary as "contrib" pkg.

thaJeztah · 2019-02-03T18:18:34Z

I suggest adding dockerd-rootless.sh, rootlesskit and vpnkit to official RPM/DEB and binary tar archive, but not sure how to build VPNKit for non-amd64.
Also, I suggest providing slirp4netns binary as "contrib" pkg.

Makes sense; if you have time; could you try opening a pull request in the https://github.com/docker/docker-ce-packaging repository? Perhaps the packaging team can work on it, but if you can prepare a PR, that may help speeding it up 🤗 (feel free to ping me if you need help/input on that one; I'll be on PTO for the next few days, but will try to catch up on notifications)

thaJeztah

LGTM! Thanks; this is really cool stuff 🥳

cyphar · 2019-02-04T03:16:53Z

@AkihiroSuda Great work. 🎉

I suggest adding dockerd-rootless.sh, rootlesskit and vpnkit to official RPM/DEB and binary tar archive, but not sure how to build VPNKit for non-amd64.

Can we use slirp4netns instead of VPNKit (asking for openSUSE when we package this).

thaJeztah · 2019-02-04T05:04:35Z

@cyphar I think the license for slirp4netns was the blocker for bundling it, but you can use it

cyphar · 2019-02-04T05:13:56Z

Right, because we use rootlesskit anyway. Thanks.

tianon · 2019-04-06T00:50:13Z

 	flags.Var(&conf.NetworkConfig.DefaultAddressPools, "default-address-pool", "Default address pools for node specific local networks")
-
+	// Mostly users don't need to set this flag explicitly.
+	flags.BoolVar(&conf.Rootless, "rootless", rootless.RunningWithNonRootUsername(), "Enable rootless mode (experimental)")


For anyone looking to thread the needle, this line appears to be the cause of #39009. 👍 ❤️

AkihiroSuda requested a review from tianon as a code owner October 16, 2018 08:03

GordonTheTurtle added the status/0-triage label Oct 16, 2018

AkihiroSuda mentioned this pull request Oct 16, 2018

[UMBRELLA] patch submission status rootless-containers/usernetes#42

Closed

3 tasks

vdemeester added status/1-design-review and removed status/0-triage labels Oct 16, 2018

vdemeester requested review from cpuguy83, dmcgowan and thaJeztah October 16, 2018 08:06

AkihiroSuda force-pushed the rootless branch 2 times, most recently from 483ab2e to e183cfb Compare October 16, 2018 08:12

vdemeester requested review from crosbymichael, tiborvass and tonistiigi October 16, 2018 08:19

AkihiroSuda force-pushed the rootless branch from e183cfb to 21cc7a8 Compare October 16, 2018 08:26

AkihiroSuda changed the title ~~Rootless mode~~ Allow running dockerd as a non-root user (Rootless mode) Oct 16, 2018

This was referenced Oct 16, 2018

Proposal: allow running dockerd as an unprivileged user (aka rootless mode) #37375

Closed

rootless: make /sys/fs/cgroup/* read-only opencontainers/runc#1869

Closed

AkihiroSuda force-pushed the rootless branch 2 times, most recently from abb3322 to 79c8968 Compare October 16, 2018 17:35

thaJeztah requested changes Oct 17, 2018

View reviewed changes

AkihiroSuda force-pushed the rootless branch from 79c8968 to 76a6d66 Compare October 19, 2018 06:58

AkihiroSuda added rebuild/windowsRS5-process labels Oct 19, 2018

GordonTheTurtle removed the rebuild/experimental label Oct 19, 2018

vdemeester self-requested a review October 22, 2018 08:35

AkihiroSuda mentioned this pull request Jan 17, 2019

How can a DinD solution like this be 'secure'? play-with-docker/play-with-docker#123

Closed

thaJeztah reviewed Jan 18, 2019

View reviewed changes

cpuguy83 approved these changes Feb 1, 2019

View reviewed changes

thaJeztah requested changes Feb 3, 2019

View reviewed changes

Comment thread daemon/config/config_unix.go Outdated

Comment thread cmd/dockerd/daemon.go Outdated

Comment thread daemon/daemon_unix.go Outdated

thaJeztah approved these changes Feb 3, 2019

View reviewed changes

AkihiroSuda mentioned this pull request Feb 3, 2019

unfork moby & containerd rootless-containers/usernetes#84

Merged

seemethere mentioned this pull request Feb 4, 2019

Build fails on armv7 rootless-containers/rootlesskit#41

Closed

tonistiigi mentioned this pull request Feb 6, 2019

hack: restore bundling vpnkit on amd64 #38684

Merged

asm0dey mentioned this pull request Feb 6, 2019

Episode #23 php-coder/ps-podcast#20

Closed

adamnovak mentioned this pull request Feb 9, 2019

Can't pull images from your Dockerfile into the container's Docker when using Docker in a container, because builds can't do priveleged actions and dockerd #38698

Closed

EML-github mentioned this pull request Feb 13, 2019

Merge #38050 causing a failure? #38725

Closed

tianon mentioned this pull request Apr 5, 2019

19.03.0-beta1: could not get XDG_RUNTIME_DIR (dockerd fails to start) #39009

Closed

tianon reviewed Apr 6, 2019

View reviewed changes

AkihiroSuda mentioned this pull request Jun 4, 2019

KEP: New cgroup driver: "none" (for Rootless) kubernetes/enhancements#1084

Closed

AkihiroSuda mentioned this pull request Mar 17, 2020

RFC: add GPL binaries (fuse-overlayfs and slirp4netns) to docker-rootless-extras.tgz #40706

Closed

thaJeztah mentioned this pull request Apr 25, 2022

minor rootless fix-ups #43521

Merged

Conversation

AkihiroSuda commented Oct 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AkihiroSuda commented Oct 16, 2018

Uh oh!

codecov Bot commented Oct 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sargun commented Oct 17, 2018

Uh oh!

AkihiroSuda commented Oct 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AkihiroSuda commented Oct 19, 2018

Uh oh!

cpuguy83 commented Jan 17, 2019

Uh oh!

AkihiroSuda commented Jan 18, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda commented Jan 25, 2019

Uh oh!

thaJeztah commented Jan 26, 2019 via email

Uh oh!

thaJeztah commented Jan 26, 2019 via email

Uh oh!

AkihiroSuda commented Jan 31, 2019

Uh oh!

cpuguy83 left a comment

Choose a reason for hiding this comment

Uh oh!

cpuguy83 commented Feb 1, 2019

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thaJeztah commented Feb 3, 2019

Uh oh!

AkihiroSuda commented Feb 3, 2019

Uh oh!

thaJeztah commented Feb 3, 2019

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda commented Oct 16, 2018 •

edited

Loading

codecov Bot commented Oct 16, 2018 •

edited

Loading

AkihiroSuda commented Oct 17, 2018 •

edited

Loading

cyphar commented Feb 4, 2019 •

edited

Loading