cc @tianon @tonistiigi @tiborvass @seemethere @andrewhsu

Codecov Report

Merging #39024 into master will decrease coverage by 0.03%.
The diff coverage is 19.23%.

@@            Coverage Diff             @@
##           master   #39024      +/-   ##
==========================================
- Coverage   37.01%   36.97%   -0.04%     
==========================================
  Files         612      612              
  Lines       45390    45441      +51     
==========================================
+ Hits        16799    16800       +1     
- Misses      26307    26352      +45     
- Partials     2284     2289       +5

I applaud the desire to split these up into more explicit options, however I don't like moving even more rootless logic to a helper script.

Having a mode whereby a user can just specify something like --rootless is extremely nice. I think the issue is just with the auto detection.

I guess this is fine to fix the bug, but would definitely like to see work around the UX for rootless.

Also, wondering if we can reproduce this issue in a test.

I don't understand the issue precisely and why we can't just ignore the $USER when --rootless is not set , but adding DOCKER_HONOR_XDG seems to limit the cases where rootless works without any extra configuration. Also, not sure about the --exec-opt. Normally we would not have options like these if we didn't have rootless mode. And in the future if we can support these with rootless they would become meaningless and a security risk.

A quick workaround for the boot2docker issue #39009 would be to just disable setting--rootless automatically depending on the $USER value, but it won't fix the rootful-docker-in-rootless-docker issue #38702 which only requires --exec-opt native.cgroupdriver=none --exec-opt native.restrict_oom_score_adj=1

@cpuguy83 @tonistiigi WDYT on #39024 (comment) ?

rebased

but it won't fix the rootful-docker-in-rootless-docker issue #38702 which only requires --exec-opt native.cgroupdriver=none --exec-opt native.restrict_oom_score_adj=1

When I tried this in #38817 I still had to set --rootless. I did not invoke dockerd through rootlesskit, so --rootless there meant "I know I'm already in rootless container so reuse parent rootless limits?". So can't we just run tests for these cases on allow them when --rootless has been set as --rootless already implies that user is ok with these limitations.

The --rootless monolith is not extensible for future potential support of --exec-opt native.cgroupdriver=cgroup2-rootless or --exec-opt native.cgroupdriver=pam_cgfs

I'd like to either get this merged or find an alternative way soon, so that we can provide almost stable package before DockerCon.

The --rootless monolith is not extensible for future potential support of --exec-opt native.cgroupdriver=cgroup2-rootless or --exec-opt native.cgroupdriver=pam_cgfs

I think we can discuss it again when these become a reality. If there is a usecase to actually choose between different drivers, exec-opt is probably a right flag. But atm we are just determining reduced featureset so that rootless mode(dockerd inside rootless container) can start.

Note that --exec-opt native.cgroupdriver=<driver> is not a new stuff: https://docs.docker.com/engine/reference/commandline/dockerd/#docker-runtime-execution-options

The native.cgroupdriver option specifies the management of the container’s cgroups. You can only specify cgroupfs or systemd. If you specify systemd and it is not available, the system errors out. If you omit the native.cgroupdriver option, cgroupfs is used.

And it had been very confusing that the cgroup driver value was substantially ignored when the daemon is running in "rootless" mode. So I feel --exec-opt native.cgroupdriver=none is better for clarifying that cgroup stuffs are disabled.

Note that --exec-opt native.cgroupdriver= is not a new stuff:

Yes, I had forgotten that. So shouldn't be an issue if there is a need for a special driver for cgroup2 in the future(although ideally it can still be avoided).

My thinking is that if user is running rootless containers --rootless should be enough and --exec-opt native.cgroupdriver=none should not be needed. And if user is running without any interest in rootless containers there shouldn't be an option for --exec-opt native.cgroupdriver=none. Unless there is a practical case other usecase. Using none has a security impact what users might not realize and is generally a bad idea if your system is cgroup capable.

And if user is running without any interest in rootless containers there shouldn't be an option for --exec-opt native.cgroupdriver=none. Unless there is a practical case other usecase. Using none has a security impact what users might not realize and is generally a bad idea if your system is cgroup capable.

Updated PR to disallow native.cgroupdriver=none when the daemon is not running in userns. (daemon/daemon_unix.go:VerifyCgroupDriver())

WDYT?

My thinking is that if user is running rootless containers --rootless should be enough

As user still need to use dockerd-rootless.sh, this does not seem to improve UX.
In future we may embed dockerd-rootless.sh equivalent to dockerd though.

Also, the behavior of --rootless is likely to be inconsistent and confusing when running dockerd as a userns root in rootless docker (#38702), because XDG must be ignored and rootlesskit-docker-proxy must not be set in #38702 usecase.

It might be good to reserve --rootless flag for embedding dockerd-rootless.sh equivalent to dockerd in future.

I agree with @tonistiigi, this should just be encompassed by --rootless, which would hopefully just get better (require less from docker-rootless.sh) over time.

this should just be encompassed by --rootless, which would hopefully just get better (require less from docker-rootless.sh) over time.

Do you mean you want --rootless to do equivalent of DOCKER_HONOR_XDG=1 (implementation might be hard) and --userland-proxy-path=$(which rootlesskit-docker-proxy) over time?
The problem is that these flags needs to be ignored for dind case: #39024 (comment)

How about combined approach - retore --rootless as the shorthard for DOCKER_HONOR_XDG=1 --exec-opt native.cgroupdriver=none --exec-opt native.restrict_oom_score_adj=1 --userland-proxy --userland-proxy-path=$which rootlesskit-docker-proxy).

For typical use case, user only needs to set --rootless via dockerd-rootless.sh.
When running dockerd as root in rootless Docker, user must not specify --rootless but must specify --exec-opt native.cgroupdriver=none --exec-opt native.restrict_oom_score_adj=1.

Coming from a mostly naive place here... but I would say let's not set --userland-proxy-path with --rootless.

Why must the XDG dir be ignored (with --rootless) in root-in-userns case?

Closing per #39138

Why must the XDG dir be ignored (with --rootless) in root-in-userns case?

Because "rootful"-in-rootless dockerd is expected to behave almost same as rootful dockerd.
So, the API path should be /var/run/docker.sock, not $XDG_RUNTIME_DIR/docker.sock.

But if the user set --rootless why would they expect it to behave the same as rootful?

same as rootful but cgroup disabled