@bergwolf PTAL?

@AkihiroSuda One more question. When a new device is attached to the host and an entry is created in /dev of the host, we should also restrict those new devices from getting created inside the container right?

I can see that already there is a bug in docker which does not update the /dev directory of the container, when new devices are attached. Not sure whether this will impact or not?

For Kata, new device wont be enabled because there is no mount for the device.

For non-kata, the flag should not be considered as a security boundary.

For non-kata, the flag should not be considered as a security boundary.

Ohkay. Got it. But can these same functionality be extended to non-kata? Or is it used only to get past the VM isolation issue in kata.

But can these same functionality be extended to non-kata

It doesn't make sense. Privileged non-kata container can execute arbitrary command on the host anyway to access any device

Ohkay!. 👍 Thanks for explaining.

LGTM! Thanks @AkihiroSuda !

@justincormack @cpuguy83 @thaJeztah PTAL?

cc @tianon WDYT?

The option seems ok, but I'm not sure why someone would use privileged and also expect it to be secure?

@cpuguy83

dind with kata needs --privileged and is expected to be secure

Isn't this combination possible with #36644, or is that tweaking something else? (haven't looked in depth).

Unrelated, this one aims at preventing Kata from mounting host /dev completely

I get conceptually that this solves a problem Kata has, but I don't think I understand why this particular solution was chosen? We've long regarded the --privileged flag as somewhat of a mistake (certainly that in hindsight, could've had a better name), especially given the vast number of users who view it as sudo for containers (ala, if it doesn't work, try it again with --privileged). The number of users I've personally watched do exactly that blindly makes me very sad on a regular basis, so I'm confused why we'd be looking to create a second option, but with slightly less "remove all the protections" applied just for this one use case?

Yes, I was thinking; what privileges does --privileged provide that cannot be given in another way? (Wanted to make a table for that to point out the missing feature(s)).

If those missing options can be added, then instead of using --privileged ("all the things!") and a --but-not-fully-privileged flag, users could instead do, e.g.

--cap-add=all
--security-opt apparmor=unconfined
--security-opt seccomp=unconfined
--security-opt systempaths=unconfined 
--security-opt host-devices=unconfined

(start with "default", and add what's needed)

$ docker run -d --name dind --runtime=kata --security-opt apparmor=unconfined --security-opt seccomp=unconfined --security-opt systempaths=unconfined --cap-add all docker:19.03-dind
$ docker exec -it dind docker run -it --rm alpine
...
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:281: applying
cgroup configuration for process caused \"mkdir /sys/fs/cgroup/cpuset/docker: read-only file system\"": unknown.
...

(Kata 1.8.0 with Moby e20b732)

after remounting cgroup as read-write before starting dockerd-entrypoint:

$ docker exec -it dind docker run -it --rm alpine
docker: Error response from daemon: cgroups: cannot find cgroup mount destination: unknown.
ERRO[0002] error waiting for container: context canceled

For DIND usecase, this DIND PR seems to work: docker-library/docker#191

$ docker run --runtime kata -e DOCKER_REMOUNT_SYS_RW=1 --cap-add all --security-opt seccomp=unconfined --security-opt systempaths=unconfined

I can close this PR unless there is still demand from Kata maintainers.

I'm keeping this PR open, as discussed in Kata ML: http://lists.katacontainers.io/pipermail/kata-dev/2019-September/001029.html

@dmcgowan @justincormack ptal

rebased

CRI-O adopted equivalent of this as well as containerd/CRI: cri-o/cri-o#2730

Aside from Kata, this PR turned out to be also useful for protecting the host console from getty running in a container: AkihiroSuda/containerized-systemd#5 (comment)

cc @cpuguy83

For the (mostly privileged) systemd case, instead of doing --privileged, I can also just remount proc as writable in the container and not have to deal with host /dev issues.

I do think it's slightly unfortunate that --privileged also mounts the host's /dev instead of leaving that to the client to decide if they actually want that.

But you can gradually add what you need instead of relying on --privileged, and that seems good enough.

rebased

I still think this is probably not the right UX here -- how long before someone asks for another privileged-without-X option because they have a different use case?

I agree with @thaJeztah that it would be really useful if we could make a table of features that --privileged can enable that can't be enabled any other way, but it'd definitely be a great exercise to at least start with the things Kata needs and work our way up from there.

Kata v2 no longer supports runc-style CLI, and Moby does not support non-runc runtimes, so closing.

@AkihiroSuda Could you please reopen this PR?
Current version of Kata works perfectly fine with the actual version of Moby but #39697 prevents usage of privileged mode.
ctr, nerdctl and other tools already allows to run Kata privileged without host devices.
Thanks!

Re #39702 (comment), do you have a sense of which things Kata needs that are enabled by --privileged but cannot be enabled in any other way?

We also have use cases for kind (Kubernetes in Docker) that could benefit from this as well. Specifically around running as privileged, but allowing for fine-grained control of which devices get injected via CDI instead of blindly injecting all of them. Please see the following discussion for more details: kubernetes-sigs/kind#3290 (review)

I'm going to close this for now as I don't think we're likely to ever accept this in it's current form; however I have proposed an alternative at #47663 that I would like to direct feedback to.