Description
Docker containers started with --privileged are still subject to the AppArmor policies of the host. These policies can cause weird failures, like the one described below that I hit. This exposes the container to side-effects due to the OS version and configuration of the host, which seems like an explicit anti-goal for Docker.
This bug was reported before in #5490 and was fixed in #14855. Unfortunately, it seems like that fix was reverted in #15386 with only the comment "we were not using it".
Steps to reproduce the issue:
- Install Docker on an Ubuntu 18.04 host.
- For this repro, ensure Docker is using the
aufs storage driver. The bug exists regardless, but aufs hits an easy repro.
- Run Bash in an Ubuntu container:
docker run --rm --privileged -it ubuntu:18.04
- In the container, install
man:
apt update && apt install -y man
- Try to view a manpage:
Describe the results you received:
root@87295e39d8da:/# man bash
man: error while loading shared libraries: libmandb-2.8.3.so: cannot open shared object file: Permission denied
Describe the results you expected:
I actually don't expect man to succeed, because the Ubuntu container is minimized and doesn't contain manpages. But the failure shouldn't involve permissions to a shared library.
Additional information you deem important (e.g. issue happens only occasionally):
Picking up at the repro steps above, let's prove the problem is AppArmor. Per Ubuntu's instructions, let's temporarily disable the usr.bin.man AppArmor profile on the host:
# we're changing the in-memory state; this doesn't persist across reboot
sudo apparmor_parser -R /etc/apparmor.d/usr.bin.man
Now, running man in the same container that was running before gives us behavior closer to what we'd expect:
root@87295e39d8da:/# man bash
No manual entry for bash
See 'man 7 undocumented' for help when manual pages are not available.
Back on the host, re-enable the AppArmor rule with:
sudo apparmor_parser /etc/apparmor.d/usr.bin.man
and the behavior switches right back:
root@87295e39d8da:/# man bash
man: error while loading shared libraries: libmandb-2.8.3.so: cannot open shared object file: Permission denied
As a bonus, we can look at dmesg on the host to see the audit message from AppArmor:
[512889.244238] audit: type=1400 audit(1545553170.210:305): apparmor="DENIED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 profile="/usr/bin/man" name="var/lib/docker/aufs/diff/8d9c0c9f95b7d0c1ac37086fe906d51e56cd85a50f6f0bc7877ffacc890d2b03/usr/lib" pid=6008 comm="man" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Fun question 1: If we started the container with --privileged, why does AppArmor have any effect?
Using docker run --privileged starts the container with the unconfined AppArmor profile. But unconfined isn't actually a profile as much as it is the lack of a profile. As soon as we start a binary like /usr/bin/man that has a matching attachment specification, AppArmor applies the corresponding profile to the new process.
Fun question 2: Why does the /usr/bin/man profile apply to a binary in a container?
AppArmor rules are namespace_relative by default.
Fun question 3: The /usr/bin/man profile allows access to the entire filesystem. Why is AppArmor denying this request?
According to the dmesg line above, the path that aufs passes to AppArmor doesn't start with a /. By default, AppArmor won't assume the path should be interpreted as absolute. (In AppArmor parlance, the default for profiles is no_attach_disconnected.) So the filesystem access doesn't match the allow rule.
Bonus fun question 4: Why do I get the same error even if I use sudo aa-complain /usr/bin/man on the host to turn the rule into "complain" mode?
Sure enough, even though a "complain" rule should never cause requests to fail, that's the behavior I saw in Ubuntu 18.04. The dmesg output even says the request should be allowed, though it still returns -EACCES:
[513604.961744] audit: type=1400 audit(1545553885.927:315): apparmor="ALLOWED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 profile="/usr/bin/man" name="var/lib/docker/aufs/diff/8d9c0c9f95b7d0c1ac37086fe906d51e56cd85a50f6f0bc7877ffacc890d2b03/usr/lib" pid=6032 comm="man" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
I haven't debugged much into this, but my guess is there's a bug in AppArmor similar to LP1656121, where AppArmor fails a request because it encounters an error while evaluating a rule.
Output of docker version:
Client:
Version: 18.09.0
API version: 1.39
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:49:01 2018
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.0
API version: 1.39 (minimum version 1.12)
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:16:44 2018
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 6
Server Version: 18.09.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 16
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-42-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.83GiB
Name: ubuntu-closet
ID: O6SP:IAU7:DIIZ:PNQE:VMMD:FTUN:FBW6:V6PM:PZ72:IMPY:IRFV:3SSG
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.):
Hyper-V VM running Ubuntu 18.04
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"
$ uname -a
Linux ubuntu-closet 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Description
Docker containers started with
--privilegedare still subject to the AppArmor policies of the host. These policies can cause weird failures, like the one described below that I hit. This exposes the container to side-effects due to the OS version and configuration of the host, which seems like an explicit anti-goal for Docker.This bug was reported before in #5490 and was fixed in #14855. Unfortunately, it seems like that fix was reverted in #15386 with only the comment "we were not using it".
Steps to reproduce the issue:
aufsstorage driver. The bug exists regardless, butaufshits an easy repro.man:apt update && apt install -y manDescribe the results you received:
Describe the results you expected:
I actually don't expect
manto succeed, because the Ubuntu container is minimized and doesn't contain manpages. But the failure shouldn't involve permissions to a shared library.Additional information you deem important (e.g. issue happens only occasionally):
Picking up at the repro steps above, let's prove the problem is AppArmor. Per Ubuntu's instructions, let's temporarily disable the
usr.bin.manAppArmor profile on the host:Now, running
manin the same container that was running before gives us behavior closer to what we'd expect:Back on the host, re-enable the AppArmor rule with:
and the behavior switches right back:
As a bonus, we can look at
dmesgon the host to see the audit message from AppArmor:Fun question 1: If we started the container with
--privileged, why does AppArmor have any effect?Using
docker run --privilegedstarts the container with theunconfinedAppArmor profile. Butunconfinedisn't actually a profile as much as it is the lack of a profile. As soon as we start a binary like/usr/bin/manthat has a matching attachment specification, AppArmor applies the corresponding profile to the new process.Fun question 2: Why does the
/usr/bin/manprofile apply to a binary in a container?AppArmor rules are
namespace_relativeby default.Fun question 3: The
/usr/bin/manprofile allows access to the entire filesystem. Why is AppArmor denying this request?According to the
dmesgline above, the path thataufspasses to AppArmor doesn't start with a/. By default, AppArmor won't assume the path should be interpreted as absolute. (In AppArmor parlance, the default for profiles isno_attach_disconnected.) So the filesystem access doesn't match the allow rule.Bonus fun question 4: Why do I get the same error even if I use
sudo aa-complain /usr/bin/manon the host to turn the rule into "complain" mode?Sure enough, even though a "complain" rule should never cause requests to fail, that's the behavior I saw in Ubuntu 18.04. The
dmesgoutput even says the request should be allowed, though it still returns-EACCES:I haven't debugged much into this, but my guess is there's a bug in AppArmor similar to LP1656121, where AppArmor fails a request because it encounters an error while evaluating a rule.
Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
Hyper-V VM running Ubuntu 18.04