Description
Docker Rootless fails to enable plugins on Docker v25.0.1 (with Debian12).
I have replicated this error with:
chris@debian:~$ docker plugin install vieux/sshfs
Plugin "vieux/sshfs" is requesting the following privileges:
- network: [host]
- mount: [/var/lib/docker/plugins/]
- mount: []
- device: [/dev/fuse]
- capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
latest: Pulling from vieux/sshfs
Digest: sha256:1d3c3e42c12138da5ef7873b97f7f32cf99fb6edde75fa4f0bcf9ed277855811
52d435ada6a4: Complete
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/docker/plugins/" to rootfs at "/mnt/state": stat /var/lib/docker/plugins/: permission denied: unknown
This error also occurs when you perform a docker plugin enable vieux/sshfs afterwards. So this is an issue with the enabling process, not the install process.
You can see more of the error with journalctl:
chris@debian:~$ journalctl --user -fu docker
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.243597136+01:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.243929909+01:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.243941735+01:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.244399134+01:00" level=info msg="starting signal loop" namespace=plugins.moby path=/run/.ro3508243726/user/1000/docker/containerd/daemon/io.containerd.runtime.v2.task/plugins.moby/6fddc473ff1bea75de4335bbe59954be87f866647327e9a3e2cb7f8a97fd025a pid=951 runtime=io.containerd.runc.v2
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.326335231+01:00" level=info msg="shim disconnected" id=6fddc473ff1bea75de4335bbe59954be87f866647327e9a3e2cb7f8a97fd025a
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.326443552+01:00" level=warning msg="cleaning up after shim disconnected" id=6fddc473ff1bea75de4335bbe59954be87f866647327e9a3e2cb7f8a97fd025a namespace=plugins.moby
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.326460424+01:00" level=info msg="cleaning up dead shim"
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.338050307+01:00" level=warning msg="cleanup warnings time=\"2024-01-30T09:17:43+01:00\" level=info msg=\"starting signal loop\" namespace=plugins.moby pid=980 runtime=io.containerd.runc.v2\ntime=\"2024-01-30T09:17:43+01:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/.ro3508243726/user/1000/docker/containerd/daemon/io.containerd.runtime.v2.task/plugins.moby/6fddc473ff1bea75de4335bbe59954be87f866647327e9a3e2cb7f8a97fd025a/init.pid: no such file or directory\"runtime=io.containerd.runc.v2\n"
Jan 30 09:17:43 debian dockerd-rootless.sh[725]: time="2024-01-30T09:17:43.338428072+01:00" level=error msg="copy shim log" error="read /proc/self/fd/14: file already closed"
Jan 30 09:17:43 debian dockerd-rootless.sh[699]: time="2024-01-30T09:17:43.344743825+01:00" level=error msg="Handler for POST /v1.44/plugins/vieux/sshfs:latest/enable returned error: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/var/lib/docker/plugins/\" to rootfs at \"/mnt/state\": stat /var/lib/docker/plugins/: permission denied: unknown"
I looked inside of the /run/user/1000/docker/containerd/daemon/io.containerd.runtime.v2.task/plugins.moby/6fddc473ff1bea75de4335bbe59954be87f866647327e9a3e2cb7f8a97fd025a directory (before it got deleted after the plugin fails to enable) and the init.pid indeed does not exist.
The problem does not exist on the same machine with Docker Rootfull.
Reproduce
I reproduced this bug on a clean Debian12 VM (using debian-12.4.0-amd64-netinst.iso). This is the entire history:
1 ls
2 sudo apt-get update
3 sudo apt-get install ca-certificates curl
4 sudo install -m 0755 -d /etc/apt/keyrings
5 sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
6 sudo chmod a+r /etc/apt/keyrings/docker.asc
7 echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
8 $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
9 sudo apt-get update
10 sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
11 sudo docker run --rm hello-world
12 sudo apt-get install -y dbus-user-session
13 sudo apt-get install -y fuse-overlayfs
14 slirp4netns --version
15 sudo systemctl disable --now docker.service docker.socket
16 dockerd-rootless-setuptool.sh install
17 sudo apt-get install uidmap
18 dockerd-rootless-setuptool.sh install
19 systemctl --user start docker
20 systemctl --user enable docker
21 sudo loginctl enable-linger $(whoami)
22 docker run --rm hello-world
23 docker plugin install ghcr.io/ibm/docker-logdna:1.0.1
24 docker context use rootless
25 docker plugin enable ghcr.io/ibm/docker-logdna:1.0.1
26 reboot
27 sudo reboot
28 docker plugin install vieux/sshfs
29*
30 docker plugin install vieux/sshfs
31 journalctl --user -fu docker
32 history
I performed the official Debian12 install instructions and the Docker Rootless installation instructions.
Expected behavior
docker plugin enable vieux/sshfs should not fail and properly enable the plugin.
docker version
Version: 25.0.1
API version: 1.44
Go version: go1.21.6
Git commit: 29cf629
Built: Tue Jan 23 23:09:46 2024
OS/Arch: linux/amd64
Context: rootless
Server: Docker Engine - Community
Engine:
Version: 25.0.1
API version: 1.44 (minimum version 1.24)
Go version: go1.21.6
Git commit: 71fa3ab
Built: Tue Jan 23 23:09:46 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.27
GitCommit: a1496014c916f9e62104b33d1bb5bd03b0858e59
runc:
Version: 1.1.11
GitCommit: v1.1.11-0-g4bccb38
docker-init:
Version: 0.19.0
GitCommit: de40ad0
rootlesskit:
Version: 2.0.0
ApiVersion: 1.1.1
NetworkDriver: slirp4netns
PortDriver: builtin
StateDir: /run/user/1000/dockerd-rootless
slirp4netns:
Version: 1.2.0
GitCommit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
docker info
Client: Docker Engine - Community
Version: 25.0.1
Context: rootless
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.12.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.24.2
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 25.0.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: false
userxattr: true
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: a1496014c916f9e62104b33d1bb5bd03b0858e59
runc version: v1.1.11-0-g4bccb38
init version: de40ad0
Security Options:
seccomp
Profile: builtin
rootless
cgroupns
Kernel Version: 6.1.0-17-amd64
Operating System: Debian GNU/Linux 12 (bookworm)
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 30.4GiB
Name: debian
ID: e90b8494-9305-4e46-9206-e397cdea0c59
Docker Root Dir: /home/chris/.local/share/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabledAdditional Info
No response
Description
Docker Rootless fails to enable plugins on Docker v25.0.1 (with Debian12).
I have replicated this error with:
vieux/sshfs(this plugin doesn't show this issue, see v25 Regression: Docker-Rootless Fails to Enable Plugins #47248 (comment))This error also occurs when you perform a
docker plugin enable vieux/sshfsafterwards. So this is an issue with the enabling process, not the install process.You can see more of the error with journalctl:
I looked inside of the
/run/user/1000/docker/containerd/daemon/io.containerd.runtime.v2.task/plugins.moby/6fddc473ff1bea75de4335bbe59954be87f866647327e9a3e2cb7f8a97fd025adirectory (before it got deleted after the plugin fails to enable) and theinit.pidindeed does not exist.The problem does not exist on the same machine with Docker Rootfull.
Reproduce
I reproduced this bug on a clean Debian12 VM (using
debian-12.4.0-amd64-netinst.iso). This is the entire history:I performed the official Debian12 install instructions and the Docker Rootless installation instructions.
Expected behavior
docker plugin enable vieux/sshfsshould not fail and properly enable the plugin.docker version
docker info
Additional Info
No response