Cannot disable userland-proxy in LXC container even though it works without it #49240
Description
Description
Docker is running inside an LXC (Incus) container and I want to disable "userland-proxy" in /etc/docker/daemon.json, but docker refuses to start:
Jan 09 00:14:08 avps-main systemd[1]: Starting docker.service - Docker Application Container Engine...
Jan 09 00:14:08 avps-main dockerd[5522]: time="2025-01-09T00:14:08.836425600-05:00" level=info msg="Starting up"
Jan 09 00:14:08 avps-main dockerd[5522]: time="2025-01-09T00:14:08.836535760-05:00" level=warning msg="Running experimental build"
Jan 09 00:14:08 avps-main dockerd[5522]: time="2025-01-09T00:14:08.837314080-05:00" level=info msg="OTEL tracing is not configured, using no-op tracer provider"
Jan 09 00:14:08 avps-main dockerd[5522]: time="2025-01-09T00:14:08.837479800-05:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Jan 09 00:14:08 avps-main dockerd[5522]: time="2025-01-09T00:14:08.891665640-05:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jan 09 00:14:08 avps-main dockerd[5522]: time="2025-01-09T00:14:08.894678720-05:00" level=info msg="Loading containers: start."
Jan 09 00:14:09 avps-main dockerd[5522]: time="2025-01-09T00:14:09.589450920-05:00" level=warning msg="could not create bridge network for id 0904eac5f42c394c5e3162c6e2338463e00e8a7f1aa8c37f3509f450e29153d6 bridge name docker0 while booting up from persistent state: cannot restrict inter-container communication or run without the userland proxy: stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory"
Jan 09 00:14:09 avps-main dockerd[5522]: time="2025-01-09T00:14:09.590939800-05:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jan 09 00:14:09 avps-main dockerd[5522]: time="2025-01-09T00:14:09.741558360-05:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Jan 09 00:14:09 avps-main dockerd[5522]: failed to start daemon: Error initializing network controller: error creating default "bridge" network: cannot restrict inter-container communication or run without the userland proxy: stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory
Jan 09 00:14:09 avps-main systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jan 09 00:14:09 avps-main systemd[1]: docker.service: Failed with result 'exit-code'.
It says "cannot restrict inter-container communication or run without the userland proxy" but this is not true, because with the default setting it actually is using iptables just fine, and the userland proxies are not used. I can kill the proxy processes and the port forwarding still works.
Reproduce
Make new incus container and install docker, put {"userland-proxy": false} in /etc/docker/daemon.json and restart docker, see if it fails
Expected behavior
Since the iptables forwards are working and the proxies are not doing anything, there should be no problem disabling them.
docker version
Client: Docker Engine - Community
Version: 27.4.1
API version: 1.47
Go version: go1.22.10
Git commit: b9d17ea
Built: Tue Dec 17 15:46:31 2024
OS/Arch: linux/arm64
Context: default
Server: Docker Engine - Community
Engine:
Version: 27.4.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.10
Git commit: c710b88
Built: Tue Dec 17 15:46:31 2024
OS/Arch: linux/arm64
Experimental: true
containerd:
Version: 1.7.24
GitCommit: 88bf19b2105c8b17560993bee28a01ddc2f97182
runc:
Version: 1.2.2
GitCommit: v1.2.2-0-g7cb3632
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
Client: Docker Engine - Community
Version: 27.4.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.19.3
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.32.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 3
Server Version: 27.4.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: true
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 88bf19b2105c8b17560993bee28a01ddc2f97182
runc version: v1.2.2-0-g7cb3632
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.1.0-28-arm64
Operating System: Ubuntu 24.04.1 LTS
OSType: linux
Architecture: aarch64
CPUs: 6
Total Memory: 7.751GiB
Name: avps-main
ID: 5d56fd11-c542-4b13-ae34-702167121a33
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: falseAdditional Info
I reproduced issue in new container on same host: Linux avps-host 6.1.0-28-arm64 #1 SMP Debian 6.1.119-1 (2024-11-22) aarch64 GNU/Linux
Incus version: 6.0.3
However, I was not able to reproduce on another host: Linux server 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Incus version: 6.5
docker version:
Client: Docker Engine - Community
Version: 27.3.1
API version: 1.47
Go version: go1.22.7
Git commit: ce12230
Built: Fri Sep 20 11:41:11 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 27.3.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.7
Git commit: 41ca978
Built: Fri Sep 20 11:41:11 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.23
GitCommit: 57f17b0a6295a39009d861b89e3b3b87b005ca27
runc:
Version: 1.1.14
GitCommit: v1.1.14-0-g2c9f560
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info:
Client: Docker Engine - Community
Version: 27.3.1
API version: 1.47
Go version: go1.22.7
Git commit: ce12230
Built: Fri Sep 20 11:41:11 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 27.3.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.7
Git commit: 41ca978
Built: Fri Sep 20 11:41:11 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.23
GitCommit: 57f17b0a6295a39009d861b89e3b3b87b005ca27
runc:
Version: 1.1.14
GitCommit: v1.1.14-0-g2c9f560
docker-init:
Version: 0.19.0
GitCommit: de40ad0
root@test-pds:~# docker info
Client: Docker Engine - Community
Version: 27.3.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.17.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.29.7
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 3
Running: 2
Paused: 0
Stopped: 1
Images: 4
Server Version: 27.3.1
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 57f17b0a6295a39009d861b89e3b3b87b005ca27
runc version: v1.1.14-0-g2c9f560
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-122-generic
Operating System: Debian GNU/Linux 12 (bookworm)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 94.22GiB
Name: test-pds
ID: 9c3c969c-ca60-4e4b-bb2d-38145aa9b04a
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
/proc/sys/net/bridge/bridge-nf-call-iptables does not exist here either yet this docker work fine.