Description
Network behaviour from userland-proxy: false carries over to userland-proxy: true.
Docker networks (at least with the default bridge, or custom bridges) are not behaving deterministically for userland-proxy: true, based on these two configs:
- System and Docker daemon is started with
userland-proxy: true (no switch from userland-proxy: false involved).
- Daemon is started or restarted with
userland-proxy: false set, and then afterwards switches to userland-proxy: true.
This is a niche bug. It will only be visible when adjusting this configuration without restarting the system. Thus likely only to occur while learning / debugging docker networking with this feature involved (commonly seen with IPv6 config advice / discussions).
I found this confusing to track down, but I am not sure what is going on behind the scenes to cause this.
Reproduction
Steps
Pay attention to the userland-proxy state and the outputs: value:
echo '{ "userland-proxy": true }' > /etc/docker/daemon.json
systemctl restart docker
docker network create test
docker run --rm -d -p 80:80 --network test --name bug traefik/whoami
curl -s http://192.168.1.42 | grep RemoteAddr (outputs: RemoteAddr: 192.168.1.42)
docker stop bug
echo '{ "userland-proxy": false }' > /etc/docker/daemon.json
systemctl restart docker
docker run --rm -d -p 80:80 --network test --name bug traefik/whoami
curl -s http://192.168.1.42 | grep RemoteAddr (outputs: RemoteAddr: 172.23.0.1)
echo '{ "userland-proxy": true }' > /etc/docker/daemon.json
systemctl restart docker
docker run --rm -d -p 80:80 --network test --name bug traefik/whoami
curl -s http://192.168.1.42 | grep RemoteAddr (outputs: RemoteAddr: 172.23.0.1)
Reproduction notes
NOTE: To reproduce:
192.168.1.42 should be substituted for an IP on the hosts local interfaces, I've used the router assigned IP here, but have also reproduced with public IP (v4 and v6 addresses) from a VPS too.
172.x.0.1 obviously being the docker network gateway connected to that container (it does not need to be a custom network, the default bridge network also exhibits this bug).
- The initial step from
userland-proxy: true is only to showcase that the expected value appears by default. The bug itself is specifically the userland-proxy: false to userland-proxy: true change.
- You could alternatively use
dockerd --userland-proxy=true and dockerd --userland-proxy=false to toggle instead of the daemon.json config, same issue.
- If you create a new network after all this when
userland-proxy: true, then test it it will resolve the correct remote address IP, while the earlier network remains affected from the previous userland-proxy: false changes until the system restarts.
Copy & paste shell commands:
# Replace IP 192.168.42 to one relevant on your machine:
echo '{ "userland-proxy": true }' > /etc/docker/daemon.json \
&& systemctl restart docker \
&& docker run --rm -d -p 80:80 traefik/whoami \
&& curl -s http://192.168.1.42 | grep RemoteAddr
# Correct output for `userland-proxy: true`:
RemoteAddr: 192.168.1.42
echo '{ "userland-proxy": false }' > /etc/docker/daemon.json \
&& systemctl restart docker \
&& docker run --rm -d -p 80:80 traefik/whoami \
&& curl -s http://192.168.1.42 | grep RemoteAddr
# Correct output for `userland-proxy: false`:
RemoteAddr: 172.23.0.1
echo '{ "userland-proxy": true }' > /etc/docker/daemon.json \
&& systemctl restart docker \
&& docker run --rm -d -p 80:80 traefik/whoami \
&& curl -s http://192.168.1.42 | grep RemoteAddr
# Incorrect output for `userland-proxy: true`:
RemoteAddr: 172.23.0.1
Expected behavior
userland-proxy: false to userland-proxy: true should not cause existing networks to behave differently than those created with userland-proxy: true (with no prior switch to userland-proxy: false involved). Both networks should behave the same way when userland-proxy: true is applied.
More info
docker version
Client: Docker Engine - Community
Version: 20.10.22
API version: 1.41
Go version: go1.18.9
Git commit: 3a2c30b
Built: Thu Dec 15 22:28:16 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.22
API version: 1.41 (minimum version 1.12)
Go version: go1.18.9
Git commit: 42c8b31
Built: Thu Dec 15 22:26:08 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.14
GitCommit: 9ba4b250366a5ddde94bb7c9d1def331423aa323
runc:
Version: 1.1.4
GitCommit: v1.1.4-0-g5fd4c4d
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
compose: Docker Compose (Docker Inc., v2.14.1)
scan: Docker Scan (Docker Inc., v0.23.0)
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 2
Server Version: 20.10.22
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9ba4b250366a5ddde94bb7c9d1def331423aa323
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
cgroupns
Kernel Version: 5.19.0-26-generic
Operating System: Ubuntu 22.10
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 968.9MiB
Name: vultr-test
ID: GNPV:BVKP:LWTR:FFLY:4LCT:RPF4:X6A7:M6DC:RWKL:WTEH:LHQS:EEFV
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional Observations
When userland-proxy: false:
- The reported remote address is that of the containers network gateway IP (expected, as similar effect to when
iptables / ip6tables NAT is disabled).
userland-proxy: false also leads to curl http://[::1] timing out (curl http://127.0.0.1 would work however), while IPv6 addresses associated to other interfaces than the loopback were at least connecting.
When userland-proxy: true:
- Despite the primary issue this bug details; -
curl http://[::1] would now connect, and it will always return the expected docker network gateway IP matching that protocol (if there was an IPv6 enabled docker network + with ip6tables + experimental enabled in daemon.json, otherwise IPv4 gateway IP returned).
- It is just the other host NICs that are expected to report back their own IP instead of the gateway of the docker network used (technically
--network host would also report loopback IP instead of the gateway IP, but that's not a concern here).
- The bug has only been observed when connecting from the host to a container; whereas connecting to the same public IP address of a host NIC remotely instead would still correctly report the expected remote IP (belonging to the connecting remote / client system). A remote client would only report the remote address IP as the docker networks Gateway IP only when
userland-proxy: false in this case (for IPv4, and with relevant IPv6 config, the IPv6 equivalent gateway when appropriate instead of the IPv4 gateway)
/etc/docker/daemon.json used for enabling IPv6 with NAT on default docker bridge (so that containers report back the expected IPv4 / IPv6 address as remote):
{
"ipv6": true,
"fixed-cidr-v6": "fd00:cafe:babe::/48",
"ip6tables": true,
"experimental": true,
"userland-proxy": true
}
NOTE: I have verified the bug without any of the IPv6 / experimental config present as well. Still affects IPv4.
Description
Network behaviour from
userland-proxy: falsecarries over touserland-proxy: true.Docker networks (at least with the default bridge, or custom bridges) are not behaving deterministically for
userland-proxy: true, based on these two configs:userland-proxy: true(no switch fromuserland-proxy: falseinvolved).userland-proxy: falseset, and then afterwards switches touserland-proxy: true.This is a niche bug. It will only be visible when adjusting this configuration without restarting the system. Thus likely only to occur while learning / debugging docker networking with this feature involved (commonly seen with IPv6 config advice / discussions).
I found this confusing to track down, but I am not sure what is going on behind the scenes to cause this.
Reproduction
Steps
Pay attention to the
userland-proxystate and theoutputs:value:echo '{ "userland-proxy": true }' > /etc/docker/daemon.jsonsystemctl restart dockerdocker network create testdocker run --rm -d -p 80:80 --network test --name bug traefik/whoamicurl -s http://192.168.1.42 | grep RemoteAddr(outputs:RemoteAddr: 192.168.1.42)docker stop bugecho '{ "userland-proxy": false }' > /etc/docker/daemon.jsonsystemctl restart dockerdocker run --rm -d -p 80:80 --network test --name bug traefik/whoamicurl -s http://192.168.1.42 | grep RemoteAddr(outputs:RemoteAddr: 172.23.0.1)echo '{ "userland-proxy": true }' > /etc/docker/daemon.jsonsystemctl restart dockerdocker run --rm -d -p 80:80 --network test --name bug traefik/whoamicurl -s http://192.168.1.42 | grep RemoteAddr(outputs:RemoteAddr: 172.23.0.1)Reproduction notes
NOTE: To reproduce:
192.168.1.42should be substituted for an IP on the hosts local interfaces, I've used the router assigned IP here, but have also reproduced with public IP (v4 and v6 addresses) from a VPS too.172.x.0.1obviously being the docker network gateway connected to that container (it does not need to be a custom network, the default bridge network also exhibits this bug).userland-proxy: trueis only to showcase that the expected value appears by default. The bug itself is specifically theuserland-proxy: falsetouserland-proxy: truechange.dockerd --userland-proxy=trueanddockerd --userland-proxy=falseto toggle instead of thedaemon.jsonconfig, same issue.userland-proxy: true, then test it it will resolve the correct remote address IP, while the earlier network remains affected from the previoususerland-proxy: falsechanges until the system restarts.Copy & paste shell commands:
Expected behavior
userland-proxy: falsetouserland-proxy: trueshould not cause existing networks to behave differently than those created withuserland-proxy: true(with no prior switch touserland-proxy: falseinvolved). Both networks should behave the same way whenuserland-proxy: trueis applied.More info
docker version
docker info
Additional Observations
When
userland-proxy: false:iptables/ip6tablesNAT is disabled).userland-proxy: falsealso leads tocurl http://[::1]timing out (curl http://127.0.0.1would work however), while IPv6 addresses associated to other interfaces than the loopback were at least connecting.When
userland-proxy: true:curl http://[::1]would now connect, and it will always return the expected docker network gateway IP matching that protocol (if there was an IPv6 enabled docker network + withip6tables+experimentalenabled indaemon.json, otherwise IPv4 gateway IP returned).--network hostwould also report loopback IP instead of the gateway IP, but that's not a concern here).userland-proxy: falsein this case (for IPv4, and with relevant IPv6 config, the IPv6 equivalent gateway when appropriate instead of the IPv4 gateway)/etc/docker/daemon.jsonused for enabling IPv6 with NAT on default docker bridge (so that containers report back the expected IPv4 / IPv6 address as remote):{ "ipv6": true, "fixed-cidr-v6": "fd00:cafe:babe::/48", "ip6tables": true, "experimental": true, "userland-proxy": true }NOTE: I have verified the bug without any of the IPv6 / experimental config present as well. Still affects IPv4.