Skip to content

userland-proxy: false does not clean-up NAT rule when switching to userland-proxy: true #44721

@polarathene

Description

@polarathene

Description

Network behaviour from userland-proxy: false carries over to userland-proxy: true.

Docker networks (at least with the default bridge, or custom bridges) are not behaving deterministically for userland-proxy: true, based on these two configs:

  • System and Docker daemon is started with userland-proxy: true (no switch from userland-proxy: false involved).
  • Daemon is started or restarted with userland-proxy: false set, and then afterwards switches to userland-proxy: true.

This is a niche bug. It will only be visible when adjusting this configuration without restarting the system. Thus likely only to occur while learning / debugging docker networking with this feature involved (commonly seen with IPv6 config advice / discussions).

I found this confusing to track down, but I am not sure what is going on behind the scenes to cause this.


Reproduction

Steps

Pay attention to the userland-proxy state and the outputs: value:

  1. echo '{ "userland-proxy": true }' > /etc/docker/daemon.json
  2. systemctl restart docker
  3. docker network create test
  4. docker run --rm -d -p 80:80 --network test --name bug traefik/whoami
  5. curl -s http://192.168.1.42 | grep RemoteAddr (outputs: RemoteAddr: 192.168.1.42)
  6. docker stop bug
  7. echo '{ "userland-proxy": false }' > /etc/docker/daemon.json
  8. systemctl restart docker
  9. docker run --rm -d -p 80:80 --network test --name bug traefik/whoami
  10. curl -s http://192.168.1.42 | grep RemoteAddr (outputs: RemoteAddr: 172.23.0.1)
  11. echo '{ "userland-proxy": true }' > /etc/docker/daemon.json
  12. systemctl restart docker
  13. docker run --rm -d -p 80:80 --network test --name bug traefik/whoami
  14. curl -s http://192.168.1.42 | grep RemoteAddr (outputs: RemoteAddr: 172.23.0.1)

Reproduction notes

NOTE: To reproduce:

  • 192.168.1.42 should be substituted for an IP on the hosts local interfaces, I've used the router assigned IP here, but have also reproduced with public IP (v4 and v6 addresses) from a VPS too.
  • 172.x.0.1 obviously being the docker network gateway connected to that container (it does not need to be a custom network, the default bridge network also exhibits this bug).
  • The initial step from userland-proxy: true is only to showcase that the expected value appears by default. The bug itself is specifically the userland-proxy: false to userland-proxy: true change.
  • You could alternatively use dockerd --userland-proxy=true and dockerd --userland-proxy=false to toggle instead of the daemon.json config, same issue.
  • If you create a new network after all this when userland-proxy: true, then test it it will resolve the correct remote address IP, while the earlier network remains affected from the previous userland-proxy: false changes until the system restarts.

Copy & paste shell commands:
# Replace IP 192.168.42 to one relevant on your machine:
echo '{ "userland-proxy": true }' > /etc/docker/daemon.json \
  && systemctl restart docker \
  && docker run --rm -d -p 80:80 traefik/whoami \
  && curl -s http://192.168.1.42 | grep RemoteAddr

# Correct output for `userland-proxy: true`:
RemoteAddr: 192.168.1.42


echo '{ "userland-proxy": false }' > /etc/docker/daemon.json \
  && systemctl restart docker \
  && docker run --rm -d -p 80:80 traefik/whoami \
  && curl -s http://192.168.1.42 | grep RemoteAddr

# Correct output for `userland-proxy: false`:
RemoteAddr: 172.23.0.1


echo '{ "userland-proxy": true }' > /etc/docker/daemon.json \
  && systemctl restart docker \
  && docker run --rm -d -p 80:80 traefik/whoami \
  && curl -s http://192.168.1.42 | grep RemoteAddr

# Incorrect output for `userland-proxy: true`:
RemoteAddr: 172.23.0.1

Expected behavior

userland-proxy: false to userland-proxy: true should not cause existing networks to behave differently than those created with userland-proxy: true (with no prior switch to userland-proxy: false involved). Both networks should behave the same way when userland-proxy: true is applied.

More info

docker version
Client: Docker Engine - Community
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:16 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:26:08 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
  compose: Docker Compose (Docker Inc., v2.14.1)
  scan: Docker Scan (Docker Inc., v0.23.0)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 20.10.22
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.19.0-26-generic
 Operating System: Ubuntu 22.10
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 968.9MiB
 Name: vultr-test
 ID: GNPV:BVKP:LWTR:FFLY:4LCT:RPF4:X6A7:M6DC:RWKL:WTEH:LHQS:EEFV
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Observations

When userland-proxy: false:

  • The reported remote address is that of the containers network gateway IP (expected, as similar effect to when iptables / ip6tables NAT is disabled).
  • userland-proxy: false also leads to curl http://[::1] timing out (curl http://127.0.0.1 would work however), while IPv6 addresses associated to other interfaces than the loopback were at least connecting.

When userland-proxy: true:

  • Despite the primary issue this bug details; - curl http://[::1] would now connect, and it will always return the expected docker network gateway IP matching that protocol (if there was an IPv6 enabled docker network + with ip6tables + experimental enabled in daemon.json, otherwise IPv4 gateway IP returned).
  • It is just the other host NICs that are expected to report back their own IP instead of the gateway of the docker network used (technically --network host would also report loopback IP instead of the gateway IP, but that's not a concern here).
  • The bug has only been observed when connecting from the host to a container; whereas connecting to the same public IP address of a host NIC remotely instead would still correctly report the expected remote IP (belonging to the connecting remote / client system). A remote client would only report the remote address IP as the docker networks Gateway IP only when userland-proxy: false in this case (for IPv4, and with relevant IPv6 config, the IPv6 equivalent gateway when appropriate instead of the IPv4 gateway)

/etc/docker/daemon.json used for enabling IPv6 with NAT on default docker bridge (so that containers report back the expected IPv4 / IPv6 address as remote):

{ 
  "ipv6": true,
  "fixed-cidr-v6": "fd00:cafe:babe::/48",
  "ip6tables": true,
  "experimental": true,
  "userland-proxy": true
}

NOTE: I have verified the bug without any of the IPv6 / experimental config present as well. Still affects IPv4.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions