28.0.0: host exposed ports cannot be accessed from other bridge network with gateway_mode=routed #49509
Description
Description
After upgrading to Docker 28.0.0, I can no longer access exposed ports in one bridge network (br_docker with gateway_mode=nat) from another bridge network (br0, gateway_mode=routed).
The issue seems to be that since v28, a RETURN rule is added to nat-DOCKER chain, that prevents the DNAT for br_docker from applying to traffic coming from br0.
$ sudo iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 73919 packets, 10M bytes)
pkts bytes target prot opt in out source destination
356 67096 DOCKER 0 -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 157 packets, 12131 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER 0 -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 67208 packets, 8858K bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE 0 -- * br0 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match src-type LOCAL
1 40 MASQUERADE 0 -- * br_docker 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match src-type LOCAL
0 0 MASQUERADE 0 -- * !br_docker 10.50.0.0/16 0.0.0.0/0
0 0 MASQUERADE 6 -- * * 10.50.0.2 10.50.0.2 tcp dpt:80
0 0 MASQUERADE 17 -- * * 10.50.0.2 10.50.0.2 udp dpt:80
0 0 MASQUERADE 6 -- * * 10.50.0.2 10.50.0.2 tcp dpt:443
0 0 MASQUERADE 17 -- * * 10.50.0.2 10.50.0.2 udp dpt:443
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
+ 51 7626 RETURN 0 -- br0 * 0.0.0.0/0 0.0.0.0/0 <----- this is the offending rule ----------
0 0 DNAT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:10.50.0.2:80
0 0 DNAT 17 -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:80 to:10.50.0.2:80
0 0 DNAT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 to:10.50.0.2:443
0 0 DNAT 17 -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:443 to:10.50.0.2:443After some digging, I found that this change of behavior was introduced in #48596 (which interestingly was about making the access in the other direction possible) in this diff.
As far as I understand, before this change, the rule was only added in nat mode and now it is added in routed mode.
Yet, the PR description seems to not mention such changes to the nat-DOCKER chain.
Reproduce
Minimal-ish example (without IPv6 – but this issue also affects IPv6 in the same way) using compose:
services:
web:
image: nginx
ports:
- "80:80"
networks:
- bridge
dummy1:
image: alpine:3
networks:
- bridge
dummy2:
image: alpine:3
networks:
- br0
networks:
bridge:
name: 1_bridge
driver: bridge
driver_opts:
com.docker.network.bridge.name: br_docker
com.docker.network.bridge.gateway_mode_ipv4: nat
ipam:
config:
- subnet: 10.50.0.0/16
gateway: 10.50.0.1
ip_range: 10.50.0.0/24
br0:
name: 2_br0
driver: bridge
driver_opts:
com.docker.network.bridge.name: br0
com.docker.network.bridge.gateway_mode_ipv4: routed
ipam:
config:
- subnet: 10.10.0.0/16
gateway: 10.10.0.1
ip_range: 10.10.0.0/24Note: In my original setup, I have inhibit_ipv4: 'true' set to true for br0, since this bridge already exists on the host and is used by a KVM guest and the host itself.
However, this issue exists independent of this setting.
Now, from dummy1, I can access the http server via the exposed port on the host (e.g. wget http://10.50.0.1, or also wget http://10.10.0.1).
From dummy2, it cannot access the exposed port on the host (times out).
Expected behavior
Should be able to access exposed port on host via wget http://10.50.0.1 from dummy2.
docker version
Client: Docker Engine - Community
Version: 28.0.0
API version: 1.48
Go version: go1.23.6
Git commit: f9ced58
Built: Wed Feb 19 22:11:04 2025
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 28.0.0
API version: 1.48 (minimum version 1.24)
Go version: go1.23.6
Git commit: af898ab
Built: Wed Feb 19 22:11:04 2025
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.25
GitCommit: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc:
Version: 1.2.4
GitCommit: v1.2.4-0-g6c52b3f
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
Client: Docker Engine - Community
Version: 28.0.0
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.21.0
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.33.0
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 9
Running: 1
Paused: 0
Stopped: 8
Images: 20
Server Version: 28.0.0
Storage Driver: btrfs
Btrfs:
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc version: v1.2.4-0-g6c52b3f
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.11.0-17-generic
Operating System: Ubuntu 24.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.536GiB
Name: silver
ID: OJ4N:65W5:UVWU:QIDN:CPCZ:EFWT:PXJ3:SMJP:M2YD:2FHG:DXBW:NAJW
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: falseAdditional Info
No response