Enable bridge netfiltering if userland-proxy=false by robmry · Pull Request #48676 · moby/moby

robmry · 2024-10-16T11:41:10Z

- What I did

Fix Docker reports "WARNING: bridge-nf-call-iptables/ip6tables is disabled" at startup after upgrading to 27.3.1 #48664

In release 27.0, ip6tables was enabled by default. That caused a problem on some hosts where iptables was explicitly disabled and loading the br_netfilter module (which loads with its nf-call-iptables settings enabled) caused user-defined iptables rules to block traffic on bridges, breaking inter-container communication.

In 27.3.0, #48511 delayed loading of the br_netfilter module until it was needed. The load now happens in the function that sets bridge-nf-call-ip[6]tables when needed. It was only called for icc=false networks.

However, br_netfilter is also needed when userland-proxy=false. Without it, packets addressed to a host-mapped port for a container on the same network are not DNAT'd properly (responses have the server container's address instead of the host's).

That means, in all releases including 26.x, if br_netfilter was loaded before the daemon started - and the OS/user/other-application had disabled bridge-nf-call-ip[6]tables (which apparently some OSs try to do), it would not be enabled by the daemon. So, ICC would fail for host-mapped ports with the userland-proxy disabled.

The change in 27.3.0 made this worse - previously, loading br_netfilter whenever iptables/ip6tables was enabled meant that bridge-netfiltering got enabled, even though the daemon didn't check it was enabled.

- How I did it

Check that br_netfilter is loaded, with bridge-nf-call-ip[6]tables enabled, if userland-proxy=false.

- How to verify it

New test that disables bridge-nf-call-iptables and the userland proxy, then tries to access another container's mapped port ... not quite the same as unloading br_netfilter - but this test fails with a daemon built from the 26.1 branch (as well as 27.x).

Manually tested on an Ubuntu VM, including with br_netfilter unloaded before starting the daemon.

- Description for the changelog

- Fix an issue that meant published ports from one container on a bridge network were not accessible
  from another container on the same network with `userland-proxy` disabled, if the kernel's `br_netfilter`
  module was not loaded and enabled. The daemon will now attempt to load the module and enable
  `bridge-nf-call-iptables` or `bridge-nf-call-ip6tables` when creating a network with the userland proxy
   disabled.

integration/networking/port_mapping_linux_test.go

thaJeztah · 2024-10-16T18:27:32Z

integration/networking/port_mapping_linux_test.go

+
+	// Find an address on the test host.
+	hostAddr := func() string {
+		conn, err := net.Dial("tcp4", "hub.docker.com:80")


if this needs to be an actual domain, we should consider using one of the designated ones (could be "example.com" here)

"example.com" is for documentation, I'm not sure using it in a test that expects it to be actually resolvable and listening is really in the spirit of the thing.

An alternative would be to look up the host's addresses using netlink, and pick one. Which is do-able as this is a Linux-only test. Other tests use this Dial trick now ... if they need to change, I could follow-up with a separate PR?

Right, yeah, I was looking for something not tied to "docker.com" and more generic. Not a big issue.

integration/networking/port_mapping_linux_test.go

thaJeztah · 2024-10-16T18:41:14Z

integration/networking/port_mapping_linux_test.go

+				container.WithCmd("wget", "http://"+hostAddr+":"+hostPort),
+			)
+			defer c.ContainerRemove(ctx, res.ContainerID, containertypes.RemoveOptions{Force: true})
+			assert.Check(t, is.Contains(res.Stderr.String(), "404 Not Found"))


Would -NS work with busybox as well, considering if the 404 Not Found would always have that format, and thus if we could somehow just get the status code only; https://serverfault.com/questions/1088710/how-to-use-wget-http-status-codes-in-an-if-else-statement-to-take-an-action-base

busybox's wget doesn't have a -N. Its -S shows HTTP response headers, so ...

/ # wget http://google.com/doesnotexist Connecting to google.com (142.250.200.46:80) wget: server returned error: HTTP/1.1 404 Not Found

Compared to ...

/ # wget -S http://google.com/doesnotexist Connecting to google.com (142.250.200.46:80) HTTP/1.1 404 Not Found wget: server returned error: HTTP/1.1 404 Not Found

So, without the -S the error message is just the HTTP first line. With -S, that first line is just shown twice. It's a standard format, very unlikely to change.

Ah, right, yeah, I was wondering if the output could contain a server response, which could be anything "different".

using -q (quiet) seems to work if we want to reduce additional output, but I guess it's fine;

/ # wget -q https://google.com/doesnotexist wget: server returned error: HTTP/1.1 404 Not Found

In release 27.0, ip6tables was enabled by default. That caused a problem on some hosts where iptables was explicitly disabled and loading the br_netfilter module (which loads with its nf-call-iptables settings enabled) caused user-defined iptables rules to block traffic on bridges, breaking inter-container communication. In 27.3.0, commit 5c499fc delayed loading of the br_netfilter module until it was needed. The load now happens in the function that sets bridge-nf-call-ip[6]tables when needed. It was only called for icc=false networks. However, br_netfilter is also needed when userland-proxy=false. Without it, packets addressed to a host-mapped port for a container on the same network are not DNAT'd properly (responses have the server container's address instead of the host's). That means, in all releases including 26.x, if br_netfilter was loaded before the daemon started - and the OS/user/other-application had disabled bridge-nf-call-ip[6]tables, it would not be enabled by the daemon. So, ICC would fail for host-mapped ports with the userland-proxy disabled. The change in 27.3.0 made this worse - previously, loading br_netfilter whenever iptables/ip6tables was enabled meant that bridge-netfiltering got enabled, even though the daemon didn't check it was enabled. So... check that br_netfilter is loaded, with bridge-nf-call-ip[6]tables enabled, if userland-proxy=false. Signed-off-by: Rob Murray <[email protected]>

thaJeztah

LGTM, thanks!

ledlamp · 2025-01-09T10:09:11Z

However, br_netfilter is also needed when userland-proxy=false. Without it, packets addressed to a host-mapped port for a container on the same network are not DNAT'd properly (responses have the server container's address instead of the host's).

Why do you need bridge filtering? All you need is SNAT.

iptables -t nat -A POSTROUTING -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE

Edit: I found the reason. With br_netfilter, the source IP is preserved (i.e. 172.17.0.3) and you can tell which container it's from. With MASQUERADE, the source IP will be the host's IP (i.e. 172.17.0.1). But this is the same case for the userland-proxies. So I think iptables MASQUERADE should be used as fallback instead of userland proxies. Userland proxy should only be used as last resort if iptables cannot be used (rarely the case).

robmry self-assigned this Oct 16, 2024

robmry added area/networking Networking impact/changelog kind/bugfix PR's that fix bugs area/networking/firewalling Networking area/networking/d/bridge Networking area/networking/portmapping Networking process/cherry-pick/27.x labels Oct 16, 2024

robmry added this to the 28.0.0 milestone Oct 16, 2024

robmry marked this pull request as ready for review October 16, 2024 12:55

robmry requested a review from akerouanton October 16, 2024 12:55

thaJeztah reviewed Oct 16, 2024

View reviewed changes

robmry force-pushed the 48664_br_netfilter_noproxy branch from 72a1bfe to 0548fe2 Compare October 17, 2024 11:33

thaJeztah approved these changes Oct 17, 2024

View reviewed changes

akerouanton approved these changes Oct 17, 2024

View reviewed changes

akerouanton merged commit 6649a32 into moby:master Oct 17, 2024

robmry deleted the 48664_br_netfilter_noproxy branch October 17, 2024 13:35

robmry mentioned this pull request Oct 17, 2024

[27.x backport] Enable bridge netfiltering if userland-proxy=false #48685

Merged

robmry added process/cherry-picked process/cherry-pick/23.0 process/cherry-pick/25.0 process/cherry-pick/26.1 and removed process/cherry-pick/27.x process/cherry-picked labels Oct 17, 2024

robmry mentioned this pull request Jan 9, 2025

Cannot disable userland-proxy in LXC container even though it works without it #49240

Open

robmry mentioned this pull request Jan 17, 2025

Allow users to ignore missing br_netfilter #49293

Merged

thaJeztah removed the process/cherry-pick/26.1 label Sep 9, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable bridge netfiltering if userland-proxy=false#48676

Enable bridge netfiltering if userland-proxy=false#48676
akerouanton merged 1 commit intomoby:masterfrom
robmry:48664_br_netfilter_noproxy

robmry commented Oct 16, 2024 •

edited

Loading

Uh oh!

Uh oh!

thaJeztah Oct 16, 2024

Uh oh!

robmry Oct 17, 2024

Uh oh!

thaJeztah Oct 17, 2024

Uh oh!

Uh oh!

thaJeztah Oct 16, 2024

Uh oh!

robmry Oct 17, 2024

Uh oh!

thaJeztah Oct 17, 2024 •

edited

Loading

Uh oh!

thaJeztah left a comment

Uh oh!

ledlamp commented Jan 9, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

robmry commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

thaJeztah Oct 16, 2024

Choose a reason for hiding this comment

Uh oh!

robmry Oct 17, 2024

Choose a reason for hiding this comment

Uh oh!

thaJeztah Oct 17, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thaJeztah Oct 16, 2024

Choose a reason for hiding this comment

Uh oh!

robmry Oct 17, 2024

Choose a reason for hiding this comment

Uh oh!

thaJeztah Oct 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

ledlamp commented Jan 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

robmry commented Oct 16, 2024 •

edited

Loading

thaJeztah Oct 17, 2024 •

edited

Loading

ledlamp commented Jan 9, 2025 •

edited

Loading