Skip to content

Allow users to ignore missing br_netfilter#49293

Merged
thaJeztah merged 1 commit intomoby:masterfrom
robmry:49240_ignore_br_netfilter_error
Jan 18, 2025
Merged

Allow users to ignore missing br_netfilter#49293
thaJeztah merged 1 commit intomoby:masterfrom
robmry:49240_ignore_br_netfilter_error

Conversation

@robmry
Copy link
Copy Markdown
Contributor

@robmry robmry commented Jan 17, 2025

- What I did

Add a workaround for users wanting to run with br_netfilter unloaded when it's needed.

Since commit 0f8fc31, the bridge driver will try to load kernel module br_netfilter if the userland proxy is disabled. If it fails, we're in unknown territory, so it's treated as an error. At the very least, containers will not be able to access host ports mapped to other containers in the same network.

Before that, and before commit 5c499fc delayed the module load until it was needed - it was loaded unconditionally, but errors were only logged.

So, on systems where the module is not available, or could not be loaded/configured, no error was reported and things "worked" (as long as you didn't try to use something that didn't work).

- How I did it

That behaviour has been useful to some. So, make it possible to ignore the problem by setting env var: DOCKER_IGNORE_BR_NETFILTER_ERROR=1.

- How to verify it

  • unloaded br_netfilter and moved aside br_netfilter.ko so that it couldn't be reloaded
  • set "userland-proxy": false
  • checked the daemon wouldn't start
DEBU[2025-01-18T11:57:21.866523729Z] Modules not loaded                            checkResult="stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory" loadErrors="<nil>" loader=ioctl modules="[br_netfilter]"
DEBU[2025-01-18T11:57:21.868739448Z] Modules not loaded                            checkResult="stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory" loadErrors="modprobe br_netfilter failed with message: \"modprobe: ERROR: could not insert 'br_netfilter': Unknown symbol in module, or unknown parameter (see dmesg)\\ninsmod /lib/modules/6.1.0-18-amd64/kernel/net/bridge/br_netfilter.ko\", error: exit status 1" loader=modprobe modules="[br_netfilter]"
DEBU[2025-01-18T11:57:21.868849544Z] releasing IPv4 pools from network bridge (4cd4e4e505a4f9876c4340ceb0c04e8e62e7e56abeff445b82908e115c0de2d1)
DEBU[2025-01-18T11:57:21.868862489Z] ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.1)
DEBU[2025-01-18T11:57:21.868883158Z] Released address Address:172.17.0.1 Sequence:Bits: 65536, Unselected: 65534, Sequence: (0x80000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0
DEBU[2025-01-18T11:57:21.868888024Z] ReleasePool(LocalDefault/172.17.0.0/16)
DEBU[2025-01-18T11:57:21.868929270Z] daemon configured with a 15 seconds minimum shutdown timeout
DEBU[2025-01-18T11:57:21.868937099Z] start clean shutdown of all containers with a 15 seconds timeout...
DEBU[2025-01-18T11:57:21.870338585Z] Unix socket /var/run/docker/libnetwork/fb21ef685198.sock was closed. The external key listener will stop.
DEBU[2025-01-18T11:57:21.870511438Z] Cleaning up old mountid : start.
DEBU[2025-01-18T11:57:21.870733127Z] Cleaning up old mountid : done.
failed to start daemon: Error initializing network controller: error creating default "bridge" network: cannot restrict inter-container communication or run without the userland proxy: stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory: set environment variable DOCKER_IGNORE_BR_NETFILTER_ERROR=1 to ignore
  • export DOCKER_IGNORE_BR_NETFILTER_ERROR=1
  • the daemon started with logs ...
DEBU[2025-01-18T11:58:03.724166300Z] Modules not loaded                            checkResult="stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory" loadErrors="<nil>" loader=ioctl modules="[br_netfilter]"
DEBU[2025-01-18T11:58:03.726415506Z] Modules not loaded                            checkResult="stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory" loadErrors="modprobe br_netfilter failed with message: \"modprobe: ERROR: could not insert 'br_netfilter': Unknown symbol in module, or unknown parameter (see dmesg)\\ninsmod /lib/modules/6.1.0-18-amd64/kernel/net/bridge/br_netfilter.ko\", error: exit status 1" loader=modprobe modules="[br_netfilter]"
WARN[2025-01-18T11:58:03.726479764Z] Continuing without enabling br_netfilter      error="cannot restrict inter-container communication or run without the userland proxy: stat /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory"

- Description for the changelog

- On a host that cannot load the `br_netfilter` module when it's needed, set environment variable
  `DOCKER_IGNORE_BR_NETFILTER_ERROR=1` to ignore the problem.
  - Some things won't work! Including disabling inter-container communication in a bridge network
    and, with the userland proxy disabled, it won't be possible to access one container's published
    ports from another container on the same network.

func enableBridgeNetFiltering(nfParam string) (retErr error) {
defer func() {
if retErr != nil {
if ignore, err := strconv.ParseBool(os.Getenv("DOCKER_IGNORE_BR_NETFILTER_ERROR")); err == nil && ignore {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that this

  • is really an escape-hatch and (possibly) temporary
  • the default will very unlikely ever be enabled and the env-var to be used to disable

I think we could simplify to just check for a non-empty value to be set, similar to what we do for some other escape-hatch and/or undocumented env-vars, e.g.

moby/daemon/daemon.go

Lines 842 to 845 in 7e78f78

// TEST_INTEGRATION_USE_SNAPSHOTTER is used for integration tests only.
if os.Getenv("TEST_INTEGRATION_USE_SNAPSHOTTER") != "" {
d.usesSnapshotter = true
} else {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure - done.

retErr = nil
return
}
retErr = fmt.Errorf("%w: DOCKER_IGNORE_BR_NETFILTER_ERROR=1 to ignore", retErr)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that we document it like this, I guess we can even just check for;

if os.Getenv("DOCKER_IGNORE_BR_NETFILTER_ERROR") == "1" {

(but just "any non-empty value" would still work))

@robmry robmry force-pushed the 49240_ignore_br_netfilter_error branch from 1b6fd6d to 9c295fd Compare January 17, 2025 23:29
func enableBridgeNetFiltering(nfParam string) (retErr error) {
defer func() {
if retErr != nil {
if _, ok := os.LookupEnv("DOCKER_IGNORE_BR_NETFILTER_ERROR"); ok {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh; I think it's better to look for a non-empty value; this would also match the env when set to an empty value;

export DOCKER_IGNORE_BR_NETFILTER_ERROR=

env | grep DOCKER_IGNORE_BR_NETFILTER_ERROR
DOCKER_IGNORE_BR_NETFILTER_ERROR=

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I updated the description to show that.

I don't like the idea that these would all work, but actually do the opposite of what they look like they'll do ...

export DOCKER_IGNORE_BR_NETFILTER_ERROR=0
export DOCKER_IGNORE_BR_NETFILTER_ERROR=false
export DOCKER_IGNORE_BR_NETFILTER_ERROR=no

But this wouldn't do anything ...

export DOCKER_IGNORE_BR_NETFILTER_ERROR=

I think we've used up 2025's quota for bikeshedding option parsers (!), so I've taken your second suggestion and changed it to check for "1".

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah sorry 🙈

So yeah; my motivation for the "empty value" was mostly to prevent cases where these could be set through scripts; I've seen a fair share of export FOO_BAR=$UNDEFINED_VAR.

export DOCKER_IGNORE_BR_NETFILTER_ERROR=0

Yeah, I can slightly more relate to that! I was mostly trying to align with other options, and not parse if not set, but perhaps we should align all of these, and have something similar to

func BoolValueOrDefault(r *http.Request, k string, d bool) bool {
if _, ok := r.Form[k]; !ok {
return d
}
return BoolValue(r, k)
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, no worries! It's worth getting this stuff right and easy to change now.

Allowing only "1" here doesnt rule anything out, we can relax the constraint and make it more bool-y in future, but it'd be more difficult to go the other way. I think it's good, clear and simple.

It does seem like we need think up a policy for option naming and handling, as we keep debating it for individual new settings. But, no doubt, we already have a fair few that won't fit whatever mould we can come up with. (Not that this particular env var should last long, ideally.)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose we could treat anything apart from "1", no value, or unset as invalid rather than ignoring it ... then it would be easier to change later.

Since commit 0f8fc31, the bridge driver will try to load kernel
module br_netfilter if the userland proxy is disabled. If it fails,
we're in unknown territory, so it's treated as an error. At the
very least, containers will not be able to access host ports
mapped to other containers in the same network.

Before that, and before commit 5c499fc delayed the module load
until it was needed - it was loaded unconditionally, but errors
were only logged.

So, on systems where the module is not available, or could not be
loaded/configured, no error was reported and things "worked" (as
long as you didn't try to use something that didn't work).

That behaviour has been useful to some. So, make it possible to
ignore the problem by setting env var:
  DOCKER_IGNORE_BR_NETFILTER_ERROR=1

Signed-off-by: Rob Murray <[email protected]>
@robmry robmry force-pushed the 49240_ignore_br_netfilter_error branch from 9c295fd to e7bd60e Compare January 18, 2025 11:59
Copy link
Copy Markdown
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah thaJeztah merged commit e718b06 into moby:master Jan 18, 2025
@robmry robmry deleted the 49240_ignore_br_netfilter_error branch February 26, 2025 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants