Add IPv6 nameserver to the internal DNS's upstreams. by robmry · Pull Request #47512 · moby/moby

robmry · 2024-03-06T11:39:27Z

- What I did

When configuring the internal DNS resolver - rather than keep IPv6 nameservers read from the host's resolv.conf in the container's resolv.conf, treat them like IPv4 addresses and use them as upstream resolvers.

Fixes #46329

- How I did it

For IPv6 nameservers, if there's a zone identifier in the address or the container itself doesn't have IPv6 support, mark the upstream addresses for use in the host's network namespace.

- How to verify it

Updated unit tests.

I gather our test hosts don't have IPv6, so I don't think we can have an integration test with an IPv6 upstream resolver (?).

On an Ubuntu VM ...

With the host's resolv.conf listing one of Google's IPv6 nameservers - before this change, the IPv6 nameserver ends up in the container's resolv.conf along with the internal resolver ...

$ docker run --rm -ti --network nnn6 --name ccc alpine
/ # cat /etc/resolv.conf
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.

nameserver 127.0.0.11
nameserver 2001:4860:4860::8888
search Home
options edns0 trust-ad ndots:0

# Based on host file: '/etc/resolv.conf' (internal resolver)
# Overrides: []
# Option ndots from: internal
/ #

With the change, the upstream IPv6 server is listed as an "ExtServer" ...

$ docker run --rm -ti --network nnn6 --name ccc alpine
/ # cat /etc/resolv.conf
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.

nameserver 127.0.0.11
search Home
options edns0 trust-ad ndots:0

# Based on host file: '/etc/resolv.conf' (internal resolver)
# ExtServers: [2001:4860:4860::8888]
# Overrides: []
# Option ndots from: internal
/ # ping -6 google.com
PING google.com (2a00:1450:4009:827::200e): 56 data bytes
64 bytes from 2a00:1450:4009:827::200e: seq=0 ttl=118 time=9.584 ms
64 bytes from 2a00:1450:4009:827::200e: seq=1 ttl=118 time=9.184 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 9.184/9.384/9.584 ms
/ # ping ccc
PING ccc (172.19.0.2): 56 data bytes
64 bytes from 172.19.0.2: seq=0 ttl=64 time=0.021 ms
^C
--- ccc ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.021/0.021/0.021 ms
/ # ping -6 ccc
PING ccc (fd00:1234::2): 56 data bytes
64 bytes from fd00:1234::2: seq=0 ttl=64 time=0.039 ms
64 bytes from fd00:1234::2: seq=1 ttl=64 time=0.144 ms
^C

When the host's resolv.conf has a zone-id in the nameserver's address, the ExtServer is listed with the host marker, meaning the upstream request will happen in the host's namespace ...

$ docker run --rm -ti --network nnn6 --name ccc alpine
/ # cat /etc/resolv.conf
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.

nameserver 127.0.0.11
search Home
options edns0 trust-ad ndots:0

# Based on host file: '/etc/resolv.conf' (internal resolver)
# ExtServers: [host(2001:4860:4860::8888%enp0s3)]
# Overrides: []
# Option ndots from: internal
/ # ping google.com
PING google.com (216.58.204.78): 56 data bytes
64 bytes from 216.58.204.78: seq=0 ttl=118 time=8.762 ms
64 bytes from 216.58.204.78: seq=1 ttl=118 time=10.361 ms
^C

When the container has IPv6 disabled, the IPv6 ExtServer is still used - but from the host's namespace ...

$ docker run --rm -ti --network nnn6 --name ccc --sysctl net.ipv6.conf.all.disable_ipv6=1  alpine
/ # cat /etc/resolv.conf
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.

nameserver 127.0.0.11
search Home
options edns0 trust-ad ndots:0

# Based on host file: '/etc/resolv.conf' (internal resolver)
# ExtServers: [host(2001:4860:4860::8888)]
# Overrides: []
# Option ndots from: internal
/ # ping google.com
PING google.com (216.58.204.78): 56 data bytes
64 bytes from 216.58.204.78: seq=0 ttl=118 time=14.229 ms
^C

- Description for the changelog

Use IPv6 nameservers from the host's `resolv.conf` as upstream resolvers for Docker Engine's internal DNS, rather than listing them in the container's `resolv.conf`.

When configuring the internal DNS resolver - rather than keep IPv6 nameservers read from the host's resolv.conf in the container's resolv.conf, treat them like IPv4 addresses and use them as upstream resolvers. For IPv6 nameservers, if there's a zone identifier in the address or the container itself doesn't have IPv6 support, mark the upstream addresses for use in the host's network namespace. Signed-off-by: Rob Murray <[email protected]>

corhere · 2024-03-06T20:47:14Z

The HostLoopback rules are getting pretty complicated and subtle with this change. With the possibility for a split horizon (i.e. the same global-scope prefix routes to a container network inside the container netns, but to a physical network in the host netns) we will certainly pick incorrectly at least some of the time. And the users impacted don't have any way to overrule our heuristics.

Since we're already making a user-facing change to DNS forwarding, what do you think of ripping the band-aid off and also overhauling the IPv4 forwarding behaviour in the same major release?

If we could start again from scratch without any need to maintain backwards-compatibility, my ideal behaviour would be:

Nameservers loaded from the host's resolv.conf are unconditionally queried from the host netns.
Nameservers set in container configuration are queried from the container netns by default, mirroring the behaviour of having the nameserver listed in the container's /etc/resolv.conf.
For each nameserver set in container configuration, an option may be enabled to have the nameserver queried from the host netns.

Note that in my ideal world, the side of the horizon custom each nameserver is resolved from is determined by the source of the address and user preference, not by any properties of the address itself.

Of course, we cannot just change the DNS forwarding behaviour for all containers in the next major release without breaking some subset of users. But I think it would be possible to make that the behaviour for new containers without breaking existing containers or giving ourselves an unreasonable maintenance burden to carry. If "forward query from host vs. container netns" is an explicit per-nameserver option in the container config, we could migrate existing containers and transform container-create requests for legacy API versions to have legacy forwarding behaviour by applying the existing rules. As for legacy containers without DNS overrides, it might be okay to unconditionally resolve nameservers loaded from the host's resolv.conf in the host's netns. Anyone relying on host resolv.conf nameservers being resolved from the container's netns is depending on really subtle and brittle behaviour. Anyone intentionally depending on such behaviour could simply work around the breaking change by overriding the nameservers in the container config. Otherwise, we could make it so that legacy containers see no breaking changes at all by extending the container config with an option to use the legacy "only resolve loopback addresses from resolv.conf in the host netns" behaviour which is implicitly enabled on migrated containers and containers created with legacy API versions.

neersighted

This change actually simplifies the semantics around proxyDNS, and the tests are quite straightforward.

LGTM; I am in favor of reconsidering the DNS semantics, but at the same time I think we can open a follow-up issue to do so.

corhere

I had confused myself into thinking incorrectly that the HostLoopback heuristics were also applied to user-configured DNS overrides. LGTM.

robmry self-assigned this Mar 6, 2024

robmry added area/networking Networking kind/bugfix PR's that fix bugs area/networking/ipv6 Networking area/networking/dns Networking labels Mar 6, 2024

robmry marked this pull request as ready for review March 6, 2024 13:07

robmry requested review from akerouanton and corhere March 6, 2024 13:07

neersighted approved these changes Mar 7, 2024

View reviewed changes

corhere approved these changes Mar 7, 2024

View reviewed changes

thaJeztah merged commit 1abf17c into moby:master Mar 7, 2024

neersighted added this to the 26.0.0 milestone Mar 7, 2024

neersighted added the impact/changelog label Mar 7, 2024

cleverer mentioned this pull request Mar 21, 2024

Feature request: IPv6-enabled embedded DNS server #41651

Closed

robmry deleted the 46329_internal_resolver_ipv6_upstream branch March 27, 2024 16:34

notnamed mentioned this pull request Apr 23, 2024

Certbot Connection Timeout Issue on IPv6-Only Servers during Installation bitwarden/self-host#220

Closed

robmry mentioned this pull request May 16, 2024

Networking overview - DNS fixes docker/docs#20049

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add IPv6 nameserver to the internal DNS's upstreams.#47512

Add IPv6 nameserver to the internal DNS's upstreams.#47512
thaJeztah merged 1 commit intomoby:masterfrom
robmry:46329_internal_resolver_ipv6_upstream

robmry commented Mar 6, 2024 •

edited

Loading

Uh oh!

corhere commented Mar 6, 2024

Uh oh!

neersighted left a comment

Uh oh!

corhere left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

robmry commented Mar 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

corhere commented Mar 6, 2024

Uh oh!

neersighted left a comment

Choose a reason for hiding this comment

Uh oh!

corhere left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

robmry commented Mar 6, 2024 •

edited

Loading