Better selection of DNS server by thaJeztah · Pull Request #41022 · moby/moby

thaJeztah · 2020-05-25T16:17:41Z

fixes / addresses 18.09 breaks containers name resolution for non default networks on systems with systemd-resolved #38243 18.09 breaks containers name resolution for non default networks on systems with systemd-resolved
fixes / addresses Wrong resolv.conf used on Ubuntu 19 (systemd-resolved enabled) #39978 Wrong resolv.conf used on Ubuntu 19 (systemd-resolved enabled)
fixes / addresses Container /etc/resolv.conf does not update when /run/systemd/resolve/resolv.conf changes docker/for-linux#889 Container /etc/resolv.conf does not update when /run/systemd/resolve/resolv.conf changes
fixes / addresses dockerd should use 127.0.0.53 when requested docker/for-linux#979 dockerd should use 127.0.0.53 when requested
relates to resolvconf: use /run/systemd/resolve/resolv.conf if systemd-resolved manages DNS libnetwork#2385 (comment)
relates to Private DNS Fails on 0.8 with systemd-resolved kubernetes-sigs/kind#1594 (comment)

Commit e353e7e (#37485) updated selection of the resolv.conf file to use in situations where systemd-resolvd is used as a resolver.

If a host uses systemd-resolvd, the system's /etc/resolv.conf file is
updated to set 127.0.0.53 as DNS, which is the local IP address for
systemd-resolvd. The DNS servers that are configured by the user will now
be stored in /run/systemd/resolve/resolv.conf, and systemd-resolvd acts
as a forwarding DNS for those.

Originally, Docker copied the DNS servers as configured in /etc/resolv.conf
as default DNS servers in containers, which failed to work if systemd-resolvd
is used (as 127.0.0.53 is not available inside the container's networking
namespace). To resolve this, e353e7e instead
detected if systemd-resolvd is in use, and in that case copied the "upstream"
DNS servers from the /run/systemd/resolve/resolv.conf configuration.

While this worked for most situations, it had some downsides, among which:

we're skipping systemd-resolvd altogether, which means that we cannot take
advantage of addition functionality provided by it (such as per-interface
DNS servers)
when updating DNS servers in the system's configuration, those changes were
not reflected in the container configuration, which could be problematic in
"developer" scenarios, when switching between networks.

This patch changes the way we select which resolv.conf to use as template
for the container's resolv.conf;

in situations where a custom network is attached to the container, and the
embedded DNS is available, we use /etc/resolv.conf unconditionally. If
systemd-resolvd is used, the embedded DNS forwards external DNS lookups to
systemd-resolvd, which in turn is responsible for forwarding requests to
the external DNS servers configured by the user.
if the container is running in "host mode" networking, we also use the
DNS server that's configured in /etc/resolv.conf. In this situation, no
embedded DNS server is available, but the container runs in the host's
networking namespace, and can use the same DNS servers as the host (which
could be systemd-resolvd or DNSMasq
if the container uses the default (bridge) network, no embedded DNS is
available, and the container has its own networking namespace. In this
situation we check if systemd-resolvd is used, in which case we skip
systemd-resolvd, and configure the upstream DNS servers as DNS for the
container. This situation is the same as is used currently, which means
that dynamically switching DNS servers won't be supported for these
containers.

Signed-off-by: Sebastiaan van Stijn [email protected]

- What I did

- How I did it

- How to verify it

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

Commit e353e7e updated selection of the `resolv.conf` file to use in situations where systemd-resolvd is used as a resolver. If a host uses `systemd-resolvd`, the system's `/etc/resolv.conf` file is updated to set `127.0.0.53` as DNS, which is the local IP address for systemd-resolvd. The DNS servers that are configured by the user will now be stored in `/run/systemd/resolve/resolv.conf`, and systemd-resolvd acts as a forwarding DNS for those. Originally, Docker copied the DNS servers as configured in `/etc/resolv.conf` as default DNS servers in containers, which failed to work if systemd-resolvd is used (as `127.0.0.53` is not available inside the container's networking namespace). To resolve this, e353e7e instead detected if systemd-resolvd is in use, and in that case copied the "upstream" DNS servers from the `/run/systemd/resolve/resolv.conf` configuration. While this worked for most situations, it had some downsides, among which: - we're skipping systemd-resolvd altogether, which means that we cannot take advantage of addition functionality provided by it (such as per-interface DNS servers) - when updating DNS servers in the system's configuration, those changes were not reflected in the container configuration, which could be problematic in "developer" scenarios, when switching between networks. This patch changes the way we select which resolv.conf to use as template for the container's resolv.conf; - in situations where a custom network is attached to the container, and the embedded DNS is available, we use `/etc/resolv.conf` unconditionally. If systemd-resolvd is used, the embedded DNS forwards external DNS lookups to systemd-resolvd, which in turn is responsible for forwarding requests to the external DNS servers configured by the user. - if the container is running in "host mode" networking, we also use the DNS server that's configured in `/etc/resolv.conf`. In this situation, no embedded DNS server is available, but the container runs in the host's networking namespace, and can use the same DNS servers as the host (which could be systemd-resolvd or DNSMasq - if the container uses the default (bridge) network, no embedded DNS is available, and the container has its own networking namespace. In this situation we check if systemd-resolvd is used, in which case we skip systemd-resolvd, and configure the upstream DNS servers as DNS for the container. This situation is the same as is used currently, which means that dynamically switching DNS servers won't be supported for these containers. Signed-off-by: Sebastiaan van Stijn <[email protected]>

thaJeztah · 2020-05-26T12:26:29Z

@arkodg @cpuguy83 PTAL (let me know if this approach makes sense)

thaJeztah · 2020-05-26T12:27:32Z

+		// namespace, so both `/etc/hosts` and `/etc/resolv.conf` should be the same
+		// as on the host itself. The container gets a copy of these files, but they
+		// may be symlinked, so resolve the original path first.
+		etcHosts, err := filepath.EvalSymlinks("/etc/hosts")


This is new; previously we didn't check if these were symlinked (which at least in case of /etc/resolv.conf may be the case for systems that were upgraded from older versions)

cpuguy83

LGTM

thaJeztah · 2020-05-28T18:54:29Z

@xinfengliu PTAL, perhaps you have thought as well if this makes sense 🤗

arkodg · 2020-05-28T23:21:42Z

thanks for the detailed code comments !
can we add tests for these 🤗 ?

xinfengliu · 2020-05-29T02:55:44Z

@xinfengliu PTAL, perhaps you have thought as well if this makes sense 🤗

@thaJeztah Sorry I'm not familiar with this part of codes, not able to comment.

thaJeztah · 2020-06-09T15:10:21Z

can we add tests for these 🤗 ?

Thinking what the best way is to test this, as we don't run the daemons with systemd inside the docker-in-docker containers; do you have suggestions for that?

arkodg

ya an integration test will be non trivial, approving this PR
thanks !

thaJeztah · 2020-06-25T11:56:38Z

@AkihiroSuda LGTY?

vincentbernat · 2020-08-07T09:12:18Z

Would it be possible this commit to also be pushed in 19.03 branch? There are more and more systemd-resolved based setup.

thaJeztah · 2020-08-10T10:36:24Z

@arkodg @cpuguy83 WDYT? Safe to backport? I recall we had some discussion about it, but don't recall where we ended up.

(I did notice there was a change that was discussed that somehow didn't make it in this PR 🤔 opened #41335 to address that)

cpuguy83 · 2020-08-10T23:44:03Z

It seems ok but also risky... I don't know...
Maybe after 20.0x is out and this has some real use?

thaJeztah · 2020-08-19T12:21:51Z

Opened a draft backport #41374, so that we can discuss/consider

thaJeztah added status/2-code-review area/networking Networking impact/changelog labels May 25, 2020

thaJeztah force-pushed the smarter_resolv branch from 95786eb to d2cb236 Compare May 25, 2020 16:19

thaJeztah force-pushed the smarter_resolv branch from d2cb236 to a5324d6 Compare May 25, 2020 16:21

thaJeztah marked this pull request as ready for review May 26, 2020 07:11

thaJeztah commented May 26, 2020

View reviewed changes

thaJeztah mentioned this pull request May 27, 2020

dockerd should use 127.0.0.53 when requested docker/for-linux#979

Open

3 tasks

cpuguy83 approved these changes May 28, 2020

View reviewed changes

thaJeztah mentioned this pull request Jun 8, 2020

DNS servers didn't change in container when DHCP changed them on the host #25618

Open

lieutdan13 approved these changes Jun 9, 2020

View reviewed changes

arkodg approved these changes Jun 9, 2020

View reviewed changes

AkihiroSuda approved these changes Jun 25, 2020

View reviewed changes

AkihiroSuda merged commit 3621812 into moby:master Jun 25, 2020

thaJeztah deleted the smarter_resolv branch June 25, 2020 14:01

thaJeztah added this to the 20.03.0 milestone Jul 3, 2020

thaJeztah mentioned this pull request Aug 10, 2020

daemon.setupPathsAndSandboxOptions() skip resolving symlinks #41335

Merged

thaJeztah mentioned this pull request Aug 19, 2020

[19.03 backport] Better selection of DNS server #41374

Closed

thaJeztah added the area/systemd label Aug 19, 2020

sluetze mentioned this pull request Dec 9, 2020

Docker version 20.10 breaks Harbor's connection to LDAP server goharbor/harbor#13718

Closed

mweinelt mentioned this pull request Dec 18, 2020

ndots option in hosts /etc/resolv.conf breaks container name resolution #41819

Open

wy65701436 mentioned this pull request Mar 9, 2021

Unable to download trivy vulnerability DB goharbor/harbor#14146

Closed

Luap99 mentioned this pull request Jun 3, 2021

Podman 2.2.1 adds an additional 127.0.1.1 to /etc/hosts breaking services binding to hostname when --net=host is specified containers/podman#10319

Closed

Ka0o0 mentioned this pull request Oct 7, 2021

Support different DNS resolver if host network is used moby/buildkit#2404

Closed

thaJeztah mentioned this pull request Sep 7, 2023

Container using network mode host does not get its resolv.conf updated when the host's resolv.conf is updated (using systemd-resolved) #46199

Open

lyoung-confluent mentioned this pull request Jun 9, 2024

Incorrect DNS options with host networking when buildkit is run as docker-container on a host with systemd-resolved moby/buildkit#5009

Open

Conversation

thaJeztah commented May 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thaJeztah commented May 26, 2020

Uh oh!

thaJeztah May 26, 2020

Choose a reason for hiding this comment

Uh oh!

cpuguy83 left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented May 28, 2020

Uh oh!

arkodg commented May 28, 2020

Uh oh!

xinfengliu commented May 29, 2020

Uh oh!

thaJeztah commented Jun 9, 2020

Uh oh!

arkodg left a comment

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Jun 25, 2020

Uh oh!

vincentbernat commented Aug 7, 2020

Uh oh!

thaJeztah commented Aug 10, 2020

Uh oh!

cpuguy83 commented Aug 10, 2020

Uh oh!

thaJeztah commented Aug 19, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

thaJeztah commented May 25, 2020 •

edited

Loading