Repository navigation
Feature request: IPv6-enabled embedded DNS server #41651
Description
Feature request: Docker's embedded DNS server is available over IPv6 in addition to the already-existing IPv4. Docker includes the IPv6 address of its embedded DNS name server when generating a containers /etc/resolv.conf.
Following the running example from below, the file's contents would be:
nameserver 127.0.0.11 # Docker's embedded DNS server, accessible over IPv4
nameserver fedc::1 # Docker's embedded DNS server, accessible over IPv6
options ndots:0
Of course, the actual local IPv6 address needs determining.
Background: Docker's embedded DNS server is hard-coded to be available at 127.0.0.11 over IPv4 only. This fact is encoded into containers' generated /etc/resolv.conf files. In our environments, the Docker daemon is configured in /etc/docker/daemon.json with the ISP's upstream IPv4 and IPv6 nameserver addresses. When Docker generates /etc/resolv.conf for a new container, the contents becomes:
nameserver 127.0.0.11 # Docker's embedded DNS server, accessible over IPv4
nameserver 2a00:0123::1 # Upstream ISP name server, accessible over IPv6
options ndots:0
Crucially, notice that the IPv4 nameserver is Docker's embedded DNS server, and that the IPv6 nameserver points to the ISP's upstream server. (Clarifying commentary trailing the nameserver directives was added by me.) This state of affairs causes sporadic run-time faults when resolving the IP addresses of peer containers by their names.
Discussion: When an Alpine container attempts to resolve the IP address by the name of one of its peer containers, Alpine's MUSL performs parallel requests, sending A and AAAA queries over UDP to each of the configured nameservers all at once, and taking the first response.
Thus, when for example Alpine container logic attempts to connect to container db, the software first attempts to resolve the hostname db into an IP address in order to connect to it. MUSL does this by sending out a total of 4 queries in parallel, based on the above /etc/resolv.conf:
- UDP request over IPv4 to Docker's embedded DNS server for the A record for
db.. The Docker daemon is of course authoritative fordb.and responds with its IP address. - UDP request over IPv4 to Docker's embedded DNS server for the AAAA record for
db.. Again, Docker responds affirmatively. - UDP request over IPv6 to the ISP's nameserver for the A record for
db.. Obviously the ISP is oblivious as to a host nameddb.run by one of its customers in a privacy-concealing environment; it returnsNXDOMAIN. - UDP request over IPv6 to the ISP's nameserver for the AAAA record for
db.. Again, the ISP returnsNXDOMAIN.
You can observe this using tcpdump -i any -n port 53 & followed by ping db -c 1 in a container setup on a bridge network with IPv6 enabled.
Now, due to network latency and myriad other reasons, the logic container receives Docker's IPv4-based embedded DNS responses generally before those of the ISP's IPv6 responses. Name resolution succeeds according to MUSL, and processing continues down its happy path. But every now and then the ISP's response packets arrive before those of Docker, and MUSL blissfully accepts the NXDOMAIN responses and reports name resolution failure back up to the software, ultimately causing a processing fault.
If Docker offered it's embedded DNS server over IPv6 as a first-class citizen alongside IPv4, name resolution would function properly.
Our usage environment:
- Docker version 19.03.13.
- Containers connected to custom bridge network with IPv6 enabled.
- Plenty of containers are based on Alpine Linux, whose MUSL performs DNS name resolutions by making parallel requests, one to each configured nameserver; this is MUSLs only mode of operation.
- Some services are at various stages of being IPv6 native (can only be resolved and accessed using IPv6).
/etc/docker/daemon.jsonspecifically configures the Docker daemon with the ISP's upstream DNS servers according to location:
{
"dns": [
"123.4.5.6", # IPv4 address of the ISP's DNS server
"2a00:0123::1" # IPv6 address of the ISP's DNS server
],
...
}
Cheers!
Activity
snowsnoot commentedon Dec 14, 2021
Pretty surprising Docker claims to support IPv6 in 2021 without supporting IPv6 on the embedded DNS server. This causes issues on my user-defined network as the containers are unable to resolve their internal hostnames on the upstream IPv6 DNS server.
cnb2412 commentedon Jan 18, 2022
Running into this very issue on an IPv6-only Server. Having a method to configure the embedded DNS to resolve queries via IPv6 would really be helpful.
g00g1 commentedon Mar 10, 2023
I am using dual-stack networking for containers and running into the same issue.
The only workaround I could imagine is creating firewall rule to block any 53/udp communications for IPv6 prefixes used by Docker containers, which obviously is too bad but solves my issue.
+1 for possibility to configure embedded DNS for IPv6 network.
cleverer commentedon Apr 9, 2023
Wow just found this issue. I have been running a fully dual-stack docker-compose based server, and this explains a lot of random errors. Absolutely mind-boggling how this doesn't work out of the box… makes any docker-compose based dual-stack deployment basically useless.
akerouanton commentedon Apr 10, 2023
Hi, after looking at the code I think we can solve this issue without making docker embedded DNS resolver available through IPv6, at least for now.
Currently, docker doesn't treat IPv4 and IPv6
--dnsaddresses the same way. If you specify IPv4--dnsaddresses, the embedded DNS resolver use them to forward queries for unknown entries. If you specify IPv6--dnsaddresses, the embedded DNS resolver ignores them and they end up in your containers'/etc/resolv.conf.This weird behavior is a vestige of the time when IPv6 support was absolutely absent from Docker. I have a patch ready for treating IPv6
--dnsaddresses like IPv4 and I believe it works properly. I'll test it tomorrow and send a PR if it's conclusive.As for why the local embedded DNS server would take more time than a roundtrip to your ISP's IPv6 DNS, there're actually multiple plausible explainations: garbage collection, many concurrent DNS queries happening at the same time, libnetwork's controller being locked, etc... For the last two explainations, I think we can/will improve that.
On the feature request itself, kind seems to require it for IPv6-only clusters. So we'll have to support this feature at some point, but that's surely more work and I'm not sure yet when that will happen.
spinda commentedon May 2, 2023
@akerouanton Did that patch end up working out? Just ran into this today.
simonwiles commentedon Sep 25, 2023
Is there any solution to this issue at present? My dual-stack docker compose set up is just failing to resolve internal names seemingly at random because of this issue, and I'd really like to do something about it!
jdannenberg commentedon Sep 25, 2023
Do you really want to hear my workaround?
Based on the very useful information
I just changed every service to not use an alpine-based image. E.g. for zabbix I changed to the ubuntu flavoured / based image.
simonwiles commentedon Sep 25, 2023
Thanks :) I don't have that option easily and immediately available to me, so I'll probably go with something a bit hacky for now; would be great to know if this will be properly fixed at some point, though!
cleverer commentedon Feb 14, 2024
@akerouanton also wondering if your patch did go anywhere.Mmy docker-compose based setup has several services constantly flapping, as they can't resolve the linked containers (which are mostly the dbs). I'm really surprised to see the ipv6 nameservers still ending up in the resolv.conf.
This is a huge oversight by docker imho… but also easily fixable by just not writing that line, no?
cleverer commentedon Mar 21, 2024
Looks like this has been finally (partially) been addressed in docker v26: #47512
The host v6 dns is not mounted to the container anymore, so docker-network internal dns resolution should properly work for dual-stack setups.
neersighted commentedon Mar 21, 2024
I'm not sure what partially means as this should also work for single-stack setups; this issue should be resolved with v26.
cleverer commentedon Mar 21, 2024
I think the expectation of the ticket would be that the embedded docker dns proxy is also available over ipv6 inside the container. For my use case, having only a IPv4 docker proxy inside the container is good enough, hence I called it partially addressed…
neersighted commentedon Mar 21, 2024
Ah; you're referring to IPv6-only inside the container network. I would say that's a different ticket, and would be a natural consequence of support for IPv6-only containers, which is not yet implemented. I think we should probably refer back to #44923, and split off a separate ticket only if we decide single-stack doesn't encompass that entire ticket.
cleverer commentedon Mar 21, 2024
Reading through the original issue description, I think this issue is precisely about making the dns server listen on IPv6 insider the container network. However I'm not aware of all the open issues and am fine with just having it as part of #44923, since I'm already very happy my main inconvenience has been resolved :)
neersighted commentedon Mar 21, 2024
The root cause of the issue was the race with the upstream server; one possible solution was a dual-stack listener, but not the only one, in my opinion.
In any case, yeah, I'd rather have fewer issuers with a wider audience than many that are fragmented 👍
thaJeztah commentedon Mar 21, 2024
Following the discussion on #47442 (comment) (and comments below that), I'm not sure if and IPv6 loopback-only DNS (even for IPv6-only containers) will happen; at least it looks like there's some motivation there describing it may not be a realistic option.
fix: improve IPv6 support in local Kind cluster setup (#3693)