Repository navigation
Description
Title: DNS resolution broken on IPv6-only clusters with Docker: enable_network_magic() is IPv4-only
What happened:
On an IPv6-only cluster (ipFamily: ipv6) running on Docker, CoreDNS pods cannot resolve external names. The node's /etc/resolv.conf contains nameserver 172.18.0.1 (the IPv4 default gateway), which is unreachable from IPv6-only pods. CoreDNS logs show:
[ERROR] plugin/errors: 2 ghcr.io. AAAA: read udp [fd00:10:244::43]:59624->[172.18.0.1]:53: connect: network is unreachable
The root cause is in enable_network_magic() in the node entrypoint script. It replaces Docker's embedded DNS (127.0.0.11) with the IPv4 default gateway:
docker_host_ip=$(ip -4 route show default | cut -d' ' -f3)This is hardcoded to IPv4. On an IPv6-only cluster, the resulting nameserver (172.18.0.1) is unreachable from pods that only have IPv6 addresses.
Note: even if the entrypoint used ip -6 route show default to get the IPv6 gateway (fc00:f853:ccd:e793::1), DNS still wouldn't work because Docker's DNS proxy does not listen on the IPv6 bridge gateway address (see moby/moby#41651). So the entrypoint fix alone is not sufficient — but it is a prerequisite for any future Docker fix to work.
What you expected to happen:
CoreDNS pods on an IPv6-only cluster should be able to forward DNS queries to an upstream resolver. The entrypoint should configure at least one IPv6-reachable nameserver in /etc/resolv.conf when the cluster uses ipFamily: ipv6.
How to reproduce it (as minimally and precisely as possible):
cat <<EOF | kind create cluster --name test --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
ipFamily: ipv6
EOF
# Check resolv.conf — only has IPv4 nameserver
docker exec test-control-plane cat /etc/resolv.conf
# nameserver 172.18.0.1
# The original Docker resolv.conf had 127.0.0.11
docker exec test-control-plane cat /etc/resolv.conf.original
# nameserver 127.0.0.11
# CoreDNS pods can't reach the IPv4 nameserver — verify from a debug pod
kubectl run -it --rm debug --image=alpine --restart=Never -- nslookup google.com
# ;; connection timed out; no servers could be reached
kind delete cluster --name testAnything else we need to know?:
The entrypoint's enable_network_magic() function is entirely IPv4-centric:
- Uses
ip -4 route show defaultto find the host IP - Only patches
iptables(notip6tables) - Only resolves
host.docker.internalviagetent ahostsv4
On Podman this works by accident: Podman's aardvark-dns puts its own IPv6 nameserver in /etc/resolv.conf (not 127.0.0.11), so the sed replacement is a no-op and the working IPv6 nameserver survives.
There is also a Docker-side issue: Docker's DNS proxy only listens on the IPv4 bridge gateway, not the IPv6 one (moby/moby#41651, open since 2020). Even if kind fixed the entrypoint to use the IPv6 gateway, there's currently nothing listening on port 53 there. A complete fix requires both kind and Docker changes.
As a workaround, running a socat DNS proxy on the node bridges the gap:
# Get the node's IPv6 address
NODE_IPV6=$(docker inspect test-control-plane -f '{{.NetworkSettings.Networks.kind.GlobalIPv6Address}}')
# Install socat and start DNS proxies (UDP + TCP)
docker exec test-control-plane bash -c "
apt-get update -qq >/dev/null 2>&1 && apt-get install -y -qq socat >/dev/null 2>&1
socat UDP6-LISTEN:53,bind=[${NODE_IPV6}],fork,reuseaddr UDP4:172.18.0.1:53 &
socat TCP6-LISTEN:53,bind=[${NODE_IPV6}],fork,reuseaddr TCP4:172.18.0.1:53 &
"
# Patch CoreDNS to forward to the node's IPv6 address instead of /etc/resolv.conf
kubectl get cm coredns -n kube-system -o json \
| sed "s|forward . /etc/resolv.conf|forward . ${NODE_IPV6}|" \
| kubectl apply -f -
kubectl rollout restart deployment/coredns -n kube-systemEnvironment:
- kind version: v0.31.0 (also reproduced with v0.29.0)
- Runtime info: Docker 29.3.0
- Kubernetes version: v1.34.0
Activity
matejvasek commentedon May 10, 2026
possibly related: #3114
BenTheElder commentedon May 11, 2026
Docker has previously stated they won't do this. So to support pure-single stack we'll have to do something like the socat bridge.
cc @aojea we discussed this before but I don't think a commitment to a particular solution materialized.
I think generally we will need to run some kind of ideally very minimal proxy on the nodes and also leverage it to drop the iptables hackery.
matejvasek commentedon May 14, 2026
I can understand that they do not wnat run DNS resolver on the loopback (
::1) address.But it does not run even on the
Gateway(i.e. some ULA address typically).BenTheElder commentedon May 14, 2026
Yeah, unfortunately it didn't seem like the project has any plans for this moby/moby#41651 (comment)
And even if they do, they'd like us to not depend on the iptables bits (understandable ... though we would probably just use our own iptables/nftables rules if it was a normal predictable socket).
I'm open to fixing this on the
kindend by running a proxy service on the node, I'm just concerned about keeping it maintainable (bugs, security fixes) and cheap (complexity, cpu/memory, latency, image size) and a little buried at the moment. Looking for @aojea's thoughts as well.It will be a bit ridiculous going from coreDNS => kind-node-proxy => docker socket / dockerd => host resolver but ... oh well, it already is a bit.
matejvasek commentedon May 14, 2026
I just checked now with a testing container and it appears that moby does not serve DNS at the gateway at all. Strange I would swear it was there before at least on IPv4 gateway IP.
matejvasek commentedon May 14, 2026
But in the KinD control plane container it works (querying gateway IP (e.g.
172.18.0.1)). Is that KinD modification?matejvasek commentedon May 14, 2026
The proxy could be simple Go program probably. If size is of essence it could be part of an existing binary as sub-command or as multi-call (dispatch on
arg[0]).matejvasek commentedon May 14, 2026
Oh that's probably in the iptables magic.
matejvasek commentedon May 14, 2026
It was little bit confusing because
podmandoes in fact serve DNS on the gateway IPs. Apparently moby does not do that at all.aojea commentedon May 19, 2026
I started that branch some time ago but stopped, there were certain things I didn't have time to followup:
route_localnet=1only works for IPv4, so local generated packets are able to be DNATed to localhost addresses in IPv4 but not IPb6, but this can be overcome in a different way, some thoughts:kind/images/base/files/usr/local/bin/entrypoint
Lines 446 to 456 in 71f3111
The proxy thing is simple, I asked copilot to clean my code and publish as a standalone proxy in https://github.com/aojea/dnsproxy64 , this is the same as
with the difference that the udp proxy is aware of the UDP/DNS session so it can track request/responses. We can also modify the golang code to open the listening sockets with IP_TRANSPARENT or IP_FREEBIND, I already have this working in kindnet for a long time for the same purpose https://github.com/search?q=repo%3Akubernetes-sigs%2Fkindnet+IP_TRANSPARENT&type=code .
We can ship the userspace proxy and inject in the image to start with a systemd unit per example.
The proxy can do "more magic" and also use the systemd socket activation directly https://github.com/coreos/go-systemd#socket-activation and scan the local sockets to find the docker embedded proxy, that will leave us with the detection of IPv6 only clusters and the configuration of the systemd socket unit.
ygao-g commentedon Aug 31, 2026
Confirming this on
ipFamily: ipv6with Docker, and adding a lighter-weightworkaround. Worth stating for anyone debugging it: the pod side is fine and IPv6
egress from pods works, so no NAT64 is involved — the only broken thing is which
resolver CoreDNS was handed.
Instead of the socat bridge, our cluster-creation script patches the Corefile to
an explicit IPv6 upstream:
Nothing installed into the node, no background process, but it costs the host's
resolver: split-horizon names stop resolving from pods, and Docker's embedded DNS
goes with it, so a local registry container's name starts returning NXDOMAIN and
needs its own
hostsblock. Both are arguments for the node-side proxy, whichkeeps the host's view intact. Fine tradeoff for a dev loop, wrong one for kind.