Skip to content

DNS resolution broken on IPv6-only clusters with Docker: enable_network_magic() is IPv4-only #4152

Open

Description

@matejvasek

Title: DNS resolution broken on IPv6-only clusters with Docker: enable_network_magic() is IPv4-only

What happened:

On an IPv6-only cluster (ipFamily: ipv6) running on Docker, CoreDNS pods cannot resolve external names. The node's /etc/resolv.conf contains nameserver 172.18.0.1 (the IPv4 default gateway), which is unreachable from IPv6-only pods. CoreDNS logs show:

[ERROR] plugin/errors: 2 ghcr.io. AAAA: read udp [fd00:10:244::43]:59624->[172.18.0.1]:53: connect: network is unreachable

The root cause is in enable_network_magic() in the node entrypoint script. It replaces Docker's embedded DNS (127.0.0.11) with the IPv4 default gateway:

docker_host_ip=$(ip -4 route show default | cut -d' ' -f3)

This is hardcoded to IPv4. On an IPv6-only cluster, the resulting nameserver (172.18.0.1) is unreachable from pods that only have IPv6 addresses.

Note: even if the entrypoint used ip -6 route show default to get the IPv6 gateway (fc00:f853:ccd:e793::1), DNS still wouldn't work because Docker's DNS proxy does not listen on the IPv6 bridge gateway address (see moby/moby#41651). So the entrypoint fix alone is not sufficient — but it is a prerequisite for any future Docker fix to work.

What you expected to happen:

CoreDNS pods on an IPv6-only cluster should be able to forward DNS queries to an upstream resolver. The entrypoint should configure at least one IPv6-reachable nameserver in /etc/resolv.conf when the cluster uses ipFamily: ipv6.

How to reproduce it (as minimally and precisely as possible):

cat <<EOF | kind create cluster --name test --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: ipv6
EOF

# Check resolv.conf — only has IPv4 nameserver
docker exec test-control-plane cat /etc/resolv.conf
# nameserver 172.18.0.1

# The original Docker resolv.conf had 127.0.0.11
docker exec test-control-plane cat /etc/resolv.conf.original
# nameserver 127.0.0.11

# CoreDNS pods can't reach the IPv4 nameserver — verify from a debug pod
kubectl run -it --rm debug --image=alpine --restart=Never -- nslookup google.com
# ;; connection timed out; no servers could be reached

kind delete cluster --name test

Anything else we need to know?:

The entrypoint's enable_network_magic() function is entirely IPv4-centric:

  • Uses ip -4 route show default to find the host IP
  • Only patches iptables (not ip6tables)
  • Only resolves host.docker.internal via getent ahostsv4

On Podman this works by accident: Podman's aardvark-dns puts its own IPv6 nameserver in /etc/resolv.conf (not 127.0.0.11), so the sed replacement is a no-op and the working IPv6 nameserver survives.

There is also a Docker-side issue: Docker's DNS proxy only listens on the IPv4 bridge gateway, not the IPv6 one (moby/moby#41651, open since 2020). Even if kind fixed the entrypoint to use the IPv6 gateway, there's currently nothing listening on port 53 there. A complete fix requires both kind and Docker changes.

As a workaround, running a socat DNS proxy on the node bridges the gap:

# Get the node's IPv6 address
NODE_IPV6=$(docker inspect test-control-plane -f '{{.NetworkSettings.Networks.kind.GlobalIPv6Address}}')

# Install socat and start DNS proxies (UDP + TCP)
docker exec test-control-plane bash -c "
  apt-get update -qq >/dev/null 2>&1 && apt-get install -y -qq socat >/dev/null 2>&1
  socat UDP6-LISTEN:53,bind=[${NODE_IPV6}],fork,reuseaddr UDP4:172.18.0.1:53 &
  socat TCP6-LISTEN:53,bind=[${NODE_IPV6}],fork,reuseaddr TCP4:172.18.0.1:53 &
"

# Patch CoreDNS to forward to the node's IPv6 address instead of /etc/resolv.conf
kubectl get cm coredns -n kube-system -o json \
  | sed "s|forward . /etc/resolv.conf|forward . ${NODE_IPV6}|" \
  | kubectl apply -f -
kubectl rollout restart deployment/coredns -n kube-system

Environment:

  • kind version: v0.31.0 (also reproduced with v0.29.0)
  • Runtime info: Docker 29.3.0
  • Kubernetes version: v1.34.0

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug.
    on May 9, 2026
  2. matejvasek commented on May 10, 2026

    @matejvasek
    Author

    possibly related: #3114

  3. BenTheElder commented on May 11, 2026

    @BenTheElder
    Member

    There is also a Docker-side issue: Docker's DNS proxy only listens on the IPv4 bridge gateway, not the IPv6 one (moby/moby#41651, open since 2020). Even if kind fixed the entrypoint to use the IPv6 gateway, there's currently nothing listening on port 53 there. A complete fix requires both kind and Docker changes.

    Docker has previously stated they won't do this. So to support pure-single stack we'll have to do something like the socat bridge.

    cc @aojea we discussed this before but I don't think a commitment to a particular solution materialized.

    I think generally we will need to run some kind of ideally very minimal proxy on the nodes and also leverage it to drop the iptables hackery.

  4. matejvasek commented on May 14, 2026

    @matejvasek
    Author

    I can understand that they do not wnat run DNS resolver on the loopback (::1) address.

    But it does not run even on the Gateway (i.e. some ULA address typically).

  5. BenTheElder commented on May 14, 2026

    @BenTheElder
    Member

    Yeah, unfortunately it didn't seem like the project has any plans for this moby/moby#41651 (comment)

    And even if they do, they'd like us to not depend on the iptables bits (understandable ... though we would probably just use our own iptables/nftables rules if it was a normal predictable socket).

    I'm open to fixing this on the kind end by running a proxy service on the node, I'm just concerned about keeping it maintainable (bugs, security fixes) and cheap (complexity, cpu/memory, latency, image size) and a little buried at the moment. Looking for @aojea's thoughts as well.

    It will be a bit ridiculous going from coreDNS => kind-node-proxy => docker socket / dockerd => host resolver but ... oh well, it already is a bit.

  6. matejvasek commented on May 14, 2026

    @matejvasek
    Author

    I just checked now with a testing container and it appears that moby does not serve DNS at the gateway at all. Strange I would swear it was there before at least on IPv4 gateway IP.

  7. matejvasek commented on May 14, 2026

    @matejvasek
    Author

    But in the KinD control plane container it works (querying gateway IP (e.g. 172.18.0.1)). Is that KinD modification?

  8. matejvasek commented on May 14, 2026

    @matejvasek
    Author

    The proxy could be simple Go program probably. If size is of essence it could be part of an existing binary as sub-command or as multi-call (dispatch on arg[0]).

  9. matejvasek commented on May 14, 2026

    @matejvasek
    Author

    But in the KinD control plane container it works (querying gateway IP (e.g. 172.18.0.1)). Is that KinD modification?

    Oh that's probably in the iptables magic.

  10. matejvasek commented on May 14, 2026

    @matejvasek
    Author

    It was little bit confusing because podman does in fact serve DNS on the gateway IPs. Apparently moby does not do that at all.

  11. aojea commented on May 19, 2026

    @aojea
    Contributor

    I'm open to fixing this on the kind end by running a proxy service on the node, I'm just concerned about keeping it maintainable (bugs, security fixes) and cheap (complexity, cpu/memory, latency, image size) and a little buried at the moment. Looking for @aojea's thoughts as well.

    I started that branch some time ago but stopped, there were certain things I didn't have time to followup:

    • route_localnet=1 only works for IPv4, so local generated packets are able to be DNATed to localhost addresses in IPv4 but not IPb6, but this can be overcome in a different way, some thoughts:
    • detection of IPv6 ONLY clusters during node startup, so we put the right address on resolv.conf
      enable_network_magic(){
      # well-known docker embedded DNS is at 127.0.0.11:53
      local docker_embedded_dns_ip='127.0.0.11'
      # first we need to detect an IP to use for reaching the docker host
      local docker_host_ip
      docker_host_ip="$( (head -n1 <(timeout 5 getent ahostsv4 'host.docker.internal') | cut -d' ' -f1) || true)"
      # if the ip doesn't exist or is a loopback address use the default gateway
      if [[ -z "${docker_host_ip}" ]] || [[ $docker_host_ip =~ ^127\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
      docker_host_ip=$(ip -4 route show default | cut -d' ' -f3)
      fi

    The proxy thing is simple, I asked copilot to clean my code and publish as a standalone proxy in https://github.com/aojea/dnsproxy64 , this is the same as

      socat UDP6-LISTEN:53,bind=[${NODE_IPV6}],fork,reuseaddr UDP4:172.18.0.1:53 &
      socat TCP6-LISTEN:53,bind=[${NODE_IPV6}],fork,reuseaddr TCP4:172.18.0.1:53 &
    

    with the difference that the udp proxy is aware of the UDP/DNS session so it can track request/responses. We can also modify the golang code to open the listening sockets with IP_TRANSPARENT or IP_FREEBIND, I already have this working in kindnet for a long time for the same purpose https://github.com/search?q=repo%3Akubernetes-sigs%2Fkindnet+IP_TRANSPARENT&type=code .

    We can ship the userspace proxy and inject in the image to start with a systemd unit per example.

    The proxy can do "more magic" and also use the systemd socket activation directly https://github.com/coreos/go-systemd#socket-activation and scan the local sockets to find the docker embedded proxy, that will leave us with the detection of IPv6 only clusters and the configuration of the systemd socket unit.

  12. added
    kind/featureCategorizes issue or PR as related to a new feature.
    on May 19, 2026
  13. ygao-g commented on Aug 31, 2026

    @ygao-g
    Contributor

    Confirming this on ipFamily: ipv6 with Docker, and adding a lighter-weight
    workaround. Worth stating for anyone debugging it: the pod side is fine and IPv6
    egress from pods works, so no NAT64 is involved — the only broken thing is which
    resolver CoreDNS was handed.

    Instead of the socat bridge, our cluster-creation script patches the Corefile to
    an explicit IPv6 upstream:

    forward . /etc/resolv.conf  ->  forward . 2001:4860:4860::8888 2001:4860:4860::8844
    

    Nothing installed into the node, no background process, but it costs the host's
    resolver: split-horizon names stop resolving from pods, and Docker's embedded DNS
    goes with it, so a local registry container's name starts returning NXDOMAIN and
    needs its own hosts block. Both are arguments for the node-side proxy, which
    keeps the host's view intact. Fine tradeoff for a dev loop, wrong one for kind.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.kind/featureCategorizes issue or PR as related to a new feature.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions