Docker for Mac 1.12.0 TCP DNS queries fail #24344
Description
Since Docker for Mac 1.12.0, fragmented DNS queries cause DNS resolution to fail when trying to resolve via 192.168.65.1:53 because the resolver can not connect via TCP to said IP.
Output of docker version:
Client:
Version: 1.12.0-rc3
API version: 1.24
Go version: go1.6.2
Git commit: 91e29e8
Built: Sat Jul 2 00:09:24 2016
OS/Arch: darwin/amd64
Experimental: true
Server:
Version: 1.12.0-rc3
API version: 1.24
Go version: go1.6.2
Git commit: 876f3a7
Built: Tue Jul 5 02:20:13 2016
OS/Arch: linux/amd64
Experimental: true
Output of docker info:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.0-rc3
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 0
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: overlay bridge host null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.14-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.954 GiB
Name: moby
ID: 7NQP:NH7Q:G2HH:3TIS:3L6P:EPES:SS5I:YEIU:7XEB:ZU37:UX27:6RTI
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 17
Goroutines: 29
System Time: 2016-07-05T20:39:53.984107317Z
EventsListeners: 1
No Proxy: *.local, 169.254/16
Username: gavinmroy
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
127.0.0.0/8
Additional environment details (AWS, VirtualBox, physical, etc.):
This only happens when the DNS result is truncated, as can happen with auth.docker.io.
Please see https://forums.docker.com/t/auth-docker-io-on-192-168-65-1-53-no-such-host/16801/4 - It's worth noting that there are multiple posts in the Docker for Mac forum about this issue.
I was hoping the forums were visible enough for reports there, but I'm guessing I should have gone here first.
Steps to reproduce the issue:
docker pull alpine
Describe the results you received:
Docker fails to pull the image due to DNS failure:
$ docker pull alpine
Using default tag: latest
Error response from daemon: Get https://registry-1.docker.io/v2/library/alpine/manifests/latest1: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io: dial tcp: lookup auth.docker.io on 192.168.65.1:53: no such host
Describe the results you expected:
Docker pulls the image.
Additional information you deem important (e.g. issue happens only occasionally):
- This has something to do with the EDNS record sizes returned for auth.docker.io and the new Alpine 3.4 based moby image. Previous versions of moby on my network do not have the issue
- The error that I get is a failure to connect to the embedded DNS server in docker via TCP
On my network, I am getting EDNS truncated responses back which should cause the DNS resolver to failover to TCP queries - Alpine 3.4 does not have a problem with truncated issues on its own, so I'm guessing that the main issue here is lack of full EDNS/TCP support in the embedded docker DNS server, which seems to cause an issue when issuing a docker login/pull or anything else asking to resolve auth.docker.io since the record seems to be longer than the docker embedded DNS server can handle.
- When I am in the moby instance, I can not resolve auth.docker.io with the default settings:
moby:~# nslookup auth.docker.io
;; Truncated, retrying in TCP mode.
;; Connection to 192.168.65.1#53(192.168.65.1) for auth.docker.io failed: connection refused.
If I change the DNS server to 8.8.8.8, I do not have a problem:
moby:~# cat /etc/resolv.conf
search local
nameserver 8.8.8.8
moby:~# nslookup auth.docker.io
Server: 8.8.8.8
Address: 8.8.8.8#53
Non-authoritative answer:
auth.docker.io canonical name = elb-registry.us-east-1.aws.dckr.io.
elb-registry.us-east-1.aws.dckr.io canonical name = us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com.
Name: us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 52.73.165.108
Name: us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 52.203.219.86
Name: us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 52.71.80.248
Name: us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 54.172.251.194
Name: us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 52.71.245.229
Name: us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 54.164.225.120
Name: us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 52.72.94.105
Name: us-east-1-elbregis-10fucsvj1tcgy-133821800.us-east-1.elb.amazonaws.com
Address: 54.174.255.71
I'm not quite sure why the results are being truncated as the resolver for my laptop has a max-udp-size and edns-udp-size of 4096. What that is material to the cause of the truncated EDNS UDP packets, it is not as much of a problem as Docker's embedded DNS server running at 192.168.65.1 doesn't seem to be listening for DNS queries via TCP.
I would assume that if 192.168.65.1 was able to resolve DNS via TCP (which is the fallback behavior when a UDP DNS response returns a fragmented reply), it would be connectable on port 53:
moby:~# exit
Welcome to Moby alpha
Kernel 4.4.14-moby on an x86_64 (/dev/ttyS0)
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
moby login: root
Welcome to the Moby alpha, based on Alpine Linux.
moby:~# telnet 192.168.65.1 53
telnet: can't connect to remote host (192.168.65.1): Connection refused