Docker container not reachable, bad TCP checksums #16841
Description
Hi,
We are unable to reach the ports n the container directly or mapped.
INFO
Docker version 1.5.0, build a8a31ef
Debian 7 backported kernel 3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.7-ckt9-3~deb8u1~bpo70+1 (2015-04-27) x86_64 GNU/Linux
Containers: 14
Images: 29
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 57
Execution Driver: native-0.2
Kernel Version: 3.16.0-0.bpo.4-amd64
Operating System: Debian GNU/Linux 7 (wheezy)
CPUs: 4
Total Memory: 7.815 GiB
Name: xxxxx
ID: PRMV:P7KM:5U6J:PYGZ:I3LQ:QWSB:ANFJ:VA23:URM4:JQOG:7TLM:4PTI
WARNING: No memory limit support
WARNING: No swap limit support
This is our additional settings:
docker0 Link encap:Ethernet HWaddr 56:84:7a:fe:97:99
inet addr:172.17.42.1 Bcast:0.0.0.0 Mask:255.255.0.0
inet6 addr: fe80::5484:7aff:fefe:9799/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:26765 errors:0 dropped:0 overruns:0 frame:0
TX packets:30835 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2173785 (2.0 MiB) TX bytes:63950837 (60.9 MiB)
vethc356415 Link encap:Ethernet HWaddr 9e:e7:da:08:87:c6
inet6 addr: fe80::9ce7:daff:fe08:87c6/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:200 errors:0 dropped:0 overruns:0 frame:0
TX packets:147 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:12828 (12.5 KiB) TX bytes:9398 (9.1 KiB)
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 xxxxxxxxxxxx 0.0.0.0 UG 0 0 0 eth0
xxxxxxxxxx 0.0.0.0 255.255.255.240 U 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
iptables -nL DOCKER
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 172.17.0.20 tcp dpt:22
ACCEPT tcp -- 0.0.0.0/0 172.17.0.20 tcp dpt:80
The running container with the interface vethc356415 is a Discourse container. These are its host and network settings:
"HostConfig": {
"Binds": [
"/var/discourse/shared/standalone:/shared",
"/var/discourse/shared/standalone/log/var-log:/var/log"
],
"CapAdd": null,
"CapDrop": null,
"ContainerIDFile": "",
"Devices": [],
"Dns": null,
"DnsSearch": null,
"ExtraHosts": null,
"IpcMode": "",
"Links": null,
"LxcConf": [],
"NetworkMode": "bridge",
"PidMode": "",
"PortBindings": {
"22/tcp": [
{
"HostIp": "",
"HostPort": "2222"
}
],
"80/tcp": [
{
"HostIp": "",
"HostPort": "10001"
}
]
},
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"RestartPolicy": {
"MaximumRetryCount": 0,
"Name": "always"
},
"SecurityOpt": null,
"VolumesFrom": null
},
"NetworkSettings": {
"Bridge": "docker0",
"Gateway": "172.17.42.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.20",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"LinkLocalIPv6Address": "fe80::42:acff:fe11:14",
"LinkLocalIPv6PrefixLen": 64,
"MacAddress": "02:42:ac:11:00:14",
"PortMapping": null,
"Ports": {
"22/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "2222"
}
],
"80/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "10001"
}
]
}
},
From inside of the container we are able to ping websites:
ping google.com
PING google.com (62.168.125.50) 56(84) bytes of data.
64 bytes from 62.168.125.50: icmp_seq=1 ttl=59 time=0.486 ms
64 bytes from 62.168.125.50: icmp_seq=2 ttl=59 time=0.520 ms
We are also able to get the index page of Discourse with wget (eg. doing a wget localhost from inside of container).
The problem is when we try to reach the container from outside (eg. from the Docker host):
$ wget 172.17.0.20
or
$ wget localhost:10001
does not work and hangs. When we tried sniffing the traffic on docker0 interface we found out that there is a TCP checksum problem:
# tcpdump -i docker0 -vvv
tcpdump: listening on docker0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:19:11.098052 IP (tos 0x0, ttl 64, id 30762, offset 0, flags [DF], proto TCP (6), length 60)
172.17.42.1.51633 > 172.17.0.20.http: Flags [S], cksum 0xcc74 (correct), seq 2431285397, win 29200, options [mss 1460,sackOK,TS val 3003252927 ecr 0,nop,wscale 7], length 0
11:19:11.098127 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.17.0.20.http > 172.17.42.1.51633: Flags [S.], cksum 0x8266 (incorrect -> 0x26fb), seq 449304527, ack 2431285398, win 28960, options [mss 1460,sackOK,TS val 3003252927 ecr 3003252927,nop,wscale 7], length 0
11:19:12.095536 IP (tos 0x0, ttl 64, id 30763, offset 0, flags [DF], proto TCP (6), length 60)
172.17.42.1.51633 > 172.17.0.20.http: Flags [S], cksum 0xcb7a (correct), seq 2431285397, win 29200, options [mss 1460,sackOK,TS val 3003253177 ecr 0,nop,wscale 7], length 0
11:19:12.095613 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.17.0.20.http > 172.17.42.1.51633: Flags [S.], cksum 0x8266 (incorrect -> 0x2601), seq 449304527, ack 2431285398, win 28960, options [mss 1460,sackOK,TS val 3003253177 ecr 3003252927,nop,wscale 7], length 0
11:19:13.303543 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.17.0.20.http > 172.17.42.1.51633: Flags [S.], cksum 0x8266 (incorrect -> 0x24d3), seq 449304527, ack 2431285398, win 28960, options [mss 1460,sackOK,TS val 3003253479 ecr 3003252927,nop,wscale 7], length 0
11:19:14.099543 IP (tos 0x0, ttl 64, id 30764, offset 0, flags [DF], proto TCP (6), length 60)
172.17.42.1.51633 > 172.17.0.20.http: Flags [S], cksum 0xc985 (correct), seq 2431285397, win 29200, options [mss 1460,sackOK,TS val 3003253678 ecr 0,nop,wscale 7], length 0
11:19:14.099625 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.17.0.20.http > 172.17.42.1.51633: Flags [S.], cksum 0x8266 (incorrect -> 0x240c), seq 449304527, ack 2431285398, win 28960, options [mss 1460,sackOK,TS val 3003253678 ecr 3003252927,nop,wscale 7], length 0
11:19:15.903555 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.17.0.20.http > 172.17.42.1.51607: Flags [S.], cksum 0x8266 (incorrect -> 0x0eeb), seq 1342518319, ack 2198677968, win 28960, options [mss 1460,sackOK,TS val 3003254129 ecr 3003244349,nop,wscale 7], length 0
Seems like the docker container sends packets that are dropped because of bad checksums (?). We tried to disable any kind of offloading on the vethc356415:
ethtool -K vethc356415 gso off lro off tso off
ethtool --offload vethc356415 tx off rx off
with no results. Right now we are stuck. Is this an issue or a configuration problem ?