Sometimes in one overlay network and only on one node. The container cannot access another containers in one overlay network. But I don't know how to reproduce this issue.
I use the tcpdump to catch the packages in overlay network namespace. Found the packages not be forward to other container's veth. But be forward to vxlan0.
tcpdump at connection issue:
/ # tcpdump -i vxlan0 -vv -nn icmp
tcpdump: listening on vxlan0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:48:18.750989 IP (tos 0x0, ttl 64, id 29257, offset 0, flags [DF], proto ICMP (1), length 84)
172.19.0.2 > 172.19.0.4: ICMP echo request, id 33536, seq 0, length 64
tcpdump at normal:
/# tcpdump -i vxlan0 -vv -nn icmp
tcpdump: listening on vxlan0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
Check the br0 forward table. The br0 bridge mac table's port is different with normal state:
/ # brctl showmacs br0
port no mac addr is local? ageing timer
1 02:42:ac:13:00:02 no 0.22
1 02:42:ac:13:00:04 no 11.51
1 02:42:ac:13:00:05 no 8.87
1 02:42:ac:13:00:06 no 3.21
1 02:42:ac:13:00:07 no 9.22
1 02:42:ac:13:00:08 no 3.86
1 02:42:ac:13:00:09 no 2.50
3 0a:9c:03:9c:2e:3e yes 0.00
7 0a:e7:50:45:5d:85 yes 0.00
6 32:b9:28:74:46:2e yes 0.00
9 6e:98:31:01:26:a4 yes 0.00
5 76:40:c5:a5:18:ca yes 0.00
2 86:22:54:09:f2:05 yes 0.00
4 c2:16:7c:9a:5c:a8 yes 0.00
1 c6:08:88:0b:da:3c yes 0.00
8 f2:35:65:d2:73:3a yes 0.00
/ # bridge fdb
0a:9c:03:9c:2e:3e dev veth1 master br0 permanent
02:42:ac:13:00:08 dev vxlan0 master br0
f2:35:65:d2:73:3a dev veth7 master br0 permanent
02:42:ac:13:00:02 dev vxlan0 master br0
c2:16:7c:9a:5c:a8 dev veth13 master br0 permanent
86:22:54:09:f2:05 dev veth0 master br0 permanent
c6:08:88:0b:da:3c dev vxlan0 master br0 permanent
02:42:ac:13:00:05 dev vxlan0 master br0
6e:98:31:01:26:a4 dev veth10 master br0 permanent
0a:e7:50:45:5d:85 dev veth6 master br0 permanent
76:40:c5:a5:18:ca dev veth18 master br0 permanent
32:b9:28:74:46:2e dev veth11 master br0 permanent
02:42:ac:13:00:04 dev vxlan0 master br0
02:42:ac:13:00:09 dev vxlan0 master br0
02:42:ac:13:00:06 dev vxlan0 master br0
02:42:ac:13:00:07 dev vxlan0 master br0
02:42:ac:13:00:02 dev vxlan0 dst 127.0.0.1 link-netnsid 0 self permanent
02:42:ac:13:00:04 dev vxlan0 dst 127.0.0.1 link-netnsid 0 self permanent
02:42:ac:13:00:05 dev vxlan0 dst 127.0.0.1 link-netnsid 0 self permanent
02:42:ac:13:00:06 dev vxlan0 dst 127.0.0.1 link-netnsid 0 self permanent
02:42:ac:13:00:07 dev vxlan0 dst 127.0.0.1 link-netnsid 0 self permanent
02:42:ac:13:00:08 dev vxlan0 dst 127.0.0.1 link-netnsid 0 self permanent
02:42:ac:13:00:09 dev vxlan0 dst 127.0.0.1 link-netnsid 0 self permanent
33:33:00:00:00:01 dev veth0 self permanent
01:00:5e:00:00:01 dev veth0 self permanent
33:33:00:00:00:01 dev veth1 self permanent
01:00:5e:00:00:01 dev veth1 self permanent
33:33:00:00:00:01 dev veth6 self permanent
01:00:5e:00:00:01 dev veth6 self permanent
33:33:00:00:00:01 dev veth7 self permanent
01:00:5e:00:00:01 dev veth7 self permanent
33:33:00:00:00:01 dev veth10 self permanent
01:00:5e:00:00:01 dev veth10 self permanent
33:33:00:00:00:01 dev veth11 self permanent
01:00:5e:00:00:01 dev veth11 self permanent
33:33:00:00:00:01 dev veth13 self permanent
01:00:5e:00:00:01 dev veth13 self permanent
33:33:00:00:00:01 dev veth18 self permanent
01:00:5e:00:00:01 dev veth18 self permanent
brctl showmacs br0 at normal state:
/ # brctl showmacs br0
port no mac addr is local? ageing timer
2 02:42:ac:13:00:02 no 0.32
4 02:42:ac:13:00:04 no 0.32
5 02:42:ac:13:00:05 no 0.79
6 02:42:ac:13:00:06 no 0.39
7 02:42:ac:13:00:07 no 0.39
8 02:42:ac:13:00:08 no 0.79
9 02:42:ac:13:00:09 no 0.50
3 0a:9c:03:9c:2e:3e yes 0.00
7 0a:e7:50:45:5d:85 yes 0.00
6 32:b9:28:74:46:2e yes 0.00
9 6e:98:31:01:26:a4 yes 0.00
5 76:40:c5:a5:18:ca yes 0.00
2 86:22:54:09:f2:05 yes 0.00
4 c2:16:7c:9a:5c:a8 yes 0.00
1 c6:08:88:0b:da:3c yes 0.00
8 f2:35:65:d2:73:3a yes 0.00
docker version and system info
# lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.2.1511 (Core)
Release: 7.2.1511
Codename: Core
# uname -a
Linux c81509ccbcb5d4d76bcdf2267f3fc3556-node1 3.10.0-514.6.2.el7.x86_64 #1 SMP Thu Feb 23 03:04:39 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
# docker version
Client:
Version: 17.06.2-ce
API version: 1.30
Go version: go1.8.3
Git commit: a176465
Built: Wed Sep 20 02:03:17 2017
OS/Arch: linux/amd64
Server:
Version: 17.06.2-ce
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: a176465
Built: Wed Sep 20 02:04:53 2017
OS/Arch: linux/amd64
Experimental: false
The workaround way I found:
- recreate br0 by recreate network or restart docker daemon.
- ip link set br0 down && ip link set br0 up
Sometimes in one overlay network and only on one node. The container cannot access another containers in one overlay network. But I don't know how to reproduce this issue.
I use the
tcpdumpto catch the packages in overlay network namespace. Found the packages not be forward to other container'sveth. But be forward tovxlan0.tcpdump at connection issue:
tcpdump at normal:
Check the br0 forward table. The br0 bridge mac table's port is different with normal state:
brctl showmacs br0at normal state:docker version and system info
The workaround way I found: