Skip to content

Failed to start container in overlay network since "br0" exchange full #18814

@BSWANG

Description

@BSWANG

I am using docker overlay network, my docker-engine information on below.

$ docker info
Containers: 5
Images: 32
Server Version: 1.9.1
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.3.0-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Total Memory: 3.858 GiB
Name: cc821b04098444cb0a020171b6bbc2630-node1
ID: OZI5:2WWI:5GFY:HV5V:YWWS:DDLU:ASPW:D6F2:572T:LYJB:6R5F:BBW6
Cluster store: etcd://10.201.145.128:2379/cc821b04098444cb0a020171b6bbc2630
Cluster advertise: 10.165.126.76:2376

When I create and delete containers in overlay network with at least one container in the network. I will get could not add veth pair inside the network sandbox: error setting interface "vethe1e64f7" master to "br0": exchange full error, and the container remain on created state.

It seems docker daemon not delete the veth pair when deleting containers, and get bridge full when continue create containers in the network since cannot create veth on sandbox's default bridge "br0".

There is one test case below:

#create docker overlay network
docker network create -d overlay multi-host-network
#ensure at least one container in the overlay network
docker run -itd --net multi-host-network busybox
#create and remove container in the network
for (( i=0; i<1024; i++ )) do docker run -itd --net multi-host-network --name test$i busybox; docker rm -f test$i; done;

And I get

...
d7064a4a6218d34da6e4502294c325184780e603c802643bbb666aa44c579347
test1020
2dcabbab16238a34739e850e48e99e32eac2f5c504cf0207afbc475dfa9ebf26
test1021
afa928d3684e7d35d0ce9898262b45148dbb53d52fe60e5041c8a0b07b4a8250
Error response from daemon: Cannot start container afa928d3684e7d35d0ce9898262b45148dbb53d52fe60e5041c8a0b07b4a8250: could not add veth pair inside the network sandbox: error setting interface "veth5cbfa6a" master to "br0": exchange full
test1022
...

Metadata

Metadata

Assignees

Labels

area/networkingNetworkingpriority/P1Important: P1 issues are a top priority and a must-have for the next release.

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions