Restarting the KV store causes Docker Daemons to peg the CPU to 100% #408
Description
Problem
Using the new networking, if I restart the KV store (consul in this case) causes the Docker daemon process to peg the CPU to 100%. See attached graph of CPU usage on an EC2 instance.
With htop you can see the Docker process taking 100% of the one core. I have seen this issue on EC2 (Ubuntu), VMWare Fusion (boot2docker) and VirtualBox (boo2docker)
Reproduce
Start a new machine with daemon and run the container with consul
$ docker-machine create swarm-consul
$ docker run -d -p 8400:8400 -p 8500:8500 -p 8600:53/udp -h swarm-consul --name swarm-consul progrium/consul -server -bootstrap
Start a second machine with daemon
$ docker-machine create foo
Ensure you have configured the networking on the Daemon
--default-network=overlay:multihost --label=com.docker.network.driver.overlay.bind_interface=eth0
Restart consul machine
$ docker-machine restart swarm-consul
Workaround
- Restart the docker daemon e.g.
sudo service docker stop; sudo service docker start - Reboot the machine the docker daemon is running on
Environment
uname -a
Linux swarm-0 3.16.0-43-generic #58~14.04.1-Ubuntu SMP Mon Jun 22 10:21:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
docker version
Client:
Version: 1.8.0-dev
API version: 1.20
Go version: go1.4.2
Git commit: 8c7cd78
Built: Tue Jul 14 23:47:18 UTC 2015
OS/Arch: linux/amd64
Experimental: true
Server:
Version: 1.8.0-dev
API version: 1.20
Go version: go1.4.2
Git commit: 8c7cd78
Built: Tue Jul 14 23:47:18 UTC 2015
OS/Arch: linux/amd64
Experimental: true
docker info
Containers: 6
Images: 43
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 55
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.16.0-43-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 2
Total Memory: 7.304 GiB
Name: swarm-0
ID: LO2F:EV7S:H25G:5RSQ:QISR:WIG6:AP25:VMGR:RBA3:IXJN:2HR5:6DKJ
Debug mode (server): true
File Descriptors: 24
Goroutines: 56
System Time: 2015-07-22T18:09:45.419817568Z
EventsListeners: 0
Init SHA1:
Init Path: /usr/bin/docker
Docker Root Dir: /var/lib/docker
WARNING: No swap limit support
Labels:
provider=amazonec2
com.docker.network.driver.overlay.bind_interface=eth0
Experimental: true