Skip to content

docker daemon panic with swarmkit and lots of services(300+) #28465

@wrfly

Description

@wrfly

10+ worker nodes and 3 manager nodes

docker info:

Containers: 209
 Running: 7
 Paused: 0
 Stopped: 202
Images: 8
Server Version: 1.12.3
Storage Driver: devicemapper
 Pool Name: docker-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 7.44 GB
 Data Space Total: 42.9 GB
 Data Space Available: 35.46 GB
 Metadata Space Used: 9.806 MB
 Metadata Space Total: 109.1 MB
 Metadata Space Available: 99.25 MB
 Thin Pool Minimum Free Space: 4.29 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local rexray
 Network: host bridge null overlay
Swarm: active
 NodeID: 22vdutsft30ybydl1sb21237r
 Is Manager: false
 Node Address: 10.10.29.24
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.22.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.64 GiB
Name: 10-10-29-24
ID: DN2L:4CMR:U5S5:XAOT:7BSZ:SZXL:FNVI:GQSR:3Q4K:KJSS:PB7D:5PGQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

logs:

Nov 16 11:18:12 10-10-135-223 dockerd[20196]: time="2016-11-16T11:18:12+08:00" level=error msg="setting up rule failed,
Nov 16 11:18:12 10-10-135-223 dockerd[20196]: panic: runtime error: invalid memory address or nil pointer dereference
Nov 16 11:18:12 10-10-135-223 dockerd[20196]: [signal 0xb code=0x1 addr=0x0 pc=0xde171e]
Nov 16 11:18:12 10-10-135-223 dockerd[20196]: goroutine 9610 [running]:
Nov 16 11:18:12 10-10-135-223 dockerd[20196]: panic(0x1a90b60, 0xc82000a080)
Nov 16 11:18:12 10-10-135-223 dockerd[20196]: /usr/local/go/src/runtime/panic.go:481 +0x3e6
Nov 16 11:18:12 10-10-135-223 dockerd[20196]: github.com/miekg/dns.getUDPSocketName(0x0, 0x0, 0x0, 0x0, 0x0)
Nov 16 11:18:12 10-10-135-223 dockerd[20196]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/miekg/dns/udp_li
Nov 16 11:18:12 10-10-135-223 dockerd[20196]: github.com/miekg/dns.setUDPSocketOptions(0x0, 0x0, 0x0)
Nov 16 11:18:12 10-10-135-223 dockerd[20196]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/miekg/dns/udp.go
Nov 16 11:18:12 10-10-135-223 dockerd[20196]: github.com/miekg/dns.(*Server).ActivateAndServe(0xc82087e1a0, 0x0, 0x0)
Nov 16 11:18:12 10-10-135-223 dockerd[20196]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/miekg/dns/server
Nov 16 11:18:12 10-10-135-223 dockerd[20196]: github.com/docker/libnetwork.(*resolver).Start.func1(0xc82087e1a0)
Nov 16 11:18:12 10-10-135-223 dockerd[20196]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwor
Nov 16 11:18:12 10-10-135-223 dockerd[20196]: created by github.com/docker/libnetwork.(*resolver).Start
Nov 16 11:18:12 10-10-135-223 dockerd[20196]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwor
Nov 16 11:18:15 10-10-135-223 dockerd[20196]: time="2016-11-16T11:18:14+08:00" level=error msg="setting up rule failed,
Nov 16 11:18:14 10-10-135-223 systemd[1]: docker.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 16 11:18:15 10-10-135-223 systemd[1]: Unit docker.service entered failed state.
Nov 16 11:18:15 10-10-135-223 systemd[1]: docker.service failed.

Metadata

Metadata

Assignees

Labels

area/swarmkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.priority/P1Important: P1 issues are a top priority and a must-have for the next release.version/1.12

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions