SIGSEGV in libnetwork.(*endpoint).deleteFromCluster #28712
Description
Description
The Docker for Mac integration tests occasionally trigger a crash of the docker daemon. It seems to be triggered by the service smoke test which is (roughly, I've had to remove some test framework boilerplate)
IMAGE_NAME=webserver
clean_up() {
docker service rm "${IMAGE_NAME}" || true
}
trap clean_up EXIT
# create docker service
docker service create --replicas 1 --name "${IMAGE_NAME}" --publish 80:80/tcp nginx:1.10.2-alpine
sleep 5
# try to load a web page every second for 120 seconds
# -- the real case calls a framework function which tries to `curl` the webpage for up to 120 minutesDescribe the results you received:
The docker daemon crashes and the logs contain:
time="2016-11-21T09:55:39.324896400Z" level=info msg="Stopping manager"
time="2016-11-21T09:55:39.325990500Z" level=debug msg="(*Agent).run exited" modu
le="node/agent"
time="2016-11-21T09:55:39.326757100Z" level=error msg="failed to remove node" er
ror="rpc error: code = 10 desc = dispatcher is stopped" method="(*Dispatcher).Se
ssion" node.id=ll0ocslii2t2lknftkk58kmfe node.session=pajjkh19rwbd9h4ocowzxhgdt
time="2016-11-21T09:55:39.327134700Z" level=error msg="LogBroker exited with an
error" error="context canceled"
time="2016-11-21T09:55:39.327826400Z" level=info msg="Manager shut down"
time="2016-11-21T09:55:39.328143500Z" level=debug msg="Sending kill signal 9 to
container 1c7b9cffcc0b8014197d2f69929b83737bd860a31d700d4054f313860f1a7127"
time="2016-11-21T09:55:39.341277300Z" level=debug msg="/usr/local/sbin/iptables,
[--wait -t filter -L DOCKER-INGRESS]"
time="2016-11-21T09:55:39.352599100Z" level=debug msg="/usr/local/sbin/iptables,
[--wait -t nat -L DOCKER-INGRESS]"
time="2016-11-21T09:55:39.3611242Z" level=debug msg="containerd: process exited"
id=1c7b9cffcc0b8014197d2f69929b83737bd860a31d700d4054f313860f1a7127 pid=init st
atus=137 systemPid=19330
time="2016-11-21T09:55:39.369023400Z" level=debug msg="libcontainerd: received c
ontainerd event: &types.Event{Type:\"exit\", Id:\"1c7b9cffcc0b8014197d2f69929b83
737bd860a31d700d4054f313860f1a7127\", Status:0x89, Pid:\"init\", Timestamp:(*tim
estamp.Timestamp)(0xc4214e8e40)}"
time="2016-11-21T09:55:39.371913500Z" level=debug msg="checkEncryption(shi07d4,
<nil>, 0, true)"
time="2016-11-21T09:55:39.375425800Z" level=debug msg="/usr/local/sbin/iptables,
[--wait -t nat -D DOCKER-INGRESS -p tcp --dport 80 -j DNAT --to-destination 172.18.0.2:80]"
time="2016-11-21T09:55:39.388322100Z" level=debug msg="/usr/local/sbin/iptables, [--wait -D DOCKER-INGRESS -m state -p tcp --sport 80 --state ESTABLISHED,RELATED -j ACCEPT]"
time="2016-11-21T09:55:39.422325000Z" level=debug msg="Revoking external connectivity on endpoint gateway_1c7b9cffcc0b (4fa59b13b6a33d922a1adb3ac7eac1b9d3dccd4b9bf2ab3c5fdef8653cf24c7c)"
time="2016-11-21T09:55:39.464712400Z" level=debug msg="/usr/local/sbin/iptables, [--wait -D DOCKER-INGRESS -p tcp --dport 80 -j ACCEPT]"
time="2016-11-21T09:55:39.506716300Z" level=debug msg="Releasing addresses for endpoint gateway_1c7b9cffcc0b's interface on network docker_gwbridge"
time="2016-11-21T09:55:39.506780800Z" level=debug msg="ReleaseAddress(LocalDefault/172.18.0.0/16, 172.18.0.3)"
time="2016-11-21T09:55:39.531994000Z" level=debug msg="Releasing addresses for endpoint webserver.1.5s6d3ottqo8y4oha7ic8j4ej7's interface on network ingress"
time="2016-11-21T09:55:39.532047400Z" level=debug msg="ReleaseAddress(LocalDefault/10.255.0.0/16, 10.255.0.5)"
time="2016-11-21T09:55:39Z" level=warning msg="Running modprobe nf_nat failed with message: `modprobe: module nf_nat not found in modules.dep`, error: exit status 1"
time="2016-11-21T09:55:39Z" level=warning msg="Running modprobe xt_conntrack failed with message: `modprobe: module xt_conntrack not found in modules.dep`, error: exit status 1"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x8268f5]
goroutine 9673 [running]:
panic(0x173e9a0, 0xc4200120b0)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/docker/docker/vendor/github.com/docker/libnetwork.(*endpoint).deleteFromCluster(0xc42146f8c0, 0xc420df1700, 0x1)
/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/agent.go:452 +0x285
github.com/docker/docker/vendor/github.com/docker/libnetwork.(*sandbox).DisableService(0xc4201014a0, 0xc420761680, 0x24cea80)
/go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/sandbox.go:679 +0xdb
github.com/docker/docker/daemon.(*Daemon).DeactivateContainerServiceBinding(0xc420100b40, 0xc42103db00, 0x25, 0x24adb80, 0xc420012d90)
/go/src/github.com/docker/docker/daemon/container_operations.go:1046 +0x87
github.com/docker/docker/daemon/cluster/executor/container.(*containerAdapter).deactivateServiceBinding(0xc4211da720, 0x0, 0x0)
/go/src/github.com/docker/docker/daemon/cluster/executor/container/adapter.go:391 +0x5c
github.com/docker/docker/daemon/cluster/executor/container.(*controller).Shutdown(0xc420fe6e10, 0x24c7140, 0xc4211da8d0, 0xc420944c80, 0x0)
/go/src/github.com/docker/docker/daemon/cluster/executor/container/controller.go:325 +0x6a
github.com/docker/docker/daemon/cluster/executor/container.(*controller).Remove(0xc420fe6e10, 0x24c7140, 0xc4211da8d0, 0xc4211da801, 0xc4216a91a0)
/go/src/github.com/docker/docker/daemon/cluster/executor/container/controller.go:373 +0x93
github.com/docker/docker/vendor/github.com/docker/swarmkit/agent.(*taskManager).run(0xc42039e600, 0x24c7140, 0xc4211da8d0)
/go/src/github.com/docker/docker/vendor/github.com/docker/swarmkit/agent/task.go:237 +0x1105
created by github.com/docker/docker/vendor/github.com/docker/swarmkit/agent.newTaskManager
/go/src/github.com/docker/docker/vendor/github.com/docker/swarmkit/agent/task.go:35 +0x19b
Additional information you deem important (e.g. issue happens only occasionally):
Definitely occasional. I don't have a good way to measure the failure rate at the moment.
Output of docker version:
Client:
Version: 1.13.0-rc1
API version: 1.25
Go version: go1.7.3
Git commit: 75fd88b
Built: Fri Nov 11 22:32:34 2016
OS/Arch: linux/amd64
At the point the diagnostics were captured the server had died. An earlier pre-failure diagnostics capture had:
Server:
Version: 1.13.0-rc1
API version: 1.25
Minimum API version: 1.12
Go version: go1.7.3
Git commit: 75fd88b
Built: Fri Nov 11 22:32:34 2016
OS/Arch: linux/amd64
Experimental: true
Output of docker info:
This is from an earlier diagnostics capture, before the failure:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.13.0-rc1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8517738ba4b82aff5662c97ca4627e7e4d03b531
runc version: ac031b5bf1cc92239461125f4c1ffb760522bbf2
init version: N/A (expected: v0.13.0)
Security Options:
seccomp
Profile: default
Kernel Version: 4.8.8-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.952 GiB
Name: moby
ID: 6PVD:ZIYZ:S4CJ:OMQW:MC3V:OBGD:5N57:3DAU:O2K2:CUQJ:L7AE:QOVM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 16
Goroutines: 26
System Time: 2016-11-21T09:44:25.8236626Z
EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
This is Docker for Mac with 1.13 rc1.