Skip to content

panic: close of nil channel #30469

@lukemarsden

Description

@lukemarsden

Docker 1.13.0 on Ubuntu 16.10 with overlayfs.

uname -a: Linux hostess 4.8.0-34-generic #36-Ubuntu SMP Wed Dec 21 17:24:18 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I ran a docker run in a (bash) loop and eventually got, in response to docker run:

rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:100: writing config to pipe caused \"write parent: broken pipe\""

Docker is now in a state where it can't start any new containers.

journalctl -u docker shows:

Jan 26 15:33:52 hostess dockerd[1127]: panic: close of nil channel
Jan 26 15:33:52 hostess dockerd[1127]: goroutine 5386 [running]:
Jan 26 15:33:52 hostess dockerd[1127]: panic(0x55f4378ac380, 0xc4202cd160)
Jan 26 15:33:52 hostess dockerd[1127]:         /usr/local/go/src/runtime/panic.go:500 +0x1a1
Jan 26 15:33:52 hostess dockerd[1127]: github.com/docker/containerd/supervisor.(*Supervisor).execExit.func1(0xc42006d130, 0xc420153520, 0x0)
Jan 26 15:33:52 hostess dockerd[1127]:         /tmp/tmp.7WL5Xw2yXV/src/github.com/docker/containerd/supervisor/exit.go:90 +0x10c
Jan 26 15:33:52 hostess dockerd[1127]: created by github.com/docker/containerd/supervisor.(*Supervisor).execExit
Jan 26 15:33:52 hostess dockerd[1127]:         /tmp/tmp.7WL5Xw2yXV/src/github.com/docker/containerd/supervisor/exit.go:91 +0xee
Jan 26 15:33:52 hostess dockerd[1127]: time="2017-01-26T15:33:52.584104029Z" level=error msg="libcontainerd: failed to receive event from containerd: rpc error: code = 13 desc = transport is closing"
Jan 26 15:33:53 hostess dockerd[1127]: time="2017-01-26T15:33:53.697412186Z" level=info msg="libcontainerd: new containerd process, pid: 29204"
Jan 26 15:33:54 hostess dockerd[1127]: time="2017-01-26T15:33:54.95232418Z" level=error msg="containerd: notify OOM events" error="open /proc/16803/cgroup: no such file or directory"
Jan 26 15:33:56 hostess dockerd[1127]: time="2017-01-26T15:33:55.207788976Z" level=info msg="libcontainerd: new containerd process, pid: 966"
Jan 26 15:33:56 hostess dockerd[1127]: time="2017-01-26T15:33:56.985502343Z" level=info msg="libcontainerd: new containerd process, pid: 3509"
Jan 26 15:33:57 hostess dockerd[1127]: time="2017-01-26T15:33:57.017577362Z" level=error msg="containerd: notify OOM events" error="open /proc/16803/cgroup: no such file or directory"
Jan 26 15:33:57 hostess dockerd[1127]: time="2017-01-26T15:33:57.039516095Z" level=error msg="containerd: deleting container" error="exit status 1: \"container e126ba60418f717fdd75a67add705c2485e0af8852c8303858425b52a869ad46 does not exist\\none or more of the container deletions failed\\n\""
Jan 26 15:33:58 hostess dockerd[1127]: time="2017-01-26T15:33:58.458657228Z" level=error msg="Handler for DELETE /v1.25/containers/node2 returned error: No such container: node2"
Jan 26 15:34:02 hostess dockerd[1127]: time="2017-01-26T15:34:02.695860092Z" level=error msg="Handler for DELETE /v1.25/containers/node1 returned error: No such container: node1"
Jan 26 15:34:02 hostess dockerd[1127]: time="2017-01-26T15:34:02.949647187Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Jan 26 15:34:02 hostess dockerd[1127]: time="2017-01-26T15:34:02.949715038Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Jan 26 15:34:02 hostess dockerd[1127]: time="2017-01-26T15:34:02.991431680Z" level=warning msg="Your kernel does not support swap memory limit."
Jan 26 15:34:02 hostess dockerd[1127]: time="2017-01-26T15:34:02.992169710Z" level=warning msg="Your kernel does not support cgroup rt period"
Jan 26 15:34:02 hostess dockerd[1127]: time="2017-01-26T15:34:02.992220595Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Jan 26 15:34:08 hostess dockerd[1127]: time="2017-01-26T15:34:08.540562939Z" level=error msg="Handler for HEAD /v1.25/containers/node1/archive returned error: lstat /var/lib/docker/overlay/ddfa3f7c2b0fd62b2a2de9f644255991f03f1cdb8a43439256c471d3886a3c70/merged/usr/local/bin/program: no such file or directory"
Jan 26 15:34:11 hostess dockerd[1127]: time="2017-01-26T15:34:11.697685141Z" level=error msg="Handler for DELETE /v1.25/containers/node2 returned error: No such container: node2"
Jan 26 15:34:11 hostess dockerd[1127]: time="2017-01-26T15:34:11.946912514Z" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Jan 26 15:34:11 hostess dockerd[1127]: time="2017-01-26T15:34:11.947610115Z" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Jan 26 15:34:12 hostess dockerd[1127]: time="2017-01-26T15:34:12.004786104Z" level=warning msg="Your kernel does not support swap memory limit."
Jan 26 15:34:12 hostess dockerd[1127]: time="2017-01-26T15:34:12.005564908Z" level=warning msg="Your kernel does not support cgroup rt period"
Jan 26 15:34:12 hostess dockerd[1127]: time="2017-01-26T15:34:12.006125481Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Jan 26 15:34:13 hostess dockerd[1127]: time="2017-01-26T15:34:13.909004448Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:100: writing config to pipe caused \\\"write parent: 
Jan 26 15:34:15 hostess dockerd[1127]: time="2017-01-26T15:34:15.924571508Z" level=error msg="Handler for HEAD /v1.25/containers/node2/archive returned error: lstat /var/lib/docker/overlay/8209e9ae582810cc84582c43d223ef5b2d26c8d7b9ea02009b070ed99671f2e5/merged/usr/local/bin/program: no such file or directory"
Jan 26 15:34:18 hostess dockerd[1127]: time="2017-01-26T15:34:18.553498698Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:100: writing config to pipe caused \\\"write parent: 
Jan 26 15:34:19 hostess dockerd[1127]: time="2017-01-26T15:34:19.125424854Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:100: writing config to pipe caused \\\"write parent: 
Jan 26 15:34:19 hostess dockerd[1127]: time="2017-01-26T15:34:19.301627205Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:100: writing config to pipe caused \\\"write parent: 
Jan 26 15:34:19 hostess dockerd[1127]: time="2017-01-26T15:34:19.458650211Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:100: writing config to pipe caused \\\"write parent: 
Jan 26 15:34:19 hostess dockerd[1127]: time="2017-01-26T15:34:19.590213528Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:100: writing config to pipe caused \\\"write parent: 
Jan 26 15:34:19 hostess dockerd[1127]: time="2017-01-26T15:34:19.718792549Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:100: writing config to pipe caused \\\"write parent: 
Jan 26 15:34:19 hostess dockerd[1127]: time="2017-01-26T15:34:19.853617646Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:100: writing config to pipe caused \\\"write parent: 
Jan 26 15:34:19 hostess dockerd[1127]: time="2017-01-26T15:34:19.978626985Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:100: writing config to pipe caused \\\"write parent: 
Jan 26 15:34:20 hostess dockerd[1127]: time="2017-01-26T15:34:20.118062448Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:100: writing config to pipe caused \\\"write parent: 
Jan 26 15:34:20 hostess dockerd[1127]: time="2017-01-26T15:34:20.232930431Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:100: writing config to pipe caused \\\"write parent: 
Jan 26 15:34:20 hostess dockerd[1127]: time="2017-01-26T15:34:20.345989470Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:100: writing config to pipe caused \\\"write parent: 
Jan 26 15:34:20 hostess dockerd[1127]: time="2017-01-26T15:34:20.476161016Z" level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused \"process_linux.go:100: writing config to pipe caused \\\"write parent: 

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/runtimeRuntimepriority/P1Important: P1 issues are a top priority and a must-have for the next release.version/1.13

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions