Skip to content

[1.12.3] Swarm node crash "fatal error: concurrent map read and map write" #28697

@JamesJJ

Description

@JamesJJ

Description

Describe the results you received:

Nov 13 10:49:12 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: time="2016-11-13T10:49:12.953788653+08:00" level=warning msg="containerd: unable to save b7afc798640b3c3ae47a488f5f1a5e5152716e7d3f7d1a3a5776ec7ebcd9483c:fa0bc2189b2b9724dc1677cb8db4b76a891e744372c85455a2d2b7b49de2f33b starttime: open /proc/12154/stat: no such file or directory"
Nov 13 10:49:12 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: time="2016-11-13T10:49:12.954122985+08:00" level=info msg="containerd: b7afc798640b3c3ae47a488f5f1a5e5152716e7d3f7d1a3a5776ec7ebcd9483c:fa0bc2189b2b9724dc1677cb8db4b76a891e744372c85455a2d2b7b49de2f33b (pid 12154) has become an orphan, killing it"
Nov 13 10:49:12 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: time="2016-11-13T10:49:12+08:00" level=info msg="Firewalld running: false"
Nov 13 10:49:13 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: time="2016-11-13T10:49:13+08:00" level=info msg="Firewalld running: false"
Nov 13 10:49:13 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: time="2016-11-13T10:49:13+08:00" level=info msg="Firewalld running: false"
Nov 13 10:49:13 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: time="2016-11-13T10:49:13+08:00" level=info msg="Firewalld running: false"
Nov 13 10:49:13 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: time="2016-11-13T10:49:13+08:00" level=info msg="Firewalld running: false"
Nov 13 10:49:13 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: time="2016-11-13T10:49:13+08:00" level=info msg="Firewalld running: false"
Nov 13 10:49:13 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: time="2016-11-13T10:49:13+08:00" level=info msg="Firewalld running: false"
Nov 13 10:49:13 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: time="2016-11-13T10:49:13+08:00" level=info msg="Firewalld running: false"
Nov 13 10:49:21 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: time="2016-11-13T10:49:21.182191759+08:00" level=warning msg="containerd: unable to save b7afc798640b3c3ae47a488f5f1a5e5152716e7d3f7d1a3a5776ec7ebcd9483c:1e7d5ea92f85ab0f8d98b9770fd9990f2549a4d23334711fa60fd658977bdc5e starttime: open /proc/12685/stat: no such file or directory"
Nov 13 10:49:21 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: time="2016-11-13T10:49:21.182590467+08:00" level=info msg="containerd: b7afc798640b3c3ae47a488f5f1a5e5152716e7d3f7d1a3a5776ec7ebcd9483c:1e7d5ea92f85ab0f8d98b9770fd9990f2549a4d23334711fa60fd658977bdc5e (pid 12685) has become an orphan, killing it"
Nov 13 10:49:21 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: time="2016-11-13T10:49:21.628721939+08:00" level=error msg="fatal task error" error="task: non-zero exit (255)" module=taskmanager task.id=97thob09zjgt0ql63ltj7f2ph
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: fatal error: concurrent map read and map write
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 220 [running]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: runtime.throw(0x1f57e60, 0x21)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc83e3398a8 sp=0xc83e339890
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: runtime.mapaccess2(0x16ebba0, 0xc82043f830, 0xc83e339ac8, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/hashmap.go:343 +0x5a fp=0xc83e3398f0 sp=0xc83e3398a8
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/libnetwork.(*controller).addServiceBinding(0xc8202562d0, 0xc830cbf990, 0x7, 0xc8399fb400, 0x19, 0xc831f97620, 0x19, 0xc835f11380, 0x40, 0xc830cbfa00, ...)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/service_linux.go:101 +0x1b8 fp=0xc83e339af0 sp=0xc83e3398f0
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/libnetwork.(*controller).handleEpTableEvent(0xc8202562d0, 0x1b5fbc0, 0xc838b095e0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/agent.go:615 +0x7c1 fp=0xc83e339f20 sp=0xc83e339af0
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/libnetwork.(*controller).(github.com/docker/libnetwork.handleEpTableEvent)-fm(0x1b5fbc0, 0xc838b095e0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/agent.go:276 +0x34 fp=0xc83e339f40 sp=0xc83e339f20
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/libnetwork.(*controller).handleTableEvents(0xc8202562d0, 0xc820c95f20, 0xc8209837f0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/agent.go:523 +0x87 fp=0xc83e339f88 sp=0xc83e339f40
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: runtime.goexit()
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc83e339f90 sp=0xc83e339f88
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/libnetwork.(*controller).agentInit
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/agent.go:276 +0x66d
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 1 [chan receive, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: main.(*DaemonCli).start(0xc82045f1a0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/cmd/dockerd/daemon.go:308 +0x2274
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: main.main()
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/cmd/dockerd/docker.go:68 +0x491
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 17 [syscall, 5494 minutes, locked to thread]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: runtime.goexit()
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 34 [syscall, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: os/signal.signal_recv(0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/sigqueue.go:116 +0x132
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: os/signal.loop()
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by os/signal.init.1
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/os/signal/signal_unix.go:28 +0x37
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 37 [syscall, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: syscall.Syscall6(0x3d, 0x4d0, 0xc820168ddc, 0x0, 0xc820090090, 0x0, 0x0, 0x446561, 0x446561, 0xc820050b38)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: syscall.wait4(0x4d0, 0xc820168ddc, 0x0, 0xc820090090, 0x90, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/syscall/zsyscall_linux_amd64.go:172 +0x7f
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: syscall.Wait4(0x4d0, 0xc820168e24, 0x0, 0xc820090090, 0x41a1e3, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/syscall/syscall_linux.go:256 +0x55
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: os.(*Process).wait(0xc820486780, 0xc82000e080, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/os/exec_unix.go:22 +0x105
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: os.(*Process).Wait(0xc820486780, 0x0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/os/doc.go:49 +0x2d
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: os/exec.(*Cmd).Wait(0xc820449900, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/os/exec/exec.go:396 +0x211
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/docker/libcontainerd.(*remote).runContainerdDaemon.func1(0xc820449900, 0xc8204575f0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/libcontainerd/remote_linux.go:418 +0x21
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/docker/libcontainerd.(*remote).runContainerdDaemon
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/libcontainerd/remote_linux.go:420 +0x12eb
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 38 [select, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc.(*Conn).transportMonitor(0xc820256870)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/clientconn.go:544 +0x1d3
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc.NewConn.func1(0xc820256870)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/clientconn.go:355 +0x1b5
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by google.golang.org/grpc.NewConn
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/clientconn.go:356 +0x4e3
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 39 [semacquire, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: sync.runtime_Syncsemacquire(0xc82048d1d0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/sema.go:241 +0x201
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: sync.(*Cond).Wait(0xc82048d1c0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/sync/cond.go:63 +0x9b
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc.(*Conn).WaitForStateChange(0xc820256870, 0x7ff55f5ae028, 0xc820060780, 0x2, 0x0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/clientconn.go:405 +0x19f
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc.(*unicastPicker).WaitForStateChange(0xc820486840, 0x7ff55f5ae028, 0xc820060780, 0x2, 0x203cee0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/picker.go:96 +0x4f
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc.(*ClientConn).WaitForStateChange(0xc82016e790, 0x7ff55f5ae028, 0xc820060780, 0x2, 0x1, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/clientconn.go:283 +0x6b
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/docker/libcontainerd.(*remote).handleConnectionChange(0xc8204575f0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/libcontainerd/remote_linux.go:139 +0xb4
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/docker/libcontainerd.New
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/libcontainerd/remote_linux.go:117 +0xb42
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 42 [IO wait]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.runtime_pollWait(0x7ff55f631d48, 0x72, 0xc8204ba000)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/netpoll.go:160 +0x60
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.(*pollDesc).Wait(0xc8204653a0, 0x72, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.(*pollDesc).WaitRead(0xc8204653a0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.(*netFD).Read(0xc820465340, 0xc8204ba000, 0x8000, 0x8000, 0x0, 0x7ff55f5ee140, 0xc8200600c8)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/net/fd_unix.go:250 +0x23a
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.(*conn).Read(0xc82015c530, 0xc8204ba000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/net/net.go:172 +0xe4
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: bufio.(*Reader).fill(0xc820484600)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/bufio/bufio.go:97 +0x1e9
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: bufio.(*Reader).Read(0xc820484600, 0xc82023e9f8, 0x9, 0x9, 0x419c61, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/bufio/bufio.go:207 +0x260
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: io.ReadAtLeast(0x7ff55f5b6288, 0xc820484600, 0xc82023e9f8, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/io/io.go:297 +0xe6
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: io.ReadFull(0x7ff55f5b6288, 0xc820484600, 0xc82023e9f8, 0x9, 0x9, 0x46a1c0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/io/io.go:315 +0x62
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: golang.org/x/net/http2.readFrameHeader(0xc82023e9f8, 0x9, 0x9, 0x7ff55f5b6288, 0xc820484600, 0x0, 0x7ff500000000, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/golang.org/x/net/http2/frame.go:236 +0xa5
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: golang.org/x/net/http2.(*Framer).ReadFrame(0xc82023e9c0, 0x0, 0x0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/golang.org/x/net/http2/frame.go:463 +0x106
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc/transport.(*framer).readFrame(0xc82045fbf0, 0x0, 0x0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/transport/http_util.go:406 +0x3d
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc/transport.(*http2Client).reader(0xc820256960)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/transport/http2_client.go:788 +0x109
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by google.golang.org/grpc/transport.newHTTP2Client
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/transport/http2_client.go:174 +0xd21
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 43 [select]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc/transport.(*http2Client).controller(0xc820256960)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/transport/http2_client.go:866 +0x5da
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by google.golang.org/grpc/transport.newHTTP2Client
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/transport/http2_client.go:201 +0x15c2
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 137 [semacquire, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: sync.runtime_Syncsemacquire(0xc820a547d0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/sema.go:241 +0x201
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: sync.(*Cond).Wait(0xc820a547c0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/sync/cond.go:63 +0x9b
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/swarmkit/agent.(*Node).ListenControlSocket.func2(0xc820a46960, 0xc820a469c0, 0xc82023e300, 0x7ff55f5ae028, 0xc820060780, 0xc82015c0f8)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/node.go:454 +0xf6
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/swarmkit/agent.(*Node).ListenControlSocket
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/node.go:460 +0x164
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 21 [select, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc.NewClientStream.func1(0x7ff55f5f7200, 0xc820256960, 0xc8200a6000, 0xc820096000)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/stream.go:151 +0x258
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by google.golang.org/grpc.NewClientStream
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/stream.go:159 +0xab2
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 22 [select]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc820065f20, 0xc82049e0d0, 0x5, 0x5, 0x0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/transport/transport.go:141 +0x7e6
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc/transport.(*Stream).Read(0xc8200a6000, 0xc82049e0d0, 0x5, 0x5, 0x10, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/transport/transport.go:294 +0x71
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: io.ReadAtLeast(0x7ff55f5f73a0, 0xc8200a6000, 0xc82049e0d0, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/io/io.go:297 +0xe6
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: io.ReadFull(0x7ff55f5f73a0, 0xc8200a6000, 0xc82049e0d0, 0x5, 0x5, 0x1, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/io/io.go:315 +0x62
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc.(*parser).recvMsg(0xc82049e0c0, 0x100000002, 0x0, 0x0, 0x0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/rpc_util.go:216 +0xb9
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc.recv(0xc82049e0c0, 0x7ff55f5b8100, 0x2ad0df0, 0xc8200a6000, 0x0, 0x0, 0x1ba8640, 0xc82ffa0800, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/rpc_util.go:300 +0x45
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc820096000, 0x1ba8640, 0xc82ffa0800, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/stream.go:234 +0xac
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/containerd/api/grpc/types.(*aPIEventsClient).Recv(0xc82049a0a0, 0xecfb9cbb2, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/containerd/api/grpc/types/api.pb.go:1032 +0x7e
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/docker/libcontainerd.(*remote).handleEventStream(0xc8204575f0, 0x7ff55f5f74f0, 0xc82049a0a0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/libcontainerd/remote_linux.go:286 +0x64
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/docker/libcontainerd.(*remote).startEventsMonitor
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/libcontainerd/remote_linux.go:280 +0x2db
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 23 [select, 5494 minutes, locked to thread]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: runtime.gopark(0x2193eb8, 0xc820165f28, 0x1d78038, 0x6, 0x18, 0x2)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/proc.go:262 +0x163
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: runtime.selectgoImpl(0xc820165f28, 0x0, 0x18)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/select.go:392 +0xa67
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: runtime.selectgo(0xc820165f28)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/select.go:215 +0x12
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: runtime.ensureSigM.func1()
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/signal1_unix.go:279 +0x358
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: runtime.goexit()
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 24 [chan receive, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/docker/pkg/signal.Trap.func1(0xc82005c420, 0xc82049e1c0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/signal/trap.go:32 +0x71
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/docker/pkg/signal.Trap
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/pkg/signal/trap.go:62 +0x2be
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 25 [chan receive, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/docker/daemon.setupDumpStackTrap.func1(0xc82005c600)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/debugtrap_unix.go:17 +0x4d
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/docker/daemon.setupDumpStackTrap
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/debugtrap_unix.go:20 +0x13f
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 26 [chan receive]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/docker/daemon/graphdriver/devmapper.(*DeviceSet).startDeviceDeletionWorker(0xc820068340)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/graphdriver/devmapper/deviceset.go:703 +0x14b
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/docker/daemon/graphdriver/devmapper.(*DeviceSet).initMetaData
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/graphdriver/devmapper/deviceset.go:735 +0x309
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 58 [select, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/swarmkit/agent.(*Node).run.func2(0x7ff55f5f72c8, 0xc820a52510, 0xc82023e300, 0xc820d1dcf0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/node.go:172 +0xff
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/swarmkit/agent.(*Node).run
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/node.go:177 +0x3bf
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 136 [select, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/swarmkit/agent.(*Node).ListenControlSocket.func1(0x7ff55f5ae028, 0xc820060780, 0xc82023e300, 0xc820a469c0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/node.go:439 +0xf7
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/swarmkit/agent.(*Node).ListenControlSocket
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/node.go:444 +0x110
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 29 [chan receive]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/docker/daemon.(*statsCollector).run(0xc820233980)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/stats_collector_unix.go:106 +0xb6
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/docker/daemon.(*Daemon).newStatsCollector
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/stats_collector_unix.go:44 +0x1f1
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 30 [chan receive, 5 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/docker/daemon.(*Daemon).execCommandGC(0xc8200681a0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/exec.go:237 +0x87
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/docker/daemon.NewDaemon
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:598 +0x339c
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 31 [select]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/libnetwork.(*controller).watchLoop(0xc8202562d0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/store.go:439 +0x13b
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/libnetwork.(*controller).startWatch
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/store.go:456 +0x112
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 192 [select]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/go-events.(*Broadcaster).run(0xc821146500)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/go-events/broadcast.go:113 +0x9a4
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/go-events.NewBroadcaster
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/go-events/broadcast.go:30 +0x199
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 135 [chan receive, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/docker/daemon/cluster.(*Cluster).startNewNode.func3(0xc82023e300, 0x7ff55f5ae028, 0xc820060780, 0xc82018c700, 0xc820a54a80)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/cluster/cluster.go:334 +0x73
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/docker/daemon/cluster.(*Cluster).startNewNode
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/cluster/cluster.go:347 +0xc7a
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 133 [select, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/swarmkit/agent.(*Node).Err(0xc82023e300, 0x7ff55f5ae028, 0xc820060780, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/node.go:355 +0x171
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/docker/daemon/cluster.(*Cluster).startNewNode.func1(0xc82023e300, 0x7ff55f5ae028, 0xc820060780, 0xc82018c700, 0xc820a54a80)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/cluster/cluster.go:310 +0x47
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/docker/daemon/cluster.(*Cluster).startNewNode
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/cluster/cluster.go:319 +0xbd4
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 132 [select, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/libnetwork.(*controller).clusterAgentInit(0xc8202562d0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/controller.go:296 +0x744
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/libnetwork.(*controller).SetClusterProvider
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/controller.go:238 +0xc4
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 131 [semacquire, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: sync.runtime_Semacquire(0xc8209ae00c)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/sema.go:47 +0x26
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: sync.(*WaitGroup).Wait(0xc8209ae000)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/sync/waitgroup.go:127 +0xb4
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/swarmkit/agent.(*Node).run(0xc82023e300, 0x7ff55f5f72c8, 0xc820a52510, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/node.go:314 +0xc75
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/swarmkit/agent.(*Node).Start
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/node.go:158 +0x386
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 130 [select, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc.(*Conn).WaitForStateChange.func1(0x7ff55f5ae028, 0xc820060780, 0xc820256870, 0xc820a7e020, 0xc820a46000)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/clientconn.go:394 +0x1b1
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by google.golang.org/grpc.(*Conn).WaitForStateChange
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/clientconn.go:402 +0x14e
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 49 [IO wait]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.runtime_pollWait(0x7ff55f631c88, 0x72, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/netpoll.go:160 +0x60
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.(*pollDesc).Wait(0xc82005ad80, 0x72, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.(*pollDesc).WaitRead(0xc82005ad80, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.(*netFD).accept(0xc82005ad20, 0x0, 0x7ff55f5b8298, 0xc83a821ba0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/net/fd_unix.go:426 +0x27c
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.(*UnixListener).AcceptUnix(0xc820163ba0, 0xc82002fed0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/net/unixsock_posix.go:305 +0x53
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.(*UnixListener).Accept(0xc820163ba0, 0x0, 0x0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/net/unixsock_posix.go:315 +0x41
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/libnetwork.(*controller).acceptClientConnections(0xc8202562d0, 0xc82005d380, 0x5c, 0x7ff55f5b7f40, 0xc820163ba0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/sandbox_externalkey_unix.go:128 +0x41
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/libnetwork.(*controller).startExternalKeyListener
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/sandbox_externalkey_unix.go:122 +0x28a
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 12112148 [select]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/swarmkit/agent.(*taskManager).run(0xc8393add00, 0x7ff55f5f72c8, 0xc82764eba0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/task.go:92 +0x1868
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/swarmkit/agent.newTaskManager
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/task.go:35 +0x1a6
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 252 [semacquire, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: sync.runtime_Syncsemacquire(0xc821a732d0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/sema.go:241 +0x201
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: sync.(*Cond).Wait(0xc821a732c0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/sync/cond.go:63 +0x9b
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/go-events.(*Queue).next(0xc821a41860, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/go-events/queue.go:103 +0xd7
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/go-events.(*Queue).run(0xc821a41860)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/go-events/queue.go:68 +0x38
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/go-events.NewQueue
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/go-events/queue.go:29 +0x1b6
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 253 [chan receive, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/libnetwork.(*controller).handleTableEvents(0xc8202562d0, 0xc821add680, 0xc8218a5e10)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/agent.go:518 +0x51
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/libnetwork.(*network).addDriverWatches
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/agent.go:482 +0x3a1
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 198 [chan receive, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/docker/daemon/cluster.(*Cluster).reconnectOnFailure(0xc82018c700, 0xc820a54a80)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/cluster/cluster.go:205 +0x57
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/docker/daemon/cluster.(*Cluster).Join
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/cluster/cluster.go:481 +0x603
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 78 [select, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc/transport.(*http2Client).controller(0xc820d98780)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/transport/http2_client.go:866 +0x5da
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by google.golang.org/grpc/transport.newHTTP2Client
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/transport/http2_client.go:201 +0x15c2
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 75 [chan receive, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/swarmkit/agent.(*worker).Listen.func1(0x7ff55f5f72c8, 0xc8209c1770, 0xc82005f7c0, 0xc820a42e40)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/worker.go:182 +0x4e
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/swarmkit/agent.(*worker).Listen
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/worker.go:186 +0x150
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 74 [semacquire]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: sync.runtime_Syncsemacquire(0xc8203a44b0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/sema.go:241 +0x201
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: sync.(*Cond).Wait(0xc8203a44a0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/sync/cond.go:63 +0x9b
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/swarmkit/agent.(*statusReporter).run(0xc8203a4480, 0x7ff55f5f72c8, 0xc8209c1770)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/reporter.go:96 +0x159
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/swarmkit/agent.newStatusReporter
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/reporter.go:44 +0x13f
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 72 [select, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc.(*Conn).transportMonitor(0xc820d984b0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/clientconn.go:544 +0x1d3
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: google.golang.org/grpc.NewConn.func1(0xc820d984b0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/clientconn.go:355 +0x1b5
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by google.golang.org/grpc.NewConn
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/google.golang.org/grpc/clientconn.go:356 +0x4e3
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 210 [IO wait]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.runtime_pollWait(0x7ff55f631388, 0x72, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/runtime/netpoll.go:160 +0x60
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.(*pollDesc).Wait(0xc820a44840, 0x72, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.(*pollDesc).WaitRead(0xc820a44840, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.(*netFD).readFrom(0xc820a447e0, 0xc839bba000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x7ff55f5ee140, 0xc8200600c8)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/net/fd_unix.go:277 +0x2a5
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.(*UDPConn).ReadFromUDP(0xc820da6138, 0xc839bba000, 0x10000, 0x10000, 0x16fd220, 0x10000, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/net/udpsock_posix.go:61 +0x117
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: net.(*UDPConn).ReadFrom(0xc820da6138, 0xc839bba000, 0x10000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /usr/local/go/src/net/udpsock_posix.go:79 +0x116
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/hashicorp/memberlist.(*Memberlist).udpListen(0xc8200a7ea0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/hashicorp/memberlist/net.go:284 +0x301
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/hashicorp/memberlist.newMemberlist
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/hashicorp/memberlist/memberlist.go:140 +0xfcb
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 64 [select]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/swarmkit/agent.(*Agent).run(0xc820aa00e0, 0x7ff55f5f72c8, 0xc8209c1770)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/agent.go:162 +0x14e3
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by github.com/docker/swarmkit/agent.(*Agent).Start
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/swarmkit/agent/agent.go:81 +0x37d
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 110 [chan receive, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: main.(*DaemonCli).setupConfigReloadTrap.func1(0xc820a891a0, 0xc82045f1a0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/cmd/dockerd/daemon_unix.go:55 +0x4d
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by main.(*DaemonCli).setupConfigReloadTrap
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/cmd/dockerd/daemon_unix.go:58 +0x14c
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 111 [chan receive, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/docker/api/server.(*Server).serveAPI(0xc820484240, 0x0, 0x0)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/server.go:95 +0x19e
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: github.com/docker/docker/api/server.(*Server).Wait(0xc820484240, 0xc820a89200)
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/server.go:189 +0x2b
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: created by main.(*DaemonCli).start
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: /root/rpmbuild/BUILD/docker-engine/cmd/dockerd/daemon.go:301 +0x2233
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ dockerd[1225]: goroutine 113 [IO wait, 5494 minutes]:
Nov 13 10:49:22 iZuf6i4mxt5qb38759bu7uZ systemd[1]: docker.service failed.
[root@iZuf6i4mxt5qb38759bu7uZ centos]# 

Output of docker version:

 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 35
 Running: 3
 Paused: 0
 Stopped: 32
Images: 5
Server Version: 1.12.3
Storage Driver: devicemapper
 Pool Name: docker_lvm_vg-docker_lvm
 Pool Blocksize: 524.3 kB
 Base Device Size: 8.59 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 5.443 GB
 Data Space Total: 99.85 GB
 Data Space Available: 94.41 GB
 Metadata Space Used: 3.486 MB
 Metadata Space Total: 3.23 GB
 Metadata Space Available: 3.226 GB
 Thin Pool Minimum Free Space: 6.989 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host null bridge overlay
Swarm: active
 NodeID: d5csr9ahsjs7ss9alldq9wctg
 Is Manager: true
 ClusterID: 2gp68m9je6pnchh825bm7lo99
 Managers: 3
 Nodes: 4
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.24.219.43
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.36.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.51 GiB
Name: iZuf6i4mxt5qb38759bu7uZ
ID: NGEF:7WA7:O7CK:YD7L:EJDV:XUGP:TWJL:SK7I:B7PF:GTN4:NGDW:GLPT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

  • Aliyun / Alibaba China Cloud
  • 4 CPU / 16 GB RAM
  • Centos 7

Metadata

Metadata

Assignees

Labels

area/networkingNetworkingkind/bugBugs are bugs. The cause may or may not be known at triage time so debugging may be needed.priority/P1Important: P1 issues are a top priority and a must-have for the next release.version/1.12

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions