This repository was archived by the owner on Jan 30, 2020. It is now read-only.
This repository was archived by the owner on Jan 30, 2020. It is now read-only.
Panic on graceful shutdown (close of closed channel) #1369
Description
For fleet version 0.10.2 on CoreOS Stable branch, there seems to be two problems:
- Might close an already closed channel upon graceful shutdown
- The ever growing list of goroutines when having etcd connection issues, almost two million in this example
... snip snip (a few days of 503) ...
Oct 08 07:52:21 fleetd[127879]: INFO client.go:298: Response 503 Service Unavailable from http://127.0.0.1:4001/ unusable
Oct 08 07:52:21 fleetd[127879]: ERROR client.go:214: Unable to get result for {Create /_coreos.com/fleet/machines/0553e988b3e14a6ebd7d3108a2ab4d8e/object}, retrying in 1s
Oct 08 07:52:22 fleetd[127879]: INFO client.go:298: Response 503 Service Unavailable from http://127.0.0.1:4001/ unusable
Oct 08 07:52:22 fleetd[127879]: ERROR client.go:214: Unable to get result for {Create /_coreos.com/fleet/machines/0553e988b3e14a6ebd7d3108a2ab4d8e/object}, retrying in 1s
Oct 08 07:52:23 fleetd[127879]: INFO client.go:298: Response 503 Service Unavailable from http://127.0.0.1:4001/ unusable
Oct 08 07:52:23 fleetd[127879]: ERROR client.go:214: Unable to get result for {Create /_coreos.com/fleet/machines/0553e988b3e14a6ebd7d3108a2ab4d8e/object}, retrying in 1s
Oct 08 07:53:24 systemd[1]: Stopping fleet daemon...
Oct 08 07:53:24 fleetd[127879]: INFO fleet.go:105: Gracefully shutting down
Oct 08 07:53:24 fleetd[127879]: panic: close of closed channel
Oct 08 07:53:24 fleetd[127879]: goroutine 1 [running]:
Oct 08 07:53:24 fleetd[127879]: main.func·002()
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/fleetd/fleet.go:106 +0x6b
Oct 08 07:53:24 fleetd[127879]: main.listenForSignals(0xc2080b0ae0)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/fleetd/fleet.go:211 +0x1f7
Oct 08 07:53:24 fleetd[127879]: main.main()
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/fleetd/fleet.go:137 +0x12c1
Oct 08 07:53:24 fleetd[127879]: goroutine 5 [syscall]:
Oct 08 07:53:24 fleetd[127879]: os/signal.loop()
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/os/signal/signal_unix.go:21 +0x1f
Oct 08 07:53:24 fleetd[127879]: created by os/signal.init·1
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/os/signal/signal_unix.go:27 +0x35
Oct 08 07:53:24 fleetd[127879]: goroutine 6 [IO wait, 2376 minutes]:
Oct 08 07:53:24 fleetd[127879]: net.(*pollDesc).Wait(0xc208011e90, 0x72, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_poll_runtime.go:84 +0x47
Oct 08 07:53:24 fleetd[127879]: net.(*pollDesc).WaitRead(0xc208011e90, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_poll_runtime.go:89 +0x43
Oct 08 07:53:24 fleetd[127879]: net.(*netFD).readMsg(0xc208011e30, 0xc208260d80, 0x10, 0x10, 0xc208338020, 0x1000, 0x1000, 0xffffffffffffffff, 0x0, 0x0, ...)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_unix.go:296 +0x54e
Oct 08 07:53:24 fleetd[127879]: net.(*UnixConn).ReadMsgUnix(0xc208034090, 0xc208260d80, 0x10, 0x10, 0xc208338020, 0x1000, 0x1000, 0xa48, 0xc2085caab4, 0x4, ...)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/unixsock_posix.go:147 +0x167
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*oobReader).Read(0xc208338000, 0xc208260d80, 0x10, 0x10, 0xc208338000, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/transport_unix.go:21 +0xc5
Oct 08 07:53:24 fleetd[127879]: io.ReadAtLeast(0x7f813ca9a510, 0xc208338000, 0xc208260d80, 0x10, 0x10, 0x10, 0x0, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/io/io.go:298 +0xf1
Oct 08 07:53:24 fleetd[127879]: io.ReadFull(0x7f813ca9a510, 0xc208338000, 0xc208260d80, 0x10, 0x10, 0xa48, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/io/io.go:316 +0x6d
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*unixTransport).ReadMessage(0xc20800bc30, 0xc208043560, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/transport_unix.go:85 +0x1bf
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).inWorker(0xc208001560)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/conn.go:241 +0x58
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).Auth
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/auth.go:118 +0xe84
Oct 08 07:53:24 fleetd[127879]: goroutine 7 [chan receive, 2376 minutes]:
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).outWorker(0xc208001560)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/conn.go:363 +0x58
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).Auth
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/auth.go:119 +0xea1
Oct 08 07:53:24 fleetd[127879]: goroutine 8 [IO wait]:
Oct 08 07:53:24 fleetd[127879]: net.(*pollDesc).Wait(0xc208090370, 0x72, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_poll_runtime.go:84 +0x47
Oct 08 07:53:24 fleetd[127879]: net.(*pollDesc).WaitRead(0xc208090370, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_poll_runtime.go:89 +0x43
Oct 08 07:53:24 fleetd[127879]: net.(*netFD).readMsg(0xc208090310, 0xc20848b4b0, 0x10, 0x10, 0xc208c99220, 0x1000, 0x1000, 0xffffffffffffffff, 0x0, 0x0, ...)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_unix.go:296 +0x54e
Oct 08 07:53:24 fleetd[127879]: net.(*UnixConn).ReadMsgUnix(0xc208034020, 0xc20848b4b0, 0x10, 0x10, 0xc208c99220, 0x1000, 0x1000, 0x79, 0xc20848aed4, 0x4, ...)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/unixsock_posix.go:147 +0x167
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*oobReader).Read(0xc208c99200, 0xc20848b4b0, 0x10, 0x10, 0xc208c99200, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/transport_unix.go:21 +0xc5
Oct 08 07:53:24 fleetd[127879]: io.ReadAtLeast(0x7f813ca9a510, 0xc208c99200, 0xc20848b4b0, 0x10, 0x10, 0x10, 0x0, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/io/io.go:298 +0xf1
Oct 08 07:53:24 fleetd[127879]: io.ReadFull(0x7f813ca9a510, 0xc208c99200, 0xc20848b4b0, 0x10, 0x10, 0x79, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/io/io.go:316 +0x6d
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*unixTransport).ReadMessage(0xc2080a0660, 0xc20805a540, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/transport_unix.go:85 +0x1bf
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).inWorker(0xc2080019e0)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/conn.go:241 +0x58
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).Auth
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/auth.go:118 +0xe84
Oct 08 07:53:24 fleetd[127879]: goroutine 9 [chan receive, 4300 minutes]:
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).outWorker(0xc2080019e0)
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/conn.go:363 +0x58
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus.(*Conn).Auth
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/godbus/dbus/auth.go:119 +0xea1
Oct 08 07:53:24 fleetd[127879]: goroutine 10 [chan receive]:
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/Godeps/_workspace/src/github.com/coreos/go-systemd/dbus.func·001()
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/coreos/go-systemd/dbus/subscription.go:6
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/Godeps/_workspace/src/github.com/coreos/go-systemd/dbus.(*Conn).dispatch
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/Godeps/_workspace/src/github.com/coreos/go-systemd/dbus/subscription.go:9
Oct 08 07:53:24 fleetd[127879]: goroutine 11 [IO wait, 2388 minutes]:
Oct 08 07:53:24 fleetd[127879]: net.(*pollDesc).Wait(0xc2080901b0, 0x72, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_poll_runtime.go:84 +0x47
Oct 08 07:53:24 fleetd[127879]: net.(*netFD).accept(0xc208090150, 0x0, 0x7f813ca98bb0, 0xc2081afdb8)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/fd_unix.go:419 +0x40b
Oct 08 07:53:24 fleetd[127879]: net.(*UnixListener).AcceptUnix(0xc20801ea00, 0x437d70, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/unixsock_posix.go:282 +0x56
Oct 08 07:53:24 fleetd[127879]: net.(*UnixListener).Accept(0xc20801ea00, 0x0, 0x0, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/unixsock_posix.go:293 +0x4c
Oct 08 07:53:24 fleetd[127879]: net/http.(*Server).Serve(0xc20805a4e0, 0x7f813ca9a9d8, 0xc20801ea00, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/http/server.go:1728 +0x92
Oct 08 07:53:24 fleetd[127879]: net/http.Serve(0x7f813ca9a9d8, 0xc20801ea00, 0x7f813ca9ad00, 0xc208032640, 0x0, 0x0)
Oct 08 07:53:24 fleetd[127879]: /usr/lib/go/src/net/http/server.go:1606 +0xa1
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/api.func·001()
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/api/server.go:49 +0x77
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/api.(*Server).Serve
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/api/server.go:53 +0x14d
Oct 08 07:53:24 fleetd[127879]: goroutine 12 [chan send, 4299 minutes]:
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/etcd.func·004()
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/etcd/client.go:246 +0x137
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/etcd.(*client).Do
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/etcd/client.go:247 +0x1ed
Oct 08 07:53:24 fleetd[127879]: goroutine 50 [chan send, 4299 minutes]:
... snip snip ...
Oct 08 07:53:24 fleetd[127879]: goroutine 1784912 [chan send]:
Oct 08 07:53:24 fleetd[127879]: github.com/coreos/fleet/etcd.func·004()
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/etcd/client.go:246 +0x137
Oct 08 07:53:24 fleetd[127879]: created by github.com/coreos/fleet/etcd.(*client).Do
Oct 08 07:53:24 fleetd[127879]: /build/amd64-usr/var/tmp/portage/app-admin/fleet-0.10.2-r1/work/gopath/src/github.com/coreos/fleet/etcd/client.go:247 +0x1ed