I have seen a few times that the swarm dispatcher is gone and the swarm cluster stopped working. The only way to recover is restarting the manager nodes.
I happen to have a stacktrace of the leader node captured at the problem time, the docker version is 19.03.12.
daemon-stack-trace.zip
In the stacktrace, there's no dispatcher stack, but there's a stack becomeFollower, this is abnormal. It is blocked in (*LogBroker).Stop
goroutine 613 [semacquire, 747 minutes]:
sync.runtime_SemacquireMutex(0xc000b1a184, 0x0, 0x1)
/.GOROOT/src/runtime/sema.go:71 +0x49
sync.(*Mutex).lockSlow(0xc000b1a180)
/.GOROOT/src/sync/mutex.go:138 +0xfe
sync.(*Mutex).Lock(...)
/.GOROOT/src/sync/mutex.go:81
sync.(*RWMutex).Lock(0xc000b1a180)
/.GOROOT/src/sync/rwmutex.go:98 +0x99
github.com/docker/docker/vendor/github.com/docker/swarmkit/manager/logbroker.(*LogBroker).Stop(0xc000b1a180, 0x0, 0x0)
/root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/swarmkit/manager/logbroker/broker.go:78 +0x3b
github.com/docker/docker/vendor/github.com/docker/swarmkit/manager.(*Manager).becomeFollower(0xc000845860)
/root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/swarmkit/manager/manager.go:1111 +0x4d
Below code means if becomeFollower() is blocked, then becomeLeader() has no chance to run when the node is re-elected as the leader. Thus dispatcher will not be run.
// handleLeadershipEvents handles the is leader event or is follower event.
func (m *Manager) handleLeadershipEvents(ctx context.Context, leadershipCh chan events.Event) {
// get the current leader and save it for logging leadership changes in
// this loop
oldLeader := m.getLeaderNodeID()
for {
select {
case leadershipEvent := <-leadershipCh:
m.mu.Lock()
if m.stopped {
m.mu.Unlock()
return
}
newState := leadershipEvent.(raft.LeadershipState)
if newState == raft.IsLeader {
m.becomeLeader(ctx)
leaderMetric.Set(1)
} else if newState == raft.IsFollower {
m.becomeFollower()
leaderMetric.Set(0)
...
I have seen a few times that the swarm dispatcher is gone and the swarm cluster stopped working. The only way to recover is restarting the manager nodes.
I happen to have a stacktrace of the leader node captured at the problem time, the docker version is
19.03.12.daemon-stack-trace.zip
In the stacktrace, there's no
dispatcherstack, but there's a stackbecomeFollower, this is abnormal. It is blocked in(*LogBroker).StopBelow code means if
becomeFollower()is blocked, thenbecomeLeader()has no chance to run when the node is re-elected as the leader. Thusdispatcherwill not be run.