Repository navigation
Commit d331f11
authored
server: release mutex while waiting for handlers in stop() (#9394)
Fixes #9393
`Stop()` deadlocks when called after `GracefulStop()` while a handler is
still running.
#### Root cause
Both `Stop()` and `GracefulStop()` call `stop()`, which acquires `s.mu`
and holds it for the entire function, including across
`s.handlersWG.Wait()`. If a handler ignores context cancellation,
`GracefulStop()` blocks in that wait while still holding `s.mu`. A
concurrent `Stop()` (the documented way to abort a `GracefulStop()` that
is taking too long) then blocks forever on `s.mu.Lock()` and can never
force the server to shut down.
#### Fix
Release `s.mu` around `s.handlersWG.Wait()` and reacquire it afterwards,
mirroring the connection-drain loop that already releases the lock via
`s.cv.Wait()`.
This is safe because none of the state guarded by `s.mu` is accessed
during the wait: by that point listeners are closed, all transports are
drained/closed, `s.conns` is `nil`, and the worker channel is closed.
`handlersWG.Wait()` only touches the `WaitGroup`, which has its own
synchronization and was never guarded by `s.mu`. A concurrent `Stop()`
can now acquire the lock and return promptly.
#### Testing
Added `TestStopAfterGracefulStopWithRunningHandler`, which starts a
handler that ignores context cancellation, closes the client connection
so the transport drains, runs `GracefulStop()` in a goroutine, and
asserts that a subsequent `Stop()` returns promptly. Verified the test
times out (deadlocks) without the fix and passes with it.
RELEASE NOTES:
* server: fix deadlock where `Stop()` blocks indefinitely when called
concurrently with an in-progress `GracefulStop()` waiting on a handler
that does not return promptly1 parent d41d9b3 commit d331f11
2 files changed
Lines changed: 87 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1719 | 1719 | | |
1720 | 1720 | | |
1721 | 1721 | | |
| 1722 | + | |
| 1723 | + | |
| 1724 | + | |
| 1725 | + | |
| 1726 | + | |
1722 | 1727 | | |
| 1728 | + | |
1723 | 1729 | | |
1724 | 1730 | | |
1725 | 1731 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
301 | 301 | | |
302 | 302 | | |
303 | 303 | | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
304 | 385 | | |
305 | 386 | | |
306 | 387 | | |
| |||
0 commit comments