Dump wedged-worker stacks with a faulthandler watchdog in CI#592
Merged
Conversation
pytest-timeout is configured but on Windows runs its thread method: a threading.Timer whose callback is Python. When a worker wedges holding the GIL (the ProactorEventLoop / reactive-timing thread-accumulation case), no Python runs, the callback never fires, and the job hangs silently until timeout-minutes kills it with no traceback. faulthandler.dump_traceback_later runs a C watchdog thread that dumps every thread without acquiring the GIL, so it fires where pytest-timeout cannot. exit=True then _exits the wedged worker, so a hang fails fast with a traceback instead of burning the job budget. Opt-in via LILBEE_TEST_HANG_DUMP_S, set above the 60s per-test pytest-timeout so it is only ever the backstop. CI surfaces the per-worker dump in a step that runs even when the test step was killed. Verified locally: dumps the exact frozen frame and _exits the worker for both a GIL-releasing sleep and a GIL-holding busy loop; no-op when the var is unset; no misfire on fast tests under xdist.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
timeout-minuteskills it, with no traceback.Solution
faulthandlerwatchdog (tests/_hang_watchdog.py, gated onLILBEE_TEST_HANG_DUMP_S), armed per test above the 60s pytest-timeout. faulthandler's C watchdog thread dumps every thread without acquiring the GIL, so it fires where pytest-timeout can't;exit=Truethen_exits the worker so the run fails fast with a traceback (exit 1 serial, 3 under xdist) instead of hanging.if: always()step, since the wedged worker can't print its own.Diagnostic only, no-op when the var is unset. Verified locally: dumps the exact frozen frame for a GIL-releasing sleep and a GIL-holding busy loop; no misfire on fast tests under xdist.