Repository navigation
fix(doctor): join the stderr drain before building the crash detail - #3165
Merged
swati510 merged 1 commit intoOct 10, 2026
Merged
Conversation
The MCP smoke check read its stderr ring the moment the child's exit code was known, but the reader thread may not have consumed the pipe's last lines yet. On a busy runner the ring was still empty then, so a server that died on import was reported as 'server exited with code 1' with the traceback lost - the intermittent test_the_registered_command_is_the_one_ launched CI failure. _drain_stderr now returns the reader thread alongside the ring, and the detail joins it (bounded by _STDERR_DRAIN_GRACE_S, 2s) before reading the tail. The pipe hits EOF when the child exits, so the join returns as soon as the last line lands; the bound keeps a grandchild holding the pipe open from stalling doctor. Two regressions: a slowed drain (the loaded-runner interleaving) still reports ModuleNotFoundError, and a reader that never finishes delays the detail by the grace period only. Closes repowise-dev#3138
Collaborator
|
Thanks @Pushpak731, nice catch on the race: the process exiting doesn't mean the reader has drained the pipe, so joining the drain thread before building the crash detail is the right fix. This flake hit about a third of our CI runs this week, so it unblocks a lot. Merging |
swati510
approved these changes
Oct 10, 2026
Contributor
Author
|
Thank you @swati510 — glad the race write-up landed the fix cleanly. Appreciate the fast review and merge! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3138
Problem
_drain_stderrstarted a daemon reader thread and returned only the ring. After the handshake failed and the exit code was known,mcp_smoke_checkread the ring immediately — but the process having exited does not mean the reader has consumed the pipe's last lines. On a busy runner the ring could still be empty at that point, so a server that died on import was reported as justserver exited with code 1and the traceback naming the fault was lost. This is the intermittenttest_the_registered_command_is_the_one_launchedCI failure quoted in the issue (Python 3.11 job; passed on 3.12/3.13 and on rerun).Reproduced deterministically
Patching the ring's
appendtotime.sleep(0.5)(the loaded-runner interleaving the issue describes — child dead and reaped while the reader is mid-line) makes the current code lose the tail:Fix
_drain_stderrreturns(ring, thread)instead of dropping the thread.exit_codeis known, the detail joins the reader with_STDERR_DRAIN_GRACE_S = 2.0before_stderr_tail. The pipe hits EOF when the child exits, so the join returns as soon as the last line lands; the bound keeps a grandchild that inherits and holds the pipe open from stalling doctor.Tests
test_a_slow_stderr_drain_still_reports_the_tail— slowed ring append (the race interleaving) still surfacesModuleNotFoundError; fails on the previous code (verified by mutation).test_a_grandchild_holding_the_pipe_cannot_stall_doctor— a reader that never finishes delays the detail by the grace period only, not forever.tests/unit/clirun fail identically on the pristine tree (local-environment only).