Skip to content

fix(doctor): join the stderr drain before building the crash detail - #3165

Merged
swati510 merged 1 commit into
repowise-dev:mainfrom
Pushpak731:fix/doctor-mcp-stderr-race
Oct 10, 2026
Merged

swati510 merged 1 commit into
repowise-dev:mainfrom
Pushpak731:fix/doctor-mcp-stderr-race

Conversation

@Pushpak731

Copy link
Copy Markdown
Contributor

Closes #3138

Problem

_drain_stderr started a daemon reader thread and returned only the ring. After the handshake failed and the exit code was known, mcp_smoke_check read the ring immediately — but the process having exited does not mean the reader has consumed the pipe's last lines. On a busy runner the ring could still be empty at that point, so a server that died on import was reported as just server exited with code 1 and the traceback naming the fault was lost. This is the intermittent test_the_registered_command_is_the_one_launched CI failure quoted in the issue (Python 3.11 job; passed on 3.12/3.13 and on rerun).

Reproduced deterministically

Patching the ring's append to time.sleep(0.5) (the loaded-runner interleaving the issue describes — child dead and reaped while the reader is mid-line) makes the current code lose the tail:

detail: server exited with code 1          # ModuleNotFoundError gone

Fix

  • _drain_stderr returns (ring, thread) instead of dropping the thread.
  • After exit_code is known, the detail joins the reader with _STDERR_DRAIN_GRACE_S = 2.0 before _stderr_tail. The pipe hits EOF when the child exits, so the join returns as soon as the last line lands; the bound keeps a grandchild that inherits and holds the pipe open from stalling doctor.

Tests

  • test_a_slow_stderr_drain_still_reports_the_tail — slowed ring append (the race interleaving) still surfaces ModuleNotFoundError; fails on the previous code (verified by mutation).
  • test_a_grandchild_holding_the_pipe_cannot_stall_doctor — a reader that never finishes delays the detail by the grace period only, not forever.
  • All 13 existing smoke tests unchanged and passing; 26 adjacent doctor tests passing. The 5 failures in the wider tests/unit/cli run fail identically on the pristine tree (local-environment only).

The MCP smoke check read its stderr ring the moment the child's exit code
was known, but the reader thread may not have consumed the pipe's last
lines yet. On a busy runner the ring was still empty then, so a server
that died on import was reported as 'server exited with code 1' with the
traceback lost - the intermittent test_the_registered_command_is_the_one_
launched CI failure.

_drain_stderr now returns the reader thread alongside the ring, and the
detail joins it (bounded by _STDERR_DRAIN_GRACE_S, 2s) before reading the
tail. The pipe hits EOF when the child exits, so the join returns as soon
as the last line lands; the bound keeps a grandchild holding the pipe
open from stalling doctor.

Two regressions: a slowed drain (the loaded-runner interleaving) still
reports ModuleNotFoundError, and a reader that never finishes delays the
detail by the grace period only.

Closes repowise-dev#3138
Copilot AI balanced review requested due to automatic review settings October 9, 2026 10:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@swati510

Copy link
Copy Markdown
Collaborator

Thanks @Pushpak731, nice catch on the race: the process exiting doesn't mean the reader has drained the pipe, so joining the drain thread before building the crash detail is the right fix. This flake hit about a third of our CI runs this week, so it unblocks a lot. Merging

@swati510
swati510 merged commit f540111 into repowise-dev:main Oct 10, 2026
10 checks passed
@Pushpak731

Copy link
Copy Markdown
Contributor Author

Thank you @swati510 — glad the race write-up landed the fix cleanly. Appreciate the fast review and merge!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

doctor's MCP smoke check can report a crashed server without its stderr

3 participants