Skip to content

fix: reconcile Linux zombie runners in the same PID namespace - #2606

Merged
nicobailon merged 3 commits into
nicobailon:mainfrom
quifox:fix/zombie-pid-liveness
Oct 1, 2026
Merged

nicobailon merged 3 commits into
nicobailon:mainfrom
quifox:fix/zombie-pid-liveness

Conversation

@quifox

@quifox quifox commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

On Linux, kill(pid, 0) succeeds for a zombie runner, leaving a recent run marked running until the stale threshold. After a successful PID probe, inspect /proc//stat for state Z only when the recorded PID namespace and the observer's namespace are both present and equal. Runs with missing or mismatched namespace identity retain the existing stale threshold. Procfs read failures retain the signal-probe fallback, and EPERM remains unknown.

The generic liveness helper keeps signal-only probing by default. Reconciliation and workflow reattachment explicitly enable zombie detection for verified namespaces, so an exited runner without a result cannot keep a reattached workflow waiting. Unscoped PID checks retain their existing behavior.

The regressions create real Linux zombies whose comm contains a closing parenthesis. They check immediate repair and read-only reattachment exit detection for matching namespaces, conservative handling for missing/mismatched identities, stale repair for uncertain reconciliation cases, and stable completion results. Missing/mismatched reattachment cases still accept a later published result. Namespace variations use controlled status/observer fixtures; the tests do not launch a live runner in a separate namespace. The Unreleased changelog credits @quifox and #2606.

Validation on Linux with Node v24.19.0:

  • Stale-run reconciliation and existing-async-run await suites: 25 passed.
  • Production typecheck: passed.
  • The missing-recorded-namespace assertion fails against the previous PR implementation and passes after this change. The matching-namespace zombie reattachment case times out against the first follow-up and completes as unavailable after the await probe is enabled.
  • Previously measured alive-PID procfs probe cost: median 4.03 us/check versus 0.19 us for a signal-only probe over seven trials of 10,000 checks. The same procfs read is now restricted to verified namespaces; this is not an end-to-end TUI measurement.

CI and automated review must be checked on the latest PR head.

@greptile-apps

greptile-apps Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium risk] Changes how the system detects dead runner processes on Linux.

No outstanding code finding was identified, but merge readiness cannot be stated until CI on the exact PR head is verified.

Summary

The PR adds namespace-verified Linux zombie detection to stale-run reconciliation and workflow reattachment while leaving unscoped PID checks signal-only.

Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Active run with recorded PID] --> B{Recorded and observer namespaces match?}
    B -- Yes --> C[Signal probe, then Linux zombie probe]
    B -- No or unknown --> D[Signal-only probe]
    C --> E{Runner dead?}
    D --> E
    E -- Yes --> F[Repair run or end reattachment without an unpublished result]
    E -- No or uncertain --> G[Retain existing wait and stale-threshold behavior]
Loading

Reviews (4) · Last reviewed commit: "fix: detect scoped zombie runners during..."

Comment thread test/unit/stale-run-reconciler.test.ts Resolved
@quifox quifox changed the title fix: recognize Linux zombie runner PIDs as dead fix: reconcile Linux zombie runners in the same PID namespace Oct 1, 2026
@nicobailon
nicobailon force-pushed the fix/zombie-pid-liveness branch from bd22b33 to a7862e4 Compare October 1, 2026 06:29
@nicobailon
nicobailon merged commit 806e367 into nicobailon:main Oct 1, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants