fix(agents): make run liveness evidence-based#101985
Conversation
|
Codex review: needs maintainer review before merge. Reviewed July 8, 2026, 8:02 AM ET / 12:02 UTC. Summary PR surface: Source +87, Tests +395. Total +482 across 14 files. Reproducibility: yes. Current main source still emits synthetic Review metrics: 1 noteworthy metric.
Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Maintainer decision needed
Security Review detailsBest possible solution: Keep this per-track PR open as the Track B review record, then land or close it through #102160 only after the integration branch is fixed, re-reviewed, and accepted by maintainers. Do we have a high-confidence way to reproduce the issue? Yes. Current main source still emits synthetic Is this the best way to solve the issue? Yes, with a merge-sequencing caveat. Removing synthetic liveness and routing stale runs through the reply-run and embedded-run owner paths is the maintainable fix for this track, but it should land through the accepted Track A or integration path rather than independently. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 83ebbcb3acf7. Label changesLabel justifications:
Evidence reviewedPR surface: Source +87, Tests +395. Total +482 across 14 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
Review history (12 earlier review cycles; latest 8 shown)
|
|
P1 fixed in f880b20: Live behavior proof is running now on this exact branch (same harness as the #101910 and #101986 E2E comments): wedged run → 10-min stale window → channel msg refused by steer capture ( On sequencing: agreed and stated in the PR body — this is stacked on #101910 (its base branch) and must not merge independently. |
f880b20 to
1eb76b6
Compare
Live behavior proof (requested by review) — and one more bug it caught, now fixedSame harness as the #101910 / #101986 E2E comments: local gateway from this branch's dist, black-hole model endpoint holding a genuine wedge open (explicit provider Artifact check — fake heartbeat gone
Proof 1 — stale steer refusal + takeover: PASS (flips the exact failure from the #101910 E2E)
Proof 2 — recovery attribution: FAILED live, root-caused, fixedDefault stuck recovery aborted the wedge at 374s and freed the lane, but the operation still terminated as
Also in this push: the review's P1 ( Validation after fixes: registry/admission/runs/recovery/announce suites green, tsgo core + core-test lanes exit 0, branch rebased onto the updated #101910 base. Full timestamped evidence in the harness report (request ledger, log lines); happy to attach the raw report if wanted. |
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
1eb76b6 to
b14ff32
Compare
36536d8 to
ef2678f
Compare
b14ff32 to
4ff1246
Compare
4ff1246 to
0f817a6
Compare
ef2678f to
c40447e
Compare
0f817a6 to
a98b238
Compare
5b4ae6c to
a075205
Compare
ae9081d to
aa874c4
Compare
a075205 to
348c468
Compare
|
Landing plan update: this track lands atomically with its siblings via the integration PR #102160 (all commits preserved, rebase merge) — sequential landing would put untested intermediate states on main (e.g. the stale takeover without the quiet-tool floor). This PR stays open as the per-track review record and closes when #102160 lands. |
What Problem This Solves
Track B (sub-tracks B1/B4/B5) of #101863 — the "truthful liveness" half of the run-lifecycle reliability design. Stacked on #101910 (Track A) and uses its primitives (
expireStaleReplyOperation,run_stalled, activity stamps).Three verified defects:
startClaudeLiveActiveToolHeartbeatemittedcli_live:tool_runningon a 10ssetIntervalwhile a tool was merely marked active, resetting the sharedlastProgressAtclock with zero evidence the CLI child was alive. Theblocked_tool_callclassifier — which correctly requires BOTH tool age AND progress age to be stale — could structurally never fire for a wedged CLI tool, and neither could any stuck-recovery abort gate.operation.abortByUser(), so watchdog kills surfaced asaborted_by_user. And any truthful-evidence fix would expose quiet-but-alive tools to the generic ~6-min abort threshold. Stuck-session liveness monitor misclassifies healthy processing runs as stale during event-loop stalls #101670 — staleness ages are rawDate.now()deltas, so an event-loop stall inflates every age at the next heartbeat tick and trips spurious recovery.Why This Change Was Made
blocked_tool_callrecovery eligibility gains a floor:max(stuckSessionAbortMs, BLOCKED_TOOL_CALL_ABORT_FLOOR_MS = 15 min). Early warnings (naming the tool) keep current thresholds; the CLI byte-level no-output watchdog (evidence-based, ≤600s for truly-silent children) is untouched.expireStaleReplyOperation(…, "stuck_recovery")→ terminal result{failed, run_stalled}instead ofaborted_by_user. The diagnostic heartbeat records its own tick time; a tick delayed >3× the interval defers all recovery decisions to the next on-time tick (a late tick means the process stalled, not the sessions).queueReplyRunMessagewhen the operation'slastActivityAtMsexceeds the stale-takeover window,prepareEmbeddedAgentQueueMessagewhen diagnosticlastProgressAgeMsexceeds the paired constant (new closed failure reasonstale_run; consumers fall through likeno_active_run). Refused messages take the normal followup-admission path where Track A's takeover can reclaim. Draining already-captured entries from dead handles stays Track C (durability).User Impact
Evidence
run_stalled(user aborts stillaborted_by_user); delayed-tick guard (fake timers); both steer gates (stale refused → fall-through, fresh accepted, no-snapshot skip);stale_runconsumer fall-through.tsgocore/core-test lanes green on Testbox; re-run green after review edits.reason=stale_runsteer refusal on a 16-min-stale lane (steer-accepted unconditionally on main); channel-turn takeover 34 ms after a rescue message on a 10-min wedge with the turn re-running on a fresh model call; recovery-attribution race found live and fixed (expire-ordering on the fix(auto-reply): bound terminal reply-run settle and reclaim stale lanes #101910 base + live-handle recovery leg here), both with regression tests.Part of #101863 (Tracks B1/B4/B5). Fixes #96168; addresses the abort-attribution and quiet-tool halves of #88870 and the heartbeat half of #101670. Stacked on #101910 — do not merge before it.