fix(agent): make stuck recovery phase-safe#93655
Conversation
|
Codex review: needs maintainer review before merge. Reviewed July 16, 2026, 11:45 PM ET / July 17, 2026, 03:45 UTC. Summary PR surface: Source +227, Tests +804. Total +1031 across 26 files. Reproducibility: yes. at source level. Current main accepts Review metrics: none identified. Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Risk before merge
Maintainer options:
Next step before merge
Maintainer decision needed
Security Review detailsBest possible solution: Land one canonical generic recovery path that preserves Do we have a high-confidence way to reproduce the issue? Yes, at source level. Current main accepts Is this the best way to solve the issue? Yes. Classifying at the active attempt is the narrowest maintainable fix because it preserves the existing fallback policy while keeping side-effecting tools, compaction, and unknown phases outside automatic replay; provider-specific stream work does not solve this generic downstream path. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 225638788572. Label changesLabel justifications:
Evidence reviewedPR surface: Source +227, Tests +804. Total +1031 across 26 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
Review history (19 earlier review cycles; latest 8 shown)
|
|
@clawsweeper re-run |
|
Holding this one rather than landing it as-is. Mapping every stuck-run recovery to an idle timeout loses the distinction between a stalled model call and a stalled tool call, and retrying the latter can replay side effects. The best fix must classify the stuck phase first and only retry states whose replay semantics are safe, with regression proof for model-call and side-effecting tool-call stalls. |
|
Updated this PR to address the replay-safety concern. What changed:
Regression proof added/updated:
Fresh local verification on the current PR head
I did not run a live opencode-go/deepseek-v4-flash stream or live cron job; the PR body now calls out that boundary explicitly. |
|
@vincentkoc I updated this PR to address your replay-safety feedback. |
74eb3d5 to
5d36d5c
Compare
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
This assigned pull request has been automatically marked as stale after being open for 27 days. |
|
This assigned pull request has been automatically marked as stale after being open for 27 days. |
|
This assigned pull request has been automatically marked as stale after being open for 27 days. |
What Problem This Solves
Fixes #103697.
Fixes an issue where users with configured model fallbacks could still receive an aborted reply when the diagnostic watchdog recovered a stuck model call. Recovery also needs to preserve the safety boundary that keeps potentially side-effecting tools and unknown phases out of automatic replay, while keeping queued turns serialized behind the recovering reply.
Why This Change Was Made
Diagnostic recovery now distinguishes cancelling the currently attached embedded attempt from cancelling the whole reply. Previously the shared caller signal was already aborted before
runWithModelFallback()could start its next candidate. A running embedded backend now receivesstuck_recoveryexactly once while the reply caller signal stays live; when the attempt drains, the next fallback backend can attach to the same operation. The watchdog also re-checks combined embedded/reply ownership before resetting the session command lane, so queued turns cannot run concurrently with fallback. Active model calls become idle timeouts, potentially side-effecting tool calls remain tool-execution timeouts, compaction stalls remain compaction timeouts, and unknown phases remain external aborts.The recovery window is bounded and does not weaken real cancellation. User and restart aborts still terminate the whole reply, queued/non-embedded recovery remains terminal, and a backend owner that does not drain within the diagnostic 15-second settle window is force-cleared after the whole reply is terminated; the registry terminal-settle timer remains a longer backstop.
This is the downstream generic recovery fix for the core in-tree embedded attempt path, independent of whichever provider or transport caused progress to stall. Alternate plugin-owned agent harnesses, including Codex and Copilot, retain their owner-specific cancellation semantics and are outside this PR. It does not change provider contracts, model routing, fallback selection policy, public configuration, schemas, persistence, channels, or ordinary
no_activityexpiry behavior.User Impact
Stuck sessions can recover without losing the safety distinction between model work and tool work:
Evidence
The evidence below covers the production-module behavior proof, focused regression suites, and static verification.
Current-head refresh
mainat71c5005d2c57988b045b2c6671489f75d3a35dc8without rewriting the PR history.run-state.tsconflict by retaining main'ssupportsQueueMessageImagescapability and this PR's typedstuck_recoveryabort semantics.git diff --check, final diff review, and the daily-fix pre-submit guard passed locally. Repository trust policy prevents executing fork test scripts locally; exact-head CI will run after this branch update.Real behavior proof
Behavior or issue addressed: A reply-owned active model attempt recovered by the diagnostic watchdog must receive one
stuck_recoverycancellation, drain successfully, keep the whole-reply caller signal and session command lane live, and start the next configured fallback candidate; a queued turn must wait until fallback completes, while tool and unknown phases remain non-replayable.Real environment tested: Linux with supported Node 24.15.0. The production-module probe ran on hosted behavior head
24f776d6f7e1263b3bf3166deb26a2ea66ce3a89; the final CI-only tail was validated on committed HEADe580471dd073ba3b7744fc00272cc351c9d2e9ea. The probe loaded the production diagnostic recovery, embedded-run registry, reply registry, attempt abort classifier, and model fallback modules; the embedded backend handle and active phase ports were controlled in-process fixtures.Exact steps or command run for the behavior patch: Ran
node --import tsx canonical-reply-owned-stuck-recovery-probe.mjson hosted head24f776d. The probe registers a reply-owned active embedded model attempt inside a real session command lane, queues a second turn behind it, invokes the production diagnostic watchdog, routes its timeout result through the production model fallback loop, and records lane ownership before the configured fallback succeeds. The finale580471dtail only removes two unused type exports, applies twoconstlint fixes in the integration fixture, and gives the existingsetImmediatepromise executor a block body; the logging owner-config runtime/integration files were rerun on that final tail (2 files, 33 tests passed).Evidence after fix:
Observed result after fix: Recovery returned
aborted=true,drained=true,forceCleared=false, andreleased=0. During recovery the reply remained active withresult=nulland an un-aborted caller signal. The command lane stayed at one active reply plus one queued turn throughout fallback, and the queued turn started only afterfallback-completed. The production fallback loop attempted the primary and fallback candidates, completed withfallback ok, and the original reply completed normally. The first and only backend cancellation wasstuck_recovery; model, side-effecting-tool, and unknown phases classified asmodel_idle_timeout,tool_execution_timeout, andexternal_abort.What was not tested: No live OpenCode/provider stream, external channel delivery, real model stream, or real tool invocation was exercised. The continuous recovery-to-fallback path used production modules with controlled model candidates, backend handle, and phase ports. This patch does not change or claim a provider/channel boundary.
Regression and static verification
tsgochecks passed.git diff --checkpassed against the synchronized base.oxlintand deadcode unused-export checks (Knipbaseline matched all 626 entries).Risk boundary
The behavior change is restricted to cancellation reason propagation, reply ownership, and command-lane release after diagnostic recovery has already decided a running embedded attempt is stale. Ordinary stale expiry continues to use
superseded; CLI/non-embedded recovery remains terminal; user/restart cancellation still terminates the whole reply; public configuration, provider contracts, schemas, persistence, and fallback selection policy are unchanged.