fix(auto-reply): bound terminal reply-run settle and reclaim stale lanes#101910
fix(auto-reply): bound terminal reply-run settle and reclaim stale lanes#101910obviyus wants to merge 5 commits into
Conversation
|
Codex review: needs maintainer review before merge. Reviewed July 8, 2026, 8:03 AM ET / 12:03 UTC. Summary PR surface: Source +113, Tests -97. Total +16 across 19 files. Reproducibility: yes. at source/proof level: current main waits on active reply runs without stale-running takeover, and the PR discussion includes live black-hole endpoint proof that abort released the lane and a later turn admitted. I did not run tests in this read-only review. Review metrics: 1 noteworthy metric.
Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Maintainer decision needed
Security Review detailsBest possible solution: Keep this PR open as the Track A review record and land or reject the lifecycle stack through #102160, which includes this branch atomically. Do we have a high-confidence way to reproduce the issue? Yes at source/proof level: current main waits on active reply runs without stale-running takeover, and the PR discussion includes live black-hole endpoint proof that abort released the lane and a later turn admitted. I did not run tests in this read-only review. Is this the best way to solve the issue? Yes for the Track A code shape, but not as the best standalone landing path; the safer maintainer path is the atomic integration PR because sibling liveness work protects this branch's new recovery defaults. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 4bf70be01a21. Label changesLabel justifications:
Evidence reviewedPR surface: Source +113, Tests -97. Total +16 across 19 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
Review history (17 earlier review cycles; latest 8 shown)
|
c867aa9 to
1888bf5
Compare
Live E2E validation (wedge/reclaim)Ran the real root-cause scenario on this branch: local gateway ( Test A — abort ⇒ bounded release: PASS
Note: release was immediate rather than via the 60s settle timer, because the abort signal propagates into the guarded model fetch and the owner unwinds cleanly (the armed settle timer is cancelled). The forced-release path is the backstop for owners that never unwind post-abort — e.g. the Test B — silent-stale takeover: works for its target shape; live-runtime wedges never reach admission
Follow-up candidates surfaced (out of scope here, feeding #101863 Track B)
Cleanup verified: no gateway/black-hole/QA-bus processes left, scratch state only, |
|
Addressed the P1 in 4d4df31 — structurally, not with a one-line patch. The finding was real (queued-followup Coverage: two new tests in Acceptance criteria run (worktree, On the maintainer decision (10-minute no-activity takeover default): intentionally left open — the PR stays in draft until the maintainer picks accept / narrow / hold-for-Track-B. One datum for that call from the live E2E: for wedges with a live steerable runtime, inbound messages are captured by the steer queue before admission ever evaluates takeover, so the 10-minute policy in practice reclaims dead-owner shapes, not quiet-but-alive runs — the false-positive window is narrower than source inspection suggests. |
|
Maintainer decision (@obviyus): accept the 10-minute zero-activity visible-turn takeover as default policy, with the stamp-seam repair already landed (4d4df31). Rationale: the takeover requires a human actively waiting plus zero real activity evidence for 10 minutes — and the live E2E on this branch showed that runs with a live steerable runtime capture inbound messages in the steer queue before admission evaluates takeover at all, so the policy in practice reclaims dead-owner shapes. Quiet-but-alive false positives are tracked under #101863; Track B's phase-aware liveness (per-phase floors, reasoning-model allowances) is the designated refinement, not a merge blocker here. Remaining before undraft: none from this review. The PR stays in draft per maintainer preference until Track A validation on a production bot is scheduled. |
36536d8 to
ef2678f
Compare
ef2678f to
c40447e
Compare
|
Both re-review P1s addressed in
Focused suites, surface pin, tsgo core/core-test lanes all green; stacked branch #101985 rebased on the new head. |
ae9081d to
aa874c4
Compare
|
Landing plan update: this track lands atomically with its siblings via the integration PR #102160 (all commits preserved, rebase merge) — sequential landing would put untested intermediate states on main (e.g. the stale takeover without the quiet-tool floor). This PR stays open as the per-track review record and closes when #102160 lands. |
What Problem This Solves
Track A of #101863: a reply run that hangs (owner promise never settles) permanently wedges its session today.
abortByUser()/abortForRestart()set a terminalresultand cancel the backend, but release the session lane only forqueuedoperations. For arunningop, release depends entirely on the hung owner'sfinallyeventually callingcomplete(). Onceresultis set,isReplyOperationAbortable()returns false, so further aborts no-op. This is therelease_lane released=0class ([Bug]: Stuck-session recovery never releases: status=aborted with released=0 on every event — wedges survive five correct-target aborts (~14h) and four ghost-target aborts (~16h) until an external restart or user /reset [2026.5.28] #92270, Conversational agent lane wedges indefinitely: ingress events never reconcile (re-delivery storm on reconnect) + abort_embedded_run leaves owner lock unreleased (released=0) #97538, [Bug]: Session write-lock held beyond run lifetime when agent run fails during tool execution #100872, Isolated cron lanes leak on claude-cli backend: queued_work_without_active_run → release_lane released=0, lanes accumulate until restart #89766): the run is dead but admission returnsactive-runforever, and every later message queues silently until gateway restart.ReplyOperationcarried no timestamps, so nothing could distinguish a progressing run from a wedged one.terminalRecovery) covers already-terminal store snapshots, never a never-returningrunningop.dispatchReplyFromConfigcompensated with a caller-side recovery loop that invoked the stuck-session-recovery machinery on config-derived wall-clock thresholds — the same heuristics reported for killing healthy long runs (Stuck-session recovery aborts long-but-active agent runs at warnMs×3 (~6min) with misleading "Reply operation aborted by user" reason #88870).Why This Change Was Made
Establishes the Track A invariant from #101863: reaching a terminal state releases the lane, bounded, no matter what the owner does — and staleness is judged by real activity evidence, not wall clock.
reply-run-registry.ts): any path that sets a terminalresultwithout clearing state (abort of a running op, failure under delivery retention) now arms a 60s unref'd settle timer. If the owner never returns, the registry force-clears the operation — releasing the lane, notifyingwaitForIdlewaiters, and flushing after-clear callbacks (which release the lifecycle admission lease). A late ownercomplete()stays a harmless no-op.startedAtMs/lastActivityAtMs/recordActivity(), stamped at phase transitions, backend attach, session rotation, steer-queue injection, and the existing progress chokepoints (assistant text, reasoning progress, tool events, embedded lifecycle events) on both embedded and CLI paths. Never from timers — this is the seam Track B extends.reply-turn-admission.ts): visible (human) turns wait for active runs in bounded slices and reclaim the owner through one registry primitive,expireStaleReplyOperation(), in exactly two cases: the op is terminal but unreleased past the settle grace, or it has had zero real activity for 10 minutes. The expired op gets the new closed failure coderun_stalled. Heartbeat, queued-followup, and control-abort turns never take over.dispatch-from-config.ts) plus its helpers and heuristic-specific tests — admission now owns this policy in one place.Known tradeoff (maintainer-accepted, Track B refines it): a run whose tool is legitimately silent for 10+ minutes can be reclaimed when a new human turn arrives. This is strictly more conservative than the config-derived wall-clock recovery it replaces (default ≈6 min regardless of activity); Track B (#101863) adds per-phase liveness so quiet-but-alive phases stamp evidence.
User Impact
/stop(or any abort) frees the lane within 60s even if the underlying run never unwinds, and a new message reclaims a silently dead run after 10 minutes instead of never.Evidence
complete()harmless after forced release; retained-failure ops still bounded; visible-turn reclaim of silent-stale and terminal-unreleased ops; fresh ops kept (waiting turn does not steal);heartbeat/queued_followupnever reclaim; settle timer cancelled when the owner clears first.dispatch-from-config.stale-recovery.test.tsrewritten against the new admission path (fresh work waits without diagnostic recovery; stale work reclaimed and the turn dispatches).reply-run-registry/reply-turn-admission/ReplyOperationFailureCode(5 shards), andtsgocore lanes — all green on Testbox; focused suites re-run green after review edits and rebase onto currentmain.runCliAgentWithLifecyclebridge seam viaonActivity; per-callback stamps deleted at both call sites.Part of #101863 (Track A). Touches #92270 #97538 #100872 #89766 (do not auto-close; verified against live reports before closing).