fix(runtime): discard poisoned resume id after FailoverError to break watchdog cascade (#79365)#79386
fix(runtime): discard poisoned resume id after FailoverError to break watchdog cascade (#79365)#79386draix wants to merge 2 commits into
Conversation
… watchdog cascade (openclaw#79365) When a resumed claude-cli live session was killed by the no-output watchdog the runtime kept offering the same persisted --resume <id> to every subsequent cold restart. Because the on-disk transcript was poisoned (corrupted tool-call state, hung MCP shutdown, etc.), each re-resumed turn hung again to the watchdog cap, producing the 5x 180s = ~15min Telegram-lane outage seen in the field until a manual gateway bounce. The historical recovery path only cleared the persisted CLI session binding for two stale-session classes: * session_expired - claude-cli reported "Conversation not found" * missing-transcript (openclaw#77030) - the on-disk .jsonl is gone Resumed-but-poisoned transcripts surfaced as FailoverError reason=timeout and left the binding untouched, hence the cascade. Fix: 1. cli-runner.ts (runPreparedCliAgent) Extend the in-runner cold-restart retry to also fire on reason=timeout when the provider is claude-cli and a resume sessionId was used. This mirrors the existing session_expired retry contract and keeps a single agent_end hook event per turn (codex review concern on openclaw#77385). The retry runs without --resume so the cold child generates a clean session id. 2. attempt-execution.ts Clear the persisted CLI session binding for the same condition so the next user turn also starts cold even when the in-runner retry was not taken (e.g. no sessionKey on the runner params). Both changes are gated on isClaudeCliProvider and the presence of a stored sessionId, so non-resumed timeouts (cold-start slow CLIs) and non-claude providers keep their previous behaviour. Tests: * cli-runner.reliability.test.ts - in-runner retry happens on poisoned-resume timeout, single agent_end success hook fires; no retry on cold-start timeout; no retry for non-claude providers. * attempt-execution.cli.test.ts - persisted binding is wiped after a poisoned-resume timeout and the retry uses a fresh session id; no retry / no clear on cold-start timeouts. Verified: pnpm test src/agents/cli-runner.spawn.test.ts \ src/agents/cli-runner.reliability.test.ts \ src/agents/command/session-store.test.ts \ src/agents/command/attempt-execution.cli.test.ts \ src/agents/command/attempt-execution.test.ts -> 5 files, 147 tests passed pnpm exec oxfmt --check --threads=1 (touched files): clean pnpm tsgo:core / tsgo:core:test: clean Fixes openclaw#79365
|
Codex review: needs real behavior proof before merge. Summary Reproducibility: yes. A high-confidence source path is to seed a stored Real behavior proof Next step before merge Security Review findings
Review detailsBest possible solution: Clear the stored resumed Do we have a high-confidence way to reproduce the issue? Yes. A high-confidence source path is to seed a stored Is this the best way to solve the issue? No, not as written. The direction is narrow, but the binding must be cleared before or independently from the cold retry so retry failures cannot preserve the poisoned session id. Full review comments:
Overall correctness: patch is incorrect Acceptance criteria:
What I checked:
Likely related people:
Remaining risk / open question:
Codex review notes: model gpt-5.5, reasoning high; reviewed against ea16a5e9e10c. |
|
@clawsweeper re-review The previous review failed before ClawSweeper could summarize the change ("Codex review failed for this PR with exit 1"). The PR contents themselves haven't changed — please re-run the review so the actual fix can be evaluated against the real-behavior-proof gate. |
…res (openclaw#79365) Address the clawsweeper P2 review on openclaw#79386. The previous fix retried cold (no --resume) inside the runner on a poisoned-resume timeout, then relied on the outer attempt-execution catch to clear the persisted CLI session binding via an inline isPoisonedResumeTimeout check against the final thrown error. When that in-runner cold retry itself failed with a different error class (a non-timeout FailoverError such as rate_limit / billing / overloaded, or a plain Error from a spawn failure), the outer gate stopped matching and the poisoned sessionId stayed persisted. The next user turn then re-resumed the same dead transcript and the watchdog cascade re-ignited — the exact failure mode openclaw#79365 was meant to break. Fix: carry the original poisoned-resume signal through retry failures via a Symbol-keyed marker. * failover-error.ts: introduce POISONED_RESUME_ORIGIN (Symbol.for), markPoisonedResumeOrigin(err), and hasPoisonedResumeOrigin(err). The marker is non-enumerable and safe to apply to any thrown value (primitives are returned unchanged; frozen / non-extensible errors fall back to a best-effort assignment). * cli-runner.ts: in the in-runner cold-retry catch, when the original error was a poisoned-resume timeout, tag the rethrown retry failure with the marker so the outer cleanup gate can still recognise this turn. Also tag the original timeout when the recovery branch was not taken (no sessionKey / sessionId) so the contract stays consistent. * command/attempt-execution.ts: broaden the cleanup gate to honour hasPoisonedResumeOrigin(err) in addition to the existing session_expired and inline timeout checks. Keeps the structure of the existing branch (still gated on activeCliSessionBinding + sessionKey + sessionStore + storePath) and the same belt-and-braces cold runCliWithSession(undefined) retry. Tests: * cli-runner.reliability.test.ts: two new regression tests covering (a) cold-retry surfaces a non-timeout FailoverError (rate_limit) and the rethrown error carries POISONED_RESUME_ORIGIN; (b) cold- retry rejects with a plain Error and the marker is still set. * command/attempt-execution.cli.test.ts: three new tests covering (a) marked FailoverError of a different reason triggers binding cleanup; (b) marked plain Error triggers binding cleanup; (c) regression guard — an unmarked rate_limit FailoverError does NOT trigger the cleanup, so the marker is the controlled surface. Verification: pnpm exec node scripts/run-vitest.mjs \ src/agents/cli-runner.spawn.test.ts \ src/agents/cli-runner.reliability.test.ts \ src/agents/command/session-store.test.ts \ src/agents/command/attempt-execution.cli.test.ts \ src/agents/command/attempt-execution.test.ts \ src/agents/failover-error.test.ts # -> 11 files, 368 tests passed (was 304 before; +5 new x 2 project # configs + the unchanged failover-error.test.ts pair) pnpm exec oxfmt --check --threads=1 \ src/agents/cli-runner/claude-live-session.ts \ src/agents/cli-runner.ts \ src/agents/command/attempt-execution.ts \ src/agents/command/session-store.ts \ src/agents/failover-error.ts \ src/agents/cli-runner.reliability.test.ts \ src/agents/command/attempt-execution.cli.test.ts # -> All matched files use the correct format. pnpm tsgo:core && pnpm tsgo:core:test # -> clean Fixes the P2 review concern on openclaw#79386 (clawsweeper, 2026-05-08).
|
Pushed Change summary (b4f5bc3 on top of 76530ed):
Regression tests added (5 new, all green ×2 project configs):
Verification: Real behavior proof status (the remaining failing gate): I still do not have a redacted Claude CLI poisoned-resume runtime capture from a live gateway in this PR. The original production incident (2026-05-07 22:31–23:18, ~15 min Telegram-lane outage) was diagnosed from runtime logs but not packaged as a reproducible artifact. I'll add the redacted capture to a follow-up commit when I can stage and run the after-fix scenario against a real claude-cli setup. Until then, the @clawsweeper re-review |
|
🦞🧹 Reason: re-review requires an open issue or PR. |
|
Closing this PR. The fix for #79365 is correct in principle, but in this repo's workflow it's blocked by two things that I can't unblock as an external contributor:
The underlying issue #79365 is still open — happy for any maintainer to pick this up. If a maintainer wants to revive this PR specifically, ping me and I'll rebase. Thanks! |
|
This PR was closed without merging, gated on Environment
Symptom Decisive isolation (the key data point): Working hypothesis: the reused session carries a stale MCP per-run token / Redacted trace (paths/UUIDs/session-ids/resume-hashes scrubbed): Reproduces on both opus and haiku, and independent of transcript size. Happy to capture additional redacted detail (e.g. the MCP handshake side, or a |
Summary
Fixes #79365 — the resume-watchdog cascade.
When a resumed
claude-clilive session is killed by the no-output watchdog (FailoverError reason=timeout), the runtime previously kept offering the same poisoned--resume <id>on every subsequent cold restart. Because the on-disk transcript was corrupted (hung MCP shutdown, stuck tool-call state, etc.), each re-resumed turn hung to the watchdog cap, producing the 5×180s = ~15 min Telegram-lane outage in the field report until a human bounced the gateway.Root cause
The persisted CLI session binding (
cliSessionBindings.claude-cli.sessionId) was only cleared for two stale-session classes:session_expired— claude-cli reported "Conversation not found" (in-runner retry atcli-runner.ts:478)missing-transcript(fix(cli-runner): drop stale claude-cli sessionId when transcript missing #77030) — the on-disk.jsonlis gone (pre-flight inprepare.ts:263)Resumed-but-poisoned transcripts hit neither path: they surface as
FailoverError reason=timeout, the binding is untouched, and the next turn re-resumes the same dead session id.Fix
Two layers, mirroring the existing
session_expiredrecovery contract:1.
src/agents/cli-runner.ts— in-runner cold retryExtend the in-runner retry inside
runPreparedCliAgentto also fire onreason === "timeout"when:claude-cli(isClaudeCliProvider)context.reusableCliSession.sessionId)sessionKeyso the caller can persist the fresh idThe retry runs the same attempt with
executeCliAttempt(undefined)— i.e. no--resumeflag — so the cold child starts a clean session and the post-run flow writes the new sessionId back viasetCliSessionBinding. A singleagent_endhook event fires withsuccess: true, addressing the codex review concern on #77385 (no failed-then-succeeded hook sequence).2.
src/agents/command/attempt-execution.ts— clear persisted bindingMirror the existing
session_expiredcleanup: when the same condition holds and the runner re-throws, drop the persisted binding from the session store so the next user turn also starts cold. Acts as a belt-and-braces safety net when the in-runner retry was not taken (e.g. nosessionKeyon the runner params).Why not the alternatives discussed on the issue
Reproduction
claude-cliagent on a Telegram-direct session and let a successful turn persistcliSessionBindings.claude-cli.sessionIdto the session store.~/.claude/projects/<project>/<sessionId>.jsonl(e.g. truncate to the middle of a tool_use block, or simulate a corrupted MCP shutdown).FailoverError reason=timeout), and the lane stays dead until manual bounce.--resume); a singleagent_endsuccess event fires; the persisted binding is replaced with the new sessionId; subsequent turns succeed.Tests
src/agents/cli-runner.reliability.test.ts(3 new tests)agent_endsuccess hook is observed.src/agents/command/attempt-execution.cli.test.ts(2 new tests)claude-clitimeouts (regression guard).Concurrent independent claude-cli sessions are unaffected — the recovery is keyed on the runner's
params.sessionKey+context.reusableCliSession.sessionId, both of which are per-session.Verification
Files changed
src/agents/cli-runner.tssrc/agents/command/attempt-execution.tssrc/agents/cli-runner.reliability.test.tssrc/agents/command/attempt-execution.cli.test.tsCHANGELOG.mdFixes #79365.