Skip to content

fix(runtime): discard poisoned resume id after FailoverError to break watchdog cascade (#79365)#79386

Closed
draix wants to merge 2 commits into
openclaw:mainfrom
draix:fix/79365-discard-poisoned-resume-id
Closed

fix(runtime): discard poisoned resume id after FailoverError to break watchdog cascade (#79365)#79386
draix wants to merge 2 commits into
openclaw:mainfrom
draix:fix/79365-discard-poisoned-resume-id

Conversation

@draix

@draix draix commented May 8, 2026

Copy link
Copy Markdown

Update 2026-05-24 — addresses clawsweeper review (P2)

The previous in-runner cold retry could leak the original poisoned-resume signal: if the cold retry itself failed with a non-timeout error (e.g. FailoverError reason=rate_limit/billing/overloaded, or a plain Error), the outer attempt-execution cleanup gate stopped matching and the poisoned --resume <id> stayed persisted, allowing the cascade to re-arm on the next user turn.

Fixed by carrying the poisoned-resume signal through retry failures with a POISONED_RESUME_ORIGIN marker (Symbol-keyed, non-enumerable, applied via markPoisonedResumeOrigin(err) and read via hasPoisonedResumeOrigin(err) from src/agents/failover-error.ts). The in-runner catch tags the rethrown retry error, and the outer cleanup honours the marker in addition to the existing session_expired and inline timeout checks.

Added regression coverage:

  • cli-runner.reliability.test.ts — cold retry surfaces a non-timeout FailoverError (rate_limit) → rethrown error carries the marker; cold retry rejects with a plain Error → marker still set.
  • command/attempt-execution.cli.test.ts — marked FailoverError of a different reason triggers binding cleanup; marked plain Error triggers binding cleanup; regression guard: an unmarked rate_limit FailoverError does NOT trigger the cleanup (the marker is the controlled surface, no leakage into unrelated failures).

Status of the real-runtime-proof gate: still pending. I do not have a captured redacted Claude CLI poisoned-resume trace from a live gateway in this PR yet (the production occurrence on 2026-05-07 22:31–23:18 was diagnosed but not packaged as a reproducible artifact bundle). Will add to a follow-up commit when I can stage and capture the redacted run.


Summary

Fixes #79365 — the resume-watchdog cascade.

When a resumed claude-cli live session is killed by the no-output watchdog (FailoverError reason=timeout), the runtime previously kept offering the same poisoned --resume <id> on every subsequent cold restart. Because the on-disk transcript was corrupted (hung MCP shutdown, stuck tool-call state, etc.), each re-resumed turn hung to the watchdog cap, producing the 5×180s = ~15 min Telegram-lane outage in the field report until a human bounced the gateway.

Root cause

The persisted CLI session binding (cliSessionBindings.claude-cli.sessionId) was only cleared for two stale-session classes:

Resumed-but-poisoned transcripts hit neither path: they surface as FailoverError reason=timeout, the binding is untouched, and the next turn re-resumes the same dead session id.

Fix

Two layers, mirroring the existing session_expired recovery contract:

1. src/agents/cli-runner.ts — in-runner cold retry

Extend the in-runner retry inside runPreparedCliAgent to also fire on reason === "timeout" when:

  • the provider is claude-cli (isClaudeCliProvider)
  • a resume sessionId was actually used (context.reusableCliSession.sessionId)
  • there is a sessionKey so the caller can persist the fresh id

The retry runs the same attempt with executeCliAttempt(undefined) — i.e. no --resume flag — so the cold child starts a clean session and the post-run flow writes the new sessionId back via setCliSessionBinding. A single agent_end hook event fires with success: true, addressing the codex review concern on #77385 (no failed-then-succeeded hook sequence).

2. src/agents/command/attempt-execution.ts — clear persisted binding

Mirror the existing session_expired cleanup: when the same condition holds and the runner re-throws, drop the persisted binding from the session store so the next user turn also starts cold. Acts as a belt-and-braces safety net when the in-runner retry was not taken (e.g. no sessionKey on the runner params).

Why not the alternatives discussed on the issue

  • Just lift the 180s cap: helps slow-but-healthy turns; does nothing for corrupted transcripts. Cascade gets slower, not gone.
  • Model fallback on first timeout: either snaps over too aggressively when claude-cli is just slow, or requires the runtime to know whether claude-cli is wedged process-wide vs. just on this turn. Discarding the resume id is the narrow fix.

Reproduction

  1. Start the gateway with an active claude-cli agent on a Telegram-direct session and let a successful turn persist cliSessionBindings.claude-cli.sessionId to the session store.
  2. Poison the on-disk transcript at ~/.claude/projects/<project>/<sessionId>.jsonl (e.g. truncate to the middle of a tool_use block, or simulate a corrupted MCP shutdown).
  3. Send a follow-up message. Without the fix: the next 5 turns each hang to the 180s no-output watchdog (FailoverError reason=timeout), and the lane stays dead until manual bounce.
  4. With the fix: the first attempt is killed by the watchdog as before, but the runner immediately retries cold (no --resume); a single agent_end success event fires; the persisted binding is replaced with the new sessionId; subsequent turns succeed.

Tests

  • src/agents/cli-runner.reliability.test.ts (3 new tests)
    • In-runner retry happens on poisoned-resume timeout; supervisor is spawned twice; a single agent_end success hook is observed.
    • No retry on no-output timeout when there was no resume id (cold start).
    • No retry for non-claude-cli providers, even with a resume id (preserves historical contract for codex/other backends).
  • src/agents/command/attempt-execution.cli.test.ts (2 new tests)
    • Persisted binding is wiped after a poisoned-resume timeout and the retry uses a fresh session id.
    • No clear / no retry on cold-start claude-cli timeouts (regression guard).

Concurrent independent claude-cli sessions are unaffected — the recovery is keyed on the runner's params.sessionKey + context.reusableCliSession.sessionId, both of which are per-session.

Verification

pnpm test src/agents/cli-runner.spawn.test.ts \
          src/agents/cli-runner.reliability.test.ts \
          src/agents/command/session-store.test.ts \
          src/agents/command/attempt-execution.cli.test.ts \
          src/agents/command/attempt-execution.test.ts
# -> 5 files, 147 tests passed (existing 142 + 5 new)

pnpm exec oxfmt --check --threads=1 \
  src/agents/cli-runner/claude-live-session.ts \
  src/agents/cli-runner.ts \
  src/agents/command/attempt-execution.ts \
  src/agents/command/session-store.ts
# -> All matched files use the correct format.

pnpm tsgo:core
pnpm tsgo:core:test
# -> clean

Files changed

File +/- Purpose
src/agents/cli-runner.ts +28 / -3 In-runner cold retry for poisoned-resume timeout
src/agents/command/attempt-execution.ts +24 / -9 Clear persisted binding on poisoned-resume timeout
src/agents/cli-runner.reliability.test.ts +236 / -3 3 regression tests + claude-cli context variant
src/agents/command/attempt-execution.cli.test.ts +112 / -0 2 regression tests
CHANGELOG.md +1 Unreleased / Fixes entry

Fixes #79365.

… watchdog cascade (openclaw#79365)

When a resumed claude-cli live session was killed by the no-output
watchdog the runtime kept offering the same persisted --resume <id>
to every subsequent cold restart. Because the on-disk transcript was
poisoned (corrupted tool-call state, hung MCP shutdown, etc.), each
re-resumed turn hung again to the watchdog cap, producing the 5x
180s = ~15min Telegram-lane outage seen in the field until a manual
gateway bounce.

The historical recovery path only cleared the persisted CLI session
binding for two stale-session classes:
  * session_expired   - claude-cli reported "Conversation not found"
  * missing-transcript (openclaw#77030) - the on-disk .jsonl is gone

Resumed-but-poisoned transcripts surfaced as FailoverError
reason=timeout and left the binding untouched, hence the cascade.

Fix:
  1. cli-runner.ts (runPreparedCliAgent)
     Extend the in-runner cold-restart retry to also fire on
     reason=timeout when the provider is claude-cli and a resume
     sessionId was used. This mirrors the existing session_expired
     retry contract and keeps a single agent_end hook event per
     turn (codex review concern on openclaw#77385). The retry runs without
     --resume so the cold child generates a clean session id.
  2. attempt-execution.ts
     Clear the persisted CLI session binding for the same condition
     so the next user turn also starts cold even when the in-runner
     retry was not taken (e.g. no sessionKey on the runner params).

Both changes are gated on isClaudeCliProvider and the presence of a
stored sessionId, so non-resumed timeouts (cold-start slow CLIs) and
non-claude providers keep their previous behaviour.

Tests:
  * cli-runner.reliability.test.ts - in-runner retry happens on
    poisoned-resume timeout, single agent_end success hook fires;
    no retry on cold-start timeout; no retry for non-claude providers.
  * attempt-execution.cli.test.ts - persisted binding is wiped after
    a poisoned-resume timeout and the retry uses a fresh session id;
    no retry / no clear on cold-start timeouts.

Verified:
  pnpm test src/agents/cli-runner.spawn.test.ts \
            src/agents/cli-runner.reliability.test.ts \
            src/agents/command/session-store.test.ts \
            src/agents/command/attempt-execution.cli.test.ts \
            src/agents/command/attempt-execution.test.ts
  -> 5 files, 147 tests passed
  pnpm exec oxfmt --check --threads=1 (touched files): clean
  pnpm tsgo:core / tsgo:core:test: clean

Fixes openclaw#79365
@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. size: M labels May 8, 2026
@clawsweeper

clawsweeper Bot commented May 8, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge.

Summary
The PR adds Claude CLI resumed-timeout recovery, clears stored session bindings in the user attempt path, adds focused regression tests, and updates the changelog.

Reproducibility: yes. A high-confidence source path is to seed a stored claude-cli binding with a present transcript, have the resumed attempt throw FailoverError reason timeout, and observe that current main only clears session_expired.

Real behavior proof
Needs real behavior proof before merge: The PR body shows unit-test and check output only, so it still needs redacted after-fix runtime proof from a real Claude CLI poisoned-resume setup; updating the PR body should trigger a fresh ClawSweeper review, or a maintainer can comment @clawsweeper re-review. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, ask a maintainer to comment @clawsweeper re-review.

Next step before merge
Contributor or maintainer action is needed because the external PR lacks real runtime proof and still has a cleanup-ordering defect; automation cannot supply the contributor's real setup proof.

Security
Cleared: No concrete security or supply-chain concern found; the diff is limited to runtime retry/session cleanup logic, focused tests, and changelog text.

Review findings

  • [P2] Clear the poisoned binding before retrying cold — src/agents/command/attempt-execution.ts:548
Review details

Best possible solution:

Clear the stored resumed claude-cli binding as soon as the poisoned timeout is identified, then retry cold or let the next turn start cold with focused regression coverage and redacted runtime proof.

Do we have a high-confidence way to reproduce the issue?

Yes. A high-confidence source path is to seed a stored claude-cli binding with a present transcript, have the resumed attempt throw FailoverError reason timeout, and observe that current main only clears session_expired.

Is this the best way to solve the issue?

No, not as written. The direction is narrow, but the binding must be cleared before or independently from the cold retry so retry failures cannot preserve the poisoned session id.

Full review comments:

  • [P2] Clear the poisoned binding before retrying cold — src/agents/command/attempt-execution.ts:548
    When runPreparedCliAgent handles the first resumed timeout, it retries cold before runAgentAttempt can clear the store. If that cold retry throws a non-timeout FailoverError or a plain error, the outer catch only sees the retry error, this condition is false, and the original activeCliSessionBinding stays persisted; the next turn can resume the same poisoned transcript. Clear the binding before the in-runner retry, or carry the original poisoned-resume signal through retry failures.
    Confidence: 0.86

Overall correctness: patch is incorrect
Overall confidence: 0.84

Acceptance criteria:

  • node scripts/run-vitest.mjs src/agents/cli-runner.reliability.test.ts src/agents/command/attempt-execution.cli.test.ts
  • redacted real Claude CLI runtime proof showing the poisoned resumed session id is not reused after the watchdog timeout

What I checked:

  • Current main retry gate: Current main passes the reusable CLI session id into executeCliAttempt, but the in-runner recovery branch only retries cold for session_expired; timeout falls through to the failed hook/rethrow path. (src/agents/cli-runner.ts:497, ea16a5e9e10c)
  • Current main store cleanup gate: Current main clears and retries the persisted CLI binding only when the final error is FailoverError with reason session_expired; other failover reasons are rethrown without cleanup. (src/agents/command/attempt-execution.ts:533, ea16a5e9e10c)
  • PR diff adds timeout retry before store cleanup: The PR adds isPoisonedResumeTimeout inside runPreparedCliAgent and retries without --resume before the outer runAgentAttempt catch can clear the session store. (src/agents/cli-runner.ts:493, 76530edfb19c)
  • PR diff cleanup depends on final thrown error: The PR's outer cleanup only runs when the error that reaches runAgentAttempt is session_expired or timeout; if the in-runner cold retry throws a different error, the original poisoned-resume signal is lost and the old binding remains stored. (src/agents/command/attempt-execution.ts:548, 76530edfb19c)
  • Retry failures are rethrown as the retry error: toCliRunFailure rethrows FailoverError instances directly, so a failed cold retry can escape as the retry error rather than the original resumed timeout. (src/agents/cli-runner.ts:280, ea16a5e9e10c)
  • Watchdog timeout classification: The no-output watchdog throws FailoverError with reason: "timeout", matching the poisoned-resume cascade path described by the linked issue. (src/agents/cli-runner/execute.ts:665, ea16a5e9e10c)

Likely related people:

  • frankekn: Commit history identifies Frank Yang's session_expired CLI recovery work as the adjacent contract this PR extends for poisoned timeout cases. (role: introduced adjacent recovery behavior; confidence: medium; commits: ed86252aa5a5; files: src/agents/cli-runner.ts, src/agents/command/attempt-execution.ts, src/agents/failover-error.ts)
  • HFConsultant: Commit history identifies the atomic persisted CLI session clearing helper used by the expected cleanup path. (role: session-store cleanup seam contributor; confidence: medium; commits: 647f4ee8ce4f; files: src/agents/command/session-store.ts, src/agents/command/attempt-execution.ts, src/agents/cli-session.ts)
  • vincentkoc: Commit history identifies work persisting Claude CLI session ids, which is the binding involved in the poisoned resume path. (role: adjacent CLI session persistence contributor; confidence: medium; commits: 84eb617a792d; files: src/agents/command/session-store.ts, src/agents/command/attempt-execution.ts, src/agents/command/session-store.test.ts)
  • Sergio Cadavid: Current-main blame in this shallow checkout points the affected runner, attempt-execution, session-store, and execute lines at a recent broad agents commit, making this a recent routing signal rather than original-behavior provenance. (role: recent area contributor; confidence: low; commits: 472523360d88; files: src/agents/cli-runner.ts, src/agents/command/attempt-execution.ts, src/agents/command/session-store.ts)

Remaining risk / open question:

  • The PR can still preserve the old poisoned binding when the internal cold retry fails with a non-timeout FailoverError or plain error.
  • The contributor has not provided redacted after-fix runtime proof from a real Claude CLI poisoned-resume setup.
  • The broader open work in fix(agents): clear poisoned claude cli sessions #81247 overlaps this recovery area, so maintainers should reconcile retry-versus-clear-only semantics before landing either path.

Codex review notes: model gpt-5.5, reasoning high; reviewed against ea16a5e9e10c.

@draix

draix commented May 10, 2026

Copy link
Copy Markdown
Author

@clawsweeper re-review

The previous review failed before ClawSweeper could summarize the change ("Codex review failed for this PR with exit 1"). The PR contents themselves haven't changed — please re-run the review so the actual fix can be evaluated against the real-behavior-proof gate.

…res (openclaw#79365)

Address the clawsweeper P2 review on openclaw#79386. The previous fix retried
cold (no --resume) inside the runner on a poisoned-resume timeout, then
relied on the outer attempt-execution catch to clear the persisted CLI
session binding via an inline isPoisonedResumeTimeout check against the
final thrown error. When that in-runner cold retry itself failed with a
different error class (a non-timeout FailoverError such as rate_limit /
billing / overloaded, or a plain Error from a spawn failure), the outer
gate stopped matching and the poisoned sessionId stayed persisted. The
next user turn then re-resumed the same dead transcript and the
watchdog cascade re-ignited — the exact failure mode openclaw#79365 was meant
to break.

Fix: carry the original poisoned-resume signal through retry failures
via a Symbol-keyed marker.

  * failover-error.ts: introduce POISONED_RESUME_ORIGIN (Symbol.for),
    markPoisonedResumeOrigin(err), and hasPoisonedResumeOrigin(err).
    The marker is non-enumerable and safe to apply to any thrown
    value (primitives are returned unchanged; frozen / non-extensible
    errors fall back to a best-effort assignment).

  * cli-runner.ts: in the in-runner cold-retry catch, when the
    original error was a poisoned-resume timeout, tag the rethrown
    retry failure with the marker so the outer cleanup gate can
    still recognise this turn. Also tag the original timeout when
    the recovery branch was not taken (no sessionKey / sessionId)
    so the contract stays consistent.

  * command/attempt-execution.ts: broaden the cleanup gate to honour
    hasPoisonedResumeOrigin(err) in addition to the existing
    session_expired and inline timeout checks. Keeps the structure
    of the existing branch (still gated on activeCliSessionBinding
    + sessionKey + sessionStore + storePath) and the same
    belt-and-braces cold runCliWithSession(undefined) retry.

Tests:

  * cli-runner.reliability.test.ts: two new regression tests covering
    (a) cold-retry surfaces a non-timeout FailoverError (rate_limit)
    and the rethrown error carries POISONED_RESUME_ORIGIN; (b) cold-
    retry rejects with a plain Error and the marker is still set.

  * command/attempt-execution.cli.test.ts: three new tests covering
    (a) marked FailoverError of a different reason triggers binding
    cleanup; (b) marked plain Error triggers binding cleanup; (c)
    regression guard — an unmarked rate_limit FailoverError does NOT
    trigger the cleanup, so the marker is the controlled surface.

Verification:

  pnpm exec node scripts/run-vitest.mjs \
    src/agents/cli-runner.spawn.test.ts \
    src/agents/cli-runner.reliability.test.ts \
    src/agents/command/session-store.test.ts \
    src/agents/command/attempt-execution.cli.test.ts \
    src/agents/command/attempt-execution.test.ts \
    src/agents/failover-error.test.ts
  # -> 11 files, 368 tests passed (was 304 before; +5 new x 2 project
  # configs + the unchanged failover-error.test.ts pair)

  pnpm exec oxfmt --check --threads=1 \
    src/agents/cli-runner/claude-live-session.ts \
    src/agents/cli-runner.ts \
    src/agents/command/attempt-execution.ts \
    src/agents/command/session-store.ts \
    src/agents/failover-error.ts \
    src/agents/cli-runner.reliability.test.ts \
    src/agents/command/attempt-execution.cli.test.ts
  # -> All matched files use the correct format.

  pnpm tsgo:core && pnpm tsgo:core:test
  # -> clean

Fixes the P2 review concern on openclaw#79386 (clawsweeper, 2026-05-08).
@draix

draix commented May 24, 2026

Copy link
Copy Markdown
Author

Pushed b4f5bc3 addressing the P2 review finding from the 2026-05-08 ClawSweeper review (clear the poisoned binding before / independently from the cold retry so retry failures cannot preserve the poisoned session id).

Change summary (b4f5bc3 on top of 76530ed):

  • src/agents/failover-error.ts — new POISONED_RESUME_ORIGIN Symbol.for(...) marker plus markPoisonedResumeOrigin(err) / hasPoisonedResumeOrigin(err) helpers. Marker is non-enumerable, safe for any thrown value (primitives unchanged; frozen errors fall back to best-effort assignment).
  • src/agents/cli-runner.ts — in the in-runner cold-retry catch on a poisoned-resume timeout, tag the rethrown retry error with the marker so the outer cleanup can still recognise this turn even when the cold retry surfaces a different FailoverError.reason (rate_limit / billing / overloaded / ...) or a plain Error. Also tag the original timeout when the recovery branch was not taken (no sessionKey / sessionId).
  • src/agents/command/attempt-execution.ts — broaden the cleanup gate to honour hasPoisonedResumeOrigin(err) in addition to the existing session_expired and inline isPoisonedResumeTimeout checks. Same structural guards (gated on activeCliSessionBinding + sessionKey + sessionStore + storePath) and the same belt-and-braces cold runCliWithSession(undefined) retry.

Regression tests added (5 new, all green ×2 project configs):

  • cli-runner.reliability.test.ts
    • cold retry surfaces a non-timeout FailoverError (rate_limit) → rethrown error carries the marker
    • cold retry rejects with a plain Error → marker still set
  • command/attempt-execution.cli.test.ts
    • marked FailoverError of a different reason → binding cleanup runs
    • marked plain Error → binding cleanup runs
    • regression guard: unmarked rate_limit FailoverError → binding cleanup does NOT run (marker is the controlled surface, not a leak into unrelated failures)

Verification:

node scripts/run-vitest.mjs \
  src/agents/cli-runner.spawn.test.ts \
  src/agents/cli-runner.reliability.test.ts \
  src/agents/command/session-store.test.ts \
  src/agents/command/attempt-execution.cli.test.ts \
  src/agents/command/attempt-execution.test.ts \
  src/agents/failover-error.test.ts
# -> 11 files, 368 tests passed (was 304 before this commit)

pnpm exec oxfmt --check --threads=1 [touched files]   # all formatted
pnpm tsgo:core && pnpm tsgo:core:test                  # clean

Real behavior proof status (the remaining failing gate): I still do not have a redacted Claude CLI poisoned-resume runtime capture from a live gateway in this PR. The original production incident (2026-05-07 22:31–23:18, ~15 min Telegram-lane outage) was diagnosed from runtime logs but not packaged as a reproducible artifact. I'll add the redacted capture to a follow-up commit when I can stage and run the after-fix scenario against a real claude-cli setup. Until then, the Real behavior proof check will keep failing on the policy gate — happy for a maintainer to apply proof: override if the test coverage + diagnosis is considered sufficient for this scoped runtime-fix.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented May 24, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper could not start a re-review for this item.

Reason: re-review requires an open issue or PR.

@draix

draix commented May 24, 2026

Copy link
Copy Markdown
Author

Closing this PR.

The fix for #79365 is correct in principle, but in this repo's workflow it's blocked by two things that I can't unblock as an external contributor:

  1. Real behavior proof gate — requires a live OpenClaw + Claude CLI session log demonstrating the watchdog cascade and its recovery. I don't have a reproducible test rig that hits the resume-watchdog path on demand.
  2. Merge conflicts with current main (branch is 9000+ commits behind by now), which would need a non-trivial rebase against an unstable target.

The underlying issue #79365 is still open — happy for any maintainer to pick this up. If a maintainer wants to revive this PR specifically, ping me and I'll rebase. Thanks!

@draix draix closed this May 24, 2026
@samiamkline-jpg

Copy link
Copy Markdown

This PR was closed without merging, gated on needs-real-behavior-proof. Here is a live-gateway repro of the exact resume-watchdog cascade, captured on a production gateway — posting it here since the linked issue #79365 is locked. The change proposed here ("discard the poisoned resume id after FailoverError + cold-retry fresh") would recover this case.

Environment

  • openclaw 2026.5.7
  • claude-cli 2.1.168 (Anthropic, opus + haiku)
  • macOS; gateway with the bundled openclaw HTTP MCP server (per-run bearer token + x-session-key/x-openclaw-* headers)

Symptom
Interactive turns that resume the claude-cli session (useResume=true reuse=reusable) hang producing zero stream output, get killed by the no-output watchdog (reason=timeout next=none — no fallback), then the lane wedges: the gateway logs stuck session ... reason=active_reply_work action=keep_lane and skips its own recovery, so the next user message queues indefinitely. sessions.abort returns no-active-run; only sessions.reset clears it. Fresh turns (useResume=false) always succeed — the discriminator is purely resume vs fresh.

Decisive isolation (the key data point):
Running plain claude --resume <session-id> outside the gateway (no openclaw flags) on the same session replied in ~2.2s (ttft 1.9s, exit 0). The resume target transcript was ~60 KB (small, healthy). So this is not claude-cli's --resume itself and not a corrupted/oversized transcript — it only hangs when --resume is combined with the openclaw child flags (--strict-mcp-config --mcp-config <openclaw HTTP MCP> --permission-prompt-tool stdio --input-format stream-json --replay-user-messages).

Working hypothesis: the reused session carries a stale MCP per-run token / x-session-key while mcpResumeHash does not invalidate reuse (possibly interacts with #75849, which intentionally stopped invalidating reuse on auth/prompt/MCP-fingerprint drift). The child stalls on the MCP handshake during resume → never emits a stream event → no-output watchdog kill. This is why the trigger is MCP-token drift rather than a corrupt transcript, but the cold-retry-fresh approach here would still recover it (fresh turns are 100% reliable in this repro).

Redacted trace (paths/UUIDs/session-ids/resume-hashes scrubbed):

# CLI exec (resume turn) — gateway.log
[agent/cli-backend] cli exec: provider=claude-cli model=opus promptChars=381 trigger=user useResume=true session=present resumeSession=<RESUME_ID> reuse=reusable historyPrompt=none

# Turn failure + no-output watchdog kill — gateway.err.log
[agent/cli-backend] claude live session turn failed: provider=claude-cli model=claude-opus-4-7 durationMs=423094 error=FailoverError
[model-fallback/decision] model fallback decision: decision=candidate_failed requested=anthropic/claude-opus-4-7 candidate=anthropic/claude-opus-4-7 reason=timeout next=none detail=CLI produced no output for 420s and was terminated.

# Lane wedges; gateway DETECTS the stuck session but SKIPS recovery — gateway.err.log
[diagnostic] stuck session recovery skipped: reason=active_reply_work action=keep_lane sessionId=<UUID> sessionKey=agent:main:tui-<TUI_ID> age=294s queueDepth=1 activeSessionId=<UUID>

Reproduces on both opus and haiku, and independent of transcript size. Happy to capture additional redacted detail (e.g. the MCP handshake side, or a --mcp-config-only repro isolating the openclaw HTTP MCP) if it helps validate the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling size: L triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Resume-watchdog cascade: discard the resume id after first FailoverError to break poisoned-state loops

2 participants