Skip to content

fix(agents): rethrow EmbeddedAttemptSessionTakeoverError before model fallback#83436

Closed
cael-dandelion-cult wants to merge 1 commit into
openclaw:mainfrom
cael-dandelion-cult:cael/upstream-session-takeover-no-fallback
Closed

fix(agents): rethrow EmbeddedAttemptSessionTakeoverError before model fallback#83436
cael-dandelion-cult wants to merge 1 commit into
openclaw:mainfrom
cael-dandelion-cult:cael/upstream-session-takeover-no-fallback

Conversation

@cael-dandelion-cult

Copy link
Copy Markdown

Summary

EmbeddedAttemptSessionTakeoverError (the embedded-attempt fingerprint guard that fires when the session JSONL changes during the released-lock window) is currently routed through runWithModelFallback as an ordinary candidate failure. Every model candidate fails with the same takeover error, the chain exhausts, the last fallback succeeds, and the session ends up effectively pinned to that fallback model — with no provider error and no normal fallback receipt.

This PR extends the existing carve-out family (isCommandLaneTaskTimeoutError, shouldRethrowAbort) to also rethrow EmbeddedAttemptSessionTakeoverError and SessionWriteLockTimeoutError before fallback classification.

Repro

Observed on Discord-channel-bound sessions under normal traffic. journalctl shows:

[diagnostic] lane task error: lane=main durationMs=... error="EmbeddedAttemptSessionTakeoverError: session file changed while embedded prompt lock was released: .../sessions/<id>.jsonl"
[model-fallback/decision] decision=candidate_failed
  requested=github-copilot/claude-opus-4.7-1m-internal
  candidate=github-copilot/claude-opus-4.7-1m-internal
  reason=unknown
  next=github-copilot/claude-opus-4.6
  detail=session file changed while embedded prompt lock was released
... chain repeats for each fallback ...
[model-fallback/decision] decision=candidate_succeeded candidate=openai-codex/gpt-5.5

Then the successful fallback persists as modelOverrideSource: auto. Manual re-pins decay back to the same fallback within minutes.

Four complete chain-walks observed on one seat between 20:13 and 20:30 PDT, cadence ~5 min.

Why this presents as silent reassertion

  • EmbeddedAttemptSessionTakeoverError is a local session-state guard (introduced in Release embedded session write lock before model I/O #82891). It is not a provider failure.
  • describeFailoverError() does not classify it as a provider-class failure, so [model-fallback/decision] logs reason=unknown. The clue only survives in detail=....
  • Manual model re-pins are not stale state — they decay because the next attempt re-hits the takeover race.
  • Seats with quieter session-file write cadence (no concurrent JSONL writes during the lock-release window) do not trigger the race and hold canon.

Relation to #66646 / #78633

Issue #66646 (Session file lock errors cascade through model fallback chain) was fixed by #78633, which exported hasSessionWriteLockTimeout and the carve-out narrative. However, the runtime intercept for runWithModelFallback was not retained on main: hasSessionWriteLockTimeout is only referenced from failover-error.ts describe/classify paths, not from the fallback runner itself. EmbeddedAttemptSessionTakeoverError (introduced by #82891) extends the same family but does not yet have an interceptor anywhere in the model-fallback chain. This PR adds both interceptors in runFallbackCandidate.

Change

+function isEmbeddedAttemptSessionTakeoverError(err: unknown): boolean { ... }
+function shouldRethrowSessionOwnershipError(err: unknown): boolean {
+  return isEmbeddedAttemptSessionTakeoverError(err) || isSessionWriteLockTimeoutError(err);
+}

 // in runFallbackCandidate
 if (isCommandLaneTaskTimeoutError(err)) {
   throw err;
 }
+if (shouldRethrowSessionOwnershipError(err)) {
+  throw err;
+}

The pattern matches the existing isCommandLaneTaskTimeoutError carve-out exactly.

Tests

Added regression in src/agents/model-fallback.test.ts:

  • session takeover error rethrown before second fallback candidate is invoked
  • session takeover error does not call onError (the candidate-failed receipt path)

Local verification on this branch:

  • pnpm vitest run src/agents/model-fallback.test.ts -t "session takeover|command-lane watchdog|unrecognized errors" → 6 passed / 108 skipped (114 total)
  • pnpm vitest run src/agents/model-fallback.test.ts → 2 files / 114 tests passed

Search key for future operators

session file changed while embedded prompt lock was released

This is the signature in journalctl. With this fix it stays a session-takeover error and is handled by the embedded runner / retry path instead of poisoning model fallback.

Out of scope (follow-up lanes)

  • [model-fallback/decision] reason=unknown should preserve the underlying error class in the decision-line reason field. The detail string carries it, but the reason field collapses to unknown. Surfacing-(2) prevents the next masked-as-unknown bug from being invisible.
  • Identifying which writer concurrently mutates the session JSONL during the embedded-prompt-lock-release window. The race itself is reduced (Release embedded session write lock before model I/O #82891 releases the lock before model I/O) but takeover still fires under traffic.

cc @sallyom (author of #78633, related fix)

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: XS triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 18, 2026
@clawsweeper

clawsweeper Bot commented May 18, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I did a careful shell check against current main, and this is already implemented.

Close: current main already has a same-or-better fix, merged via #83550 and present in the latest release, so this stale/conflicting branch is no longer needed.

So I’m closing this as already implemented rather than keeping a duplicate issue open.

Review details

Best possible solution:

Keep the merged non-provider coordination classifier and fallback rethrow path from #83550, and close this stale branch.

Do we have a high-confidence way to reproduce the issue?

Yes. Source and regression tests show that, without the main fix, embedded takeover or session write-lock errors can be treated as fallback candidate failures; current main now rethrows those local coordination errors on the first candidate.

Is this the best way to solve the issue?

Yes for cleanup. The current main implementation is the better solution because it centralizes recursive coordination-error detection and preserves explicit provider fallback metadata, while this branch is narrower and now obsolete.

Security review:

Security review cleared: The branch only changes local fallback error classification and tests; no dependency, workflow, secret, permission, or supply-chain surface is added.

AGENTS.md: found and applied where relevant.

What I checked:

  • Current main fail-fast path: Current main rethrows non-provider runtime coordination errors before fallback bookkeeping records a candidate failure, preventing local session takeover/write-lock errors from walking the fallback chain. (src/agents/model-fallback.ts:1567, f1b8827d20c8)
  • Current main recursive classifier: Current main detects embedded attempt session takeover and session write-lock contention through nested error, cause, and reason chains while preserving provider-classified failures. (src/agents/failover-error.ts:351, f1b8827d20c8)
  • Regression coverage on main: Current tests cover direct embedded takeover, provider-prompt cleanup takeover, direct session write-lock timeout, and provider metadata remaining authoritative over nested session locks. (src/agents/model-fallback.test.ts:1090, f1b8827d20c8)
  • Merged fix provenance: Merged PR fix(agents): skip model fallback for embedded session takeover and session write-lock errors [AI-assisted] #83550 added the non-provider coordination classifier, fallback rethrow path, and regression tests; its merge commit is the canonical fix for the same fallback-masking behavior. (6a5a1353c7f0)
  • Git history provenance: The merged fix is composed of luyao618's classifier/fallback change and Peter Steinberger's provider-metadata preservation fixup; the merge commit landed on 2026-05-18T14:30:58+01:00. (6a5a1353c7f0)
  • Release provenance: The v2026.6.6 release tag contains the same fail-fast fallback path, recursive coordination classifier, and regression tests, so the behavior is shipped rather than only on current main. (src/agents/model-fallback.ts:1563, 8c802aa68351)

Likely related people:

  • luyao618: Authored the merged classifier and fallback rethrow implementation in the canonical fix PR. (role: fix author; confidence: high; commits: 91feed332795, 6a5a1353c7f0; files: src/agents/model-fallback.ts, src/agents/model-fallback.test.ts, src/agents/failover-error.ts)
  • steipete: Authored the provider-metadata preservation fixup and posted the pre-merge verification for the canonical merged PR. (role: recent reviewer and fixup owner; confidence: high; commits: 12cb2bab5ecb, 6a5a1353c7f0; files: src/agents/failover-error.ts, src/agents/model-fallback.test.ts)
  • sallyom: Authored the earlier merged session write-lock fallback guard that this behavior builds on. (role: related prior fix author; confidence: high; commits: a74894a95401; files: src/agents/model-fallback.ts, src/agents/model-fallback.test.ts, src/agents/failover-error.ts)
  • amknight: Authored the merged prompt-lock-release work that introduced the embedded session takeover error family handled by the fallback fix. (role: embedded session-lock feature owner; confidence: high; commits: 8a060b2904d4; files: src/agents/pi-embedded-runner/run/attempt.session-lock.ts)

Codex review notes: model internal, reasoning high; reviewed against f1b8827d20c8; fix evidence: release v2026.6.6, commit 6a5a1353c7f0.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. P1 High-priority user-facing bug, regression, or broken workflow. impact:session-state Session, memory, transcript, context, or agent state can drift or corrupt. impact:auth-provider Auth, provider routing, model choice, or SecretRef resolution may break. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. labels May 18, 2026
kays0x added a commit to kays0x/openclaw that referenced this pull request May 19, 2026
…unner-owned shape

ClawSweeper review on openclaw#84046 flagged the prior count-only check as too
broad: concurrent transcript rewrites, file replacements, branch
changes, and same-user-count mutations could all keep the user-message
count unchanged while changing session state. Tighten the fence
disambiguation so a stat-fingerprint mismatch is only tolerated when
ALL of these hold:

- dev/ino match (no atomic replacement onto a new inode)
- birthtimeNs matches (no unlink+recreate-same-inode case; common on
  tmpfs/ext4 with rapid recreation, where dev and ino are reused)
- size grew (file is append-only between fence-set and assert)
- bytes [0, fenceSize) hash identical to the fenced prefix hash (no
  in-place rewrite of earlier transcript content)
- bytes [fenceSize, currentSize) contain only message entries with role
  assistant or tool (no new user-role turn, no malformed tail)

Anything else — replacement, shrink, in-place rewrite, new user turn,
or compaction-style mutation — remains a real takeover. Snapshot now
captures the prefix hash alongside the fingerprint at releaseForPrompt
and refreshSessionFileFence so the slow path can verify rigorously.

Adds tests covering in-place rewrite and unlink+recreate replacement
(both with prefix bytes coincidentally matching and roles that would
otherwise look runner-owned); existing append-only and new-user-turn
tests preserved.

Refs: openclaw#83436, openclaw#83615
Addresses ClawSweeper P1 finding on prior revision of openclaw#84046.
kays0x added a commit to kays0x/openclaw that referenced this pull request May 19, 2026
…t non-message tail entries

ClawSweeper re-review on openclaw#84046 flagged two P1 issues on the prior
revision of the append-only fence:

- The role allowlist used 'tool', but OpenClaw persists tool outputs
  as message.role === 'toolResult' (see
  src/agents/pi-embedded-runner/transcript-file-state.ts:182). Real
  tool-using turns would still false-trip the takeover. Update the
  allowlist to match isAgentMessage's runner-owned roles: assistant,
  toolResult, bashExecution.

- The slow path silently 'continue'd through non-message tail entries
  (custom, branch_summary, compaction, session). That let a different
  owner mutate session state during the released-lock window without
  a new user turn. Fail closed: any non-message entry in the tail
  rejects the fence.

Test updates: existing acceptance test now appends assistant +
toolResult + bashExecution (the three valid persisted runner-owned
roles); new test asserts a 'type: custom' tail entry trips takeover
even though it grew append-only.

Refs: openclaw#83436, openclaw#83615
Addresses ClawSweeper P1 findings on second revision of openclaw#84046.
kays0x added a commit to kays0x/openclaw that referenced this pull request May 19, 2026
…l entries

ClawSweeper P2 finding on the prior revision: the slow path accepted
any 'type: message' entry with role in {assistant, toolResult,
bashExecution} without checking the full persisted message contract.
Malformed or externally-synthesized entries carrying an accepted role
would advance the fence over state they do not actually represent.

Add a local isPersistedRunnerOwnedMessage validator that mirrors
isAgentMessage in transcript-file-state.ts for the three accepted
roles: assistant needs content present; toolResult needs string
toolCallId/toolName, boolean isError, array content; bashExecution
needs string command/output, boolean cancelled/truncated, and
number-or-undefined exitCode. Reject everything else.

Tests: existing acceptance test updated to use array content for
toolResult (matching the persisted schema); new test asserts a
toolResult message missing toolName/isError/content trips takeover
even though role is accepted.

Refs: openclaw#83436, openclaw#83615
Addresses ClawSweeper P2 finding on third revision of openclaw#84046.
kays0x added a commit to kays0x/openclaw that referenced this pull request May 19, 2026
ClawSweeper P2 finding on prior revision: the local validator was a
hand-rolled subset of isAgentMessage in transcript-file-state.ts, and
each round of review surfaced a new schema gap (assistant content
just had to exist; toolResult content just had to be an array;
bashExecution missed nothing here yet). Reproducing the contract
locally is the wrong layer — transcript-file-state.ts already owns
the canonical persisted message contract.

Export isAgentMessage from transcript-file-state.ts and call it from
the fence slow path, then gate on the runner-owned role subset
(assistant, toolResult, bashExecution). user-role and custom-role
messages remain real takeover signals even when their contract is
valid.

The local helper drops to ten lines: full contract check via the
canonical validator, then role-subset gate. No new tests needed —
the existing malformed-toolResult regression already covers the
schema mismatch the canonical validator now catches more strictly
(missing toolName/isError + non-array content).

Refs: openclaw#83436, openclaw#83615
Addresses ClawSweeper P2 finding on fourth revision of openclaw#84046.
@clawsweeper clawsweeper Bot added status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. and removed impact:session-state Session, memory, transcript, context, or agent state can drift or corrupt. impact:auth-provider Auth, provider routing, model choice, or SecretRef resolution may break. merge-risk: 🚨 auth-provider 🚨 May break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jun 15, 2026
@clawsweeper

clawsweeper Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper applied the proposed close for this PR.

@clawsweeper clawsweeper Bot closed this Jun 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling P1 High-priority user-facing bug, regression, or broken workflow. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: XS status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant