Skip to content

Embedded agent run deterministically self-takeovers (EmbeddedAttemptSessionTakeoverError) on any multi-step turn — owned-write refresh is ineffective #95228

Description

@rvdlaar

Edit (v2): corrected the session-writer mechanism — it is in-place writeFileSync/appendFileSync (stable inode), not atomic temp+rename. The earlier "inode/ctime perturbation via rename" line was wrong and is removed; this narrows the localization (see Code-level mechanism and Localization). Evidence/repro/elimination unchanged.

Severity

On the affected version, every embedded agent run that performs at least one tool call aborts itself before finishing — i.e. no multi-step embedded turn completes. Deterministic (13/13 on a minimal repro), no concurrency or fan-out required. Flagging as high-impact for anyone on v2026.6.1 whose embedded runs do tool calls.

Environment

  • Observed on a fork tracking upstream v2026.6.1 (merge of v2026.5.18 → v2026.6.1).
  • The session-takeover fence is upstream/native as of v2026.6.1 (the fork's prior fence impl was retired and assimilated by upstream at this version).
  • Fork context: this fork wraps sessionManager.appendMessage/appendCustomEntry (a guard that adds an onMessagePersisted callback). The writer, the fence, and the fingerprint logic discussed below are all stock upstream files — the fork wrapper only adds a call to the upstream refreshAfterOwnedSessionWrite. We have not confirmed whether stock upstream reproduces; confirming that is part of the ask.

Summary

The run releases its embedded prompt lock for the model call; during that window its own session-record write lands in the .jsonl; on reacquireAfterPromptassertSessionFileFence the fence sees the file changed and throws:

EmbeddedAttemptSessionTakeoverError: session file changed while embedded prompt lock was released: <sessionFile>

There is no second writer (one runId per takeover'd file). The fence misreads the run's own write as a competing takeover.

Reproduction (deterministic)

  1. Spawn a single embedded agent on a trivial task that makes one tool call (e.g. "write SMOKE-OK to a file" → one tool call + one assistant message appended).
  2. Use a real model with normal latency (a non-trivial prompt-release window).
  3. Result: EmbeddedAttemptSessionTakeoverError on reacquire, before the task completes. 13/13.
Exact harness invocation we used (fork; adapt to a stock embedded entrypoint)
spawnAgent({ agentId: "henry", message: "Write SMOKE-OK to <file>", timeout: 300, todoId: <unique> })
// → ok:false; run log shows EmbeddedAttemptSessionTakeoverError naming the run's own session .jsonl

Each run used a fresh, unique session file. Model that answered: a healthy cloud primary (~5s), no fallback.

Evidence

  • 13/13 deterministic on minimal single-write spawns (each a fresh session file).
  • Fast-fallback independent: failing runs ran on a healthy primary (answered ~5s, zero suspended … skipping lines). Historically 4401/6046 (73%) of takeover-trigger events occurred with nothing suspended (suspendedModels:[], provider health OK).
  • The fence names the small session .jsonl (~1.6–2.1 KB), not the large .trajectory.jsonl.
  • The model call succeeds (assistant write toolCall is produced); the throw happens after, on reacquire.

Code-level mechanism (all stock upstream files)

The session writer is in-place — inode is stable (src/config/sessions/transcript-jsonl.ts):

export function writeJsonlEntriesSync(filePath, entries) { writeFileSync(filePath, serializeJsonlEntries(entries), "utf-8"); } // truncate+write in place
export function appendJsonlEntrySync(filePath, entry)   { appendFileSync(filePath, serializeJsonlEntry(entry), "utf-8"); }   // append in place

SessionManager.persist() (src/agents/sessions/session-manager.ts) calls these on the assistant flush (writeJsonlEntriesSync once hasAssistant, then appendJsonlEntrySync). So across these writes dev/ino do not change — only size/mtimeNs/ctimeNs.

The fence fingerprint keys on all five fields (src/agents/embedded-agent-runner/run/attempt.session-lock.ts):

type SessionFileFingerprint = { exists: false } | { exists: true; dev; ino; size; mtimeNs; ctimeNs };
// sameSessionFileFingerprint(...) compares dev && ino && size && mtimeNs && ctimeNs

So an in-place own write does change the fingerprint (size/mtime/ctime), which is what assertSessionFileFence compares on reacquire.

The owned-write refresh updates only fenceFingerprint (not the ownedSessionFileWrites generation map, which only publishOwnedSessionFileFence populates):

refreshAfterOwnedSessionWrite(): void {
  if (fenceActive && !takeoverDetected) {
    fenceFingerprint = readSessionFileFingerprintSync(sessionFile);
    fenceSnapshot = { fingerprint: fenceFingerprint };
  }
}

The benign-append escape hatch does not cover normal assistant messages. assertSessionFileFence's benign check (sessionFenceAdvanceIsBenign) only whitelists lines where message.provider === "openclaw" and model ∈:

const TRANSCRIPT_ONLY_OPENCLAW_ASSISTANT_MODELS = new Set(["delivery-mirror", "gateway-injected"]);

A normal agent assistant message (e.g. provider ollama-cloud/openai/etc.) is never benign → if the fence sees it as a change with no fingerprint/owned match, it throws.

What we ruled out (fork-side; runtime-refuted, each rebuilt + deployed + tested)

  1. Force publishOwnedWrite:true on the owned-transcript-write context feeding appendSessionTranscriptMessage10/10 still fail (the write bypasses that context).
  2. Diagnostic logging in appendSessionTranscriptMessage (confirmed live in dist) → 0 hits while the .jsonl was written and the takeover threw (the embedded run's .jsonl writes bypass the owned-context path entirely; the writer is SessionManager.persist).
  3. Wire refreshAfterOwnedSessionWrite() at the SessionManager.persist() sync-write chokepoint (covering every write, incl. appendCustomEntry) → 10/10 still fail. The refresh was already wired at onMessagePersisted and the takeover was 13/13 with it in place; broadening coverage changed nothing.

Conclusion: the defect is not call-coverage of refreshAfterOwnedSessionWrite. The refresh, invoked after the run's own write, does not neutralize the takeover.

Localization (two candidates; inode-perturbation is excluded — writes are in-place)

For assertSessionFileFence to still throw after refreshAfterOwnedSessionWrite ran, the post-refresh fenceFingerprint must differ from the file's state at reacquire. Given in-place writes (stable inode), that means one of:

  1. Window-bracketing missfenceActive is false when the persist/refresh fires (the session-manager write lands outside the [releaseForPrompt, reacquireAfterPrompt] window, or the refresh runs in a different async context), so the refresh no-ops and the stale baseline trips on reacquire; or
  2. A write the refresh doesn't cover lands after the last refresh — e.g. a session write outside SessionManager.persist (the agent-SDK's own persistence, a post-stream flush) that mutates size/mtime after the refresh snapshot and before reacquire's stat.

Either way, because the appended assistant line is not in the benign whitelist and refreshAfterOwnedSessionWrite records no owned-generation entry, none of assertSessionFileFence's three escape hatches (fingerprint-equal / owned-generation-match / benign) catch it → throw.

The single diagnostic that decides it

On a throwing run, add temporary logging inside refreshAfterOwnedSessionWrite and assertSessionFileFence:

  • in refreshAfterOwnedSessionWrite: log fenceActive and the just-read fenceFingerprint (size/mtimeNs/ctimeNs).
  • in assertSessionFileFence: at entry, log fenceActive, fenceFingerprint vs current, and which branch is taken (equal / owned-match / benign / throw).

If fenceActive was false during the assistant-write refresh → candidate 1 (window miss). If it was true and fenceFingerprint matched immediately post-refresh but current differs at reacquire → candidate 2 (uncovered later write); logging the time-ordered .jsonl stat (size/mtime) at refresh vs reacquire then names the extra writer.

Proposed fix directions (for discussion)

  • If candidate 1: ensure the embedded session write is bracketed inside the fence-active window (or have persist() route through the same locked path the fence tracks).
  • If candidate 2: identify the post-refresh writer and bring it under the owned-write path, or broaden the benign-append allowance so the run's own normal assistant/tool appends to its own session file are treated as owned (today only delivery-mirror/gateway-injected are benign).
  • Independent hardening: a normal agent appending to its own session during its own prompt-release window is the common case — it may warrant first-class "owned append" handling rather than relying on fingerprint equality holding across the model-I/O window.

Ask

  • Is this a known issue in the v2026.6.1 native session-takeover fence?
  • Can you confirm whether stock upstream reproduces (trivial embedded spawn, real model, one tool call)?
  • Guidance on the intended owned-write model: should a run's own in-place session append during the prompt-release window be treated as an owned write (and is the benign whitelist intentionally limited to delivery-mirror/gateway-injected)?

We can provide full logs, watch timelines, the elimination-run artifacts, and run the diagnostic above on request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions