Skip to content

EmbeddedAttemptSessionTakeoverError on long runs: one run co-executes in wrapper (cron-nested/main) + session lane → fence aborts (root-cause repro; refs #84460) #91236

Description

@DouglasCherry

Summary

EmbeddedAttemptSessionTakeoverError: session file changed while embedded prompt lock was released fires deterministically on long-running embedded agent turns because a single run executes across two command lanes at once — a wrapper lane (cron-nested for scheduled jobs, main for delivery/inbound) and the per-session lane (session:agent:<id>:...:run:<runId>). Both lanes append to the same session JSONL, so when one lane releases its prompt-lock during the model call, the other lane's append advances the file fingerprint and the session-file fence aborts the run.

The fence is working as designed — it is detecting a genuine concurrent write. The root cause is the double-lane co-execution of one run, not the fence.

This is the same symptom as #84460 (closed as not-planned) and related to #83510 / #31010, but this report adds a clean cron-path reproduction and a root-cause analysis, since the impact (scheduled jobs failing daily) is significant.

Affected version

OpenClaw 2026.5.24-beta.2 (ghcr.io/openclaw/openclaw:2026.5.24-beta.2), single-host docker gateway.

Evidence (one morning's scheduled cron runs, all aborted)

lane=cron-nested durationMs=180340  agents/cashflow/sessions/<sid>.jsonl
lane=cron-nested durationMs=97280   agents/main/sessions/<sid>.jsonl
lane=cron-nested durationMs=82233   agents/collections/sessions/<sid>.jsonl
lane=cron-nested durationMs=54274   agents/credit/sessions/<sid>.jsonl

Each event is emitted twice in the same millisecond — once for lane=cron-nested and once for lane=session:agent:<id>:cron:<cronId>:run:<runId> — referencing the same session file and the same runId. The interactive delivery path shows the identical pattern with lane=main + lane=session:agent:<id>:discord:channel:<cid>.

Key behavioural signal: the failure scales with run duration. Turns under ~50–55s complete cleanly; turns past that reliably abort (54s, 82s, 97s, 180s above). A direct openclaw agent turn (no wrapper lane) ran 102s clean — i.e. it only fails when a wrapper lane is also in play.

Root cause analysis

  1. applyGatewayLaneConcurrency() wires independent lanes: cron, cron-nested, main, subagent. A scheduled/delivered run is enqueued on the wrapper lane and executed on a per-session lane; both end up invoking the embedded runner against the same session file concurrently.
  2. The session-file fence (assertSessionFileFencesessionFenceAdvanceIsBenign) only treats a concurrent append as benign when every appended line passes isTranscriptOnlyOpenClawAssistantLine. The wrapper lane's concurrent writes include non-transcript lines (tool calls, etc.), so the append is judged non-benign and the run throws EmbeddedAttemptSessionTakeoverError.

Proposed fix (at root)

Either:

  • (Preferred) Single-lane execution: ensure one run executes in exactly one lane — the wrapper (cron-nested/main) should delegate to and await the session lane rather than co-executing the embedded turn, so only one writer touches the session file; or
  • Ownership tracking: register the wrapper lane's writes via recordOwnedSessionFileWrite(...) so assertSessionFileFence recognises them as owned (same-process) writes instead of foreign takeovers.

Note: simply widening sessionFenceAdvanceIsBenign (e.g. return lines.length > 0) suppresses the error but masks real concurrent-write contention and risks interleaved session corruption — it is a workaround, not a fix.

Impact

Repro

  1. Configure an agent with a tool-using task that takes >60s of model time.
  2. Trigger it via the cron-nested (scheduled) or main (delivery) path — not a direct agent invocation.
  3. Observe paired lane task error entries for the wrapper lane and the session lane against the same session file, with EmbeddedAttemptSessionTakeoverError.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions