Skip to content

Serialize session recovery snapshots / 串行化会话恢复快照#6074

Merged
SivanCola merged 3 commits into
esengine:main-v2from
SivanCola:fix/session-recovery-handoff-serialization
Jul 6, 2026
Merged

Serialize session recovery snapshots / 串行化会话恢复快照#6074
SivanCola merged 3 commits into
esengine:main-v2from
SivanCola:fix/session-recovery-handoff-serialization

Conversation

@SivanCola

@SivanCola SivanCola commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Serialize the controller snapshot/recovery handoff with a controller-local mutex.
  • Serialize every session/path swap (NewSession, ClearSession, forkNamed, Branch, SwitchBranch, Resume, SetSessionPath) and the post-cancel transcript flush (replaceSessionAfterCancel) behind the same mutex, so a swap or cancel-flush can never interleave with an in-flight save/recovery handoff.
  • Add regression tests: one blocks the first recovery callback and races additional snapshots; the other races SetSessionPath, Resume, and the cancel flush against a blocked handoff and verifies none of them move controller state until the handoff commits, with only one recovery transcript created.

Root Cause

The session file layer already serialized individual file writes per path, but controller snapshot recovery also updates controller-owned state outside that file lock: the active session path, guardian path, checkpoints, and rewrite baseline. If another snapshot entered while the first recovery handoff was still in progress, it could observe the stale path or baseline and treat the same in-memory transcript as a new snapshot conflict. In desktop autosave/action-save overlap, that produced repeated recovery branches and duplicate recovery notices for one conversation.

The same tear exists on the swap side: snapshot reads sessionPath and the executor session as two separate steps, and the session-swap paths mutated both without any coordination. A swap landing between the two reads pairs the old path with the new session — best case that manufactures the same spurious conflict/recovery cascade (including adoptDiskSession silently reverting a just-completed switch); worst case an append-shaped content overlap writes one transcript's messages into another transcript's file, because the freshly loaded session has no persisted baseline for the old path and the revision CAS is disarmed. replaceSessionAfterCancel was a second save/recovery handoff running entirely outside the lock, and ClearSession had a window where a racing save could resurrect the just-removed transcript.

Solution

Make Controller.snapshot serialize the full save/recovery/meta-update sequence with a controller-local snapshotMu. The lock is scoped to the controller and does not hold App.mu, so recovery callbacks can still update desktop tab state without a lock-order inversion. Once the first recovery commits the new path and baseline, later snapshots see the updated controller state and save normally instead of deriving another recovery branch.

Extend the same mutex over the swap sections of the session lifecycle paths and over the whole cancel-flush cleanup — including the in-memory truncation, so a save already holding the lock can never capture a half-cleaned transcript and adopt the longer on-disk partial back over it. ClearSession holds it from artifact removal through the swap; Resume releases it before recoverInterruptedTurn/maybeColdResumePrune, which snapshot on their own (the lock is not reentrant, and the field comment now documents that constraint).

Verification

  • go test ./internal/control -run 'TestConcurrentSnapshotsShareSingleRecoveryHandoff|TestSessionSwapWaitsForRecoveryHandoff' -count=1
  • go test -race ./internal/control -run 'TestConcurrentSnapshotsShareSingleRecoveryHandoff|TestSessionSwapWaitsForRecoveryHandoff' -count=1
  • go test ./internal/control -count=1 and go test -race ./internal/control -count=1
  • go test ./...
  • cd desktop && go test ./...

@SivanCola SivanCola requested a review from esengine as a code owner July 6, 2026 07:08
@github-actions github-actions Bot added v2 Go rewrite (1.x) — main-v2 branch, active development agent Core agent loop (internal/agent, internal/control) labels Jul 6, 2026
SivanCola added 2 commits July 6, 2026 15:38
snapshotMu closed the snapshot-vs-snapshot recovery race, but snapshot
still reads sessionPath and the executor session as two separate steps,
and every session/path swap mutated that state without the lock. A swap
landing between the two reads pairs the old path with the new session:
best case it manufactures the same spurious conflict/recovery cascade,
worst case an append-shaped mismatch writes one transcript's messages
into another transcript's file (the loaded session has no persisted
baseline for the old path, so the revision CAS is disarmed).

Take snapshotMu across the swap sections of NewSession, ClearSession,
forkNamed, Branch, SwitchBranch, Resume, and SetSessionPath. ClearSession
holds it from artifact removal onward so a racing save cannot resurrect
the just-removed transcript. Resume releases it before
recoverInterruptedTurn/maybeColdResumePrune, which snapshot on their own
(the lock is not reentrant).

replaceSessionAfterCancel was a second save/recovery handoff outside the
lock: its post-cancel flush calls SaveRewrite and recoverSnapshotConflict
directly. Serialize it the same way and read the path under the lock so
an in-flight recovery retarget cannot leave it stale.

Regression test blocks the first recovery callback and races
SetSessionPath, Resume, and the cancel flush against the in-flight
handoff: none may complete until the handoff commits, and only one
recovery transcript may exist afterwards.
replaceSessionAfterCancel took snapshotMu only around the SaveRewrite/
recovery flush, but the in-memory Replace ran before it. A mid-turn
autosave already holding the lock could then capture the truncated
transcript mid-save, read the longer partial autosave on disk as a
stale-prefix conflict, and adopt it back into the executor — silently
undoing the cancel cleanup before the flush could persist it, and
pointing the later SaveRewrite at the resurrected session.

Hold snapshotMu from the top of the cleanup so the truncation, todo
rebuild, path read, and flush are one atomic handoff.

The CancelFlush regression case now performs a real truncation (drop the
assistant reply, the shape stripTurnMessagesAfter produces) instead of a
same-content no-op, asserts the in-memory transcript is untouched while
the blocked handoff is still in flight, and verifies the truncation
lands on the recovery transcript afterwards. The blocked-phase assertion
fails against the previous code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Core agent loop (internal/agent, internal/control) v2 Go rewrite (1.x) — main-v2 branch, active development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant