Skip to content

Harden session CAS meta ledger against torn reads and lost updates / 加固会话 CAS 元数据账本防撕裂读与丢失更新#6025

Merged
SivanCola merged 2 commits into
esengine:main-v2from
SivanCola:fix/session-meta-cas-hardening
Jul 5, 2026
Merged

Harden session CAS meta ledger against torn reads and lost updates / 加固会话 CAS 元数据账本防撕裂读与丢失更新#6025
SivanCola merged 2 commits into
esengine:main-v2from
SivanCola:fix/session-meta-cas-hardening

Conversation

@SivanCola

Copy link
Copy Markdown
Collaborator

Summary

Windows users on v1.16.x–v1.17.0 hit recurring bogus "save conflict" warnings during normal single-window chats — every app start could fork another recovery branch, accumulating dozens of *-recovery-*.jsonl files and sidecars (#6014: 24 recovery files; #5998, #6003, and the "session changed on disk; adopted the newer transcript" loop in #5996).

The transcript CAS itself (from #5856) is sound as a goal; this PR hardens its ledger so it can no longer manufacture conflicts out of transient Windows filesystem behavior. It deliberately does not redesign the mechanism (kept minimal for a hotfix).

Root causes

The CAS ledger (Revision/ContentDigest in the .jsonl.meta sidecar) was maintained without protection:

  1. Unreadable ledger silently became revision 0. sessionContentRevision swallowed read errors into (0, nil); LoadSession did the same when establishing the baseline; recordSessionContentRevision swallowed them into "don't write the ledger at all". On Windows a sidecar read can fail transiently (AV/indexer handles; fileutil.ReplaceFile's non-atomic copyOnto fallback tears readers). Any of these desynced baseline vs. ledger, and the next honest save misread as a foreign write → Diverged → recovery branch. This is the "triggers on every app start" shape.
  2. Unlocked read-modify-write rolled the ledger back. RenameSession, topic-title propagation (updateTopicSessionTitles, runs after autosaves), legacy topic migration, and restoreSessionTopicIndex load→mutate→save meta without LockSessionMetaPath; a concurrent save's revision bump landing in that window was overwritten with the stale value. The write-side window is amplified to ~720ms by ReplaceFile's retry ladder on Windows.
  3. Index write failure stranded the baseline. save() returned an error after the transcript and revision were already durable if writeSessionEventIndex failed (AV-held handle), skipping markPersisted — baseline behind disk → next save conflicts.
  4. Snapshots were captured before locking. Concurrent in-process savers (turn-end, periodic autosave, desktop TurnDone) could land out of order; the stalest capture then read the newer transcript as a stale-prefix conflict, producing the repeated "adopted the newer transcript" notices.

Changes

  • loadBranchMetaRetry: short backoff (20/50/100ms) over I/O and JSON-decode failures, separating Windows torn reads from real corruption; missing sidecar stays an immediate legitimate ok=false.
  • sessionContentRevision / recordSessionContentRevision now fail the save on a persistently unreadable ledger instead of fabricating revision 0 / silently skipping the bump. The transcript bytes are already durable; the autosave retry heals once the sidecar reads cleanly.
  • LoadSession with an unreadable sidecar still opens the session but records a revision-unknown baseline (sessionPersistState.revisionKnown): revision-based CAS is disarmed (content digest/prefix checks still catch real divergence), and ownership checks (ownsPersistedState) anchor on digest+version so owned rewrites (compaction) don't become permanent conflicts. A missing sidecar remains a known revision-0 baseline; a successful save re-learns the real revision.
  • The four unlocked meta RMW paths now hold the per-path meta lock for the whole load→mutate→save cycle (the legacy-migration pass runs outside the caller's lock — it takes its own per-session locks).
  • writeSessionEventIndex failures downgrade to a warning at both save() call sites; the index is a listing accelerator and must not strand the baseline.
  • save() captures the snapshot under the save locks, eliminating out-of-order captures (lock-order audited: saveLock → Session.mu is the pre-existing order; no reverse path exists).

Tests

New regression suite (session_meta_ledger_test.go, all -race):

  • Corrupt ledger → save fails closed, no recovery branch, ledger untouched; heals after repair.
  • Unreadable-meta load → session opens, appends don't conflict, owned rewrite (SaveRewrite) proceeds via digest+version ownership. Reverting the fix reproduces the original bogus diverged … from snapshot revision 0 error.
  • Missing meta stays a known zero baseline (behavior unchanged).
  • Event-index path pre-created as a directory → save still succeeds and the baseline advances.
  • Concurrent SaveSnapshot × RenameSession/SetBranchModel hammer: revision stays monotonic, no conflicts.
  • Deterministic under-lock capture test (fails with the old pre-lock capture) plus a concurrent-savers zero-conflict stress.

Full suites: go test ./internal/agent -race, ./internal/control, desktop go test . — all pass; GOOS=windows GOARCH=amd64 builds clean in both modules.

Cache impact

Cache-impact: none. Session persistence, sidecar bookkeeping, and lock scope only; no provider-visible prompts, tool schemas, request serialization, or compaction changes.

Fixes #6014. Fixes #5998. Fixes #5996. Refs #6003 (the recovery-branch loop; its bash-sandbox half is tracked by #5937).

…updates

The transcript CAS keyed off the .jsonl.meta sidecar, but the ledger was
maintained unprotected. On Windows this manufactured conflicts in normal
single-window use: recovery branches forked on every app start, recovery
files accumulated, and 'adopted the newer transcript' notices looped.

- Retry transient sidecar reads (I/O and JSON-decode failures, 20/50/100ms)
  before giving up: ReplaceFile's non-atomic copyOnto fallback tears
  concurrent readers, and AV/indexer handles fail reads transiently.
- Stop swallowing unreadable-ledger errors into revision 0: a persistently
  unreadable ledger now fails the save (transcript bytes are already
  durable; the autosave retry heals it) instead of desyncing the baseline
  and misreading the next honest save as a foreign write.
- LoadSession with an unreadable sidecar still opens the session but marks
  the baseline revision-unknown: revision CAS is disarmed (digest/prefix
  checks still catch real divergence) and owned rewrites anchor on
  digest+version, so compaction cannot become a permanent conflict. A
  missing sidecar remains a known revision-0 baseline.
- Hold the per-path meta lock across the whole read-modify-write in
  RenameSession, topic-title propagation, legacy topic migration, and
  restoreSessionTopicIndex, so a concurrent save's revision bump can no
  longer be rolled back by a stale read-back.
- Downgrade event-index write failures to a warning at both save() sites:
  the index is a listing accelerator and must not strand the baseline
  behind a transcript+revision that already landed.
- Capture save() snapshots under the save locks so concurrent in-process
  savers can no longer land out of order and misread the newer transcript
  as a stale-prefix conflict.

Regression tests cover fail-closed saves on a corrupt ledger, unknown-
revision baselines (load, append, owned rewrite), missing-meta behavior,
index-failure tolerance, a concurrent writers hammer asserting revision
monotonicity, and deterministic under-lock capture.

Fixes esengine#6014. Fixes esengine#5998. Fixes esengine#5996. Refs esengine#6003.
@SivanCola SivanCola requested a review from esengine as a code owner July 5, 2026 14:34
@github-actions github-actions Bot added v2 Go rewrite (1.x) — main-v2 branch, active development desktop Wails desktop app (desktop/**) agent Core agent loop (internal/agent, internal/control) and removed v2 Go rewrite (1.x) — main-v2 branch, active development labels Jul 5, 2026
A save that lands its transcript bytes and then fails to record the
revision (fail-closed recordSessionContentRevision, or a crash between
the two writes) leaves the sidecar describing older content. The
same-content retry then took the up-to-date shortcut and never touched
the ledger again, so revision/content_digest stayed stale until the
next content-bearing save — contradicting the "later save can bump the
revision" recovery the fail-closed path promises. Idle sessions never
healed at all.

checkSnapshotWrite now surfaces the ledger digest it already reads, and
the up-to-date path records the deferred revision when that digest no
longer matches the on-disk transcript, reproducing the exact state the
interrupted save would have left. A missing or digest-less legacy
sidecar stays untouched: stamping those would bump revisions other
runtimes still hold as baselines.
@SivanCola SivanCola merged commit 8ab68bf into esengine:main-v2 Jul 5, 2026
14 checks passed
SivanCola added a commit that referenced this pull request Jul 5, 2026
…rs, retire lock sidecars

Follow-up polish on the lease/CAS stack after the #6023/#6025 hotfixes,
closing the remaining ways a healthy single-window session could still
present as busy, leak holder details, or litter the session directory.

- Reclaim now arbitrates on the OS lock alone: a missing or unreadable
  lease.json (user cleanup, AV quarantine, crash tear) with a free lock is
  a leftover, not a holder. Previously an orphaned in-process entry whose
  info file disappeared could never be reclaimed - the fallback acquire
  re-hit the orphaned entry forever and every rebuild stayed busy. The
  desktop gate mirrors this: nil lease info allows the reclaim attempt,
  foreign holders are still refused.
- Raw lease errors (session path + host-pid-writer id) can no longer reach
  the UI: startup bind failures (StartupErr feeds the topbar banner and tab
  metas), ClearSession's bridge returns, and the recovery-lease callback
  (chat notices via snapshot errors) all render the user-facing busy
  message; raw details stay in slog. The busy message drops its dangling
  'before changing this setting' clause when no setting applies.
- SessionLease.Release retires the .lock/.lease.lock sidecars through the
  existing atomic take-then-remove helpers (non-blocking, no-op against any
  live holder or in-flight save), so ordinary use stops accumulating lock
  files that previously waited for the next boot reconcile (#6014).
- Autosave failures warn once per burst and at most once per 5 minutes per
  tab: retries are slog-only, the user notice fires when the burst gives
  up. Explicit action saves keep immediate feedback. Every failure is still
  logged.
- SaveRecoveryBranch tolerates event-index write failures like the other
  save paths: the recovery transcript and meta are already durable, so a
  failed listing accelerator no longer discards a successful recovery.
- Lease test suite now feeds user-shape (mixed-case) paths to the APIs
  under test instead of pre-canonicalized ones - the pattern that let the
  Windows case-fold mismatch (#5999) escape the suite - plus regressions
  for orphan-without-info, corrupt-info, foreign-info-refused, sidecar
  retirement on release, StartupErr sanitization, and the autosave warning
  debounce.

Refs #6014 #5999 #6027.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Core agent loop (internal/agent, internal/control) desktop Wails desktop app (desktop/**)

Projects

None yet

1 participant