perf(memory): target session delta syncs with the dirty set instead of full directory scans#91841
Conversation
|
Codex review: needs maintainer review before merge. Reviewed June 10, 2026, 10:53 AM ET / 14:53 UTC. Summary PR surface: Source +21, Tests +183. Total +204 across 2 files. Reproducibility: unclear. The review failed before ClawSweeper could establish a reproduction path. Review metrics: none identified. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Risk before merge
Maintainer options:
Next step before merge
Review detailsBest possible solution: Retry the Codex review after fixing the execution failure. Do we have a high-confidence way to reproduce the issue? Unclear. The review failed before ClawSweeper could establish a reproduction path. Is this the best way to solve the issue? Unclear. Retry the review first so ClawSweeper can evaluate the actual issue and fix direction. AGENTS.md: unclear because the file could not be read completely. Codex review notes: reasoning high; reviewed against ac21e89c13e4. Label changesLabel changes:
Label justifications:
Evidence reviewedPR surface: Source +21, Tests +183. Total +204 across 2 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
ed11c85
into
ak/memory-sync-prune-guard
Co-authored-by: Alex Knight <[email protected]>
Co-authored-by: Alex Knight <[email protected]>
Summary
Every 5s-debounced session delta sync calls
sync({ reason: "session-delta" })with no targets, sosyncSessionFilesperforms a full sessions-directory enumeration — and then skips every listed file that is not in the dirty set anyway. The enumeration's only unique contribution to a delta sync is stale-row prune reconciliation; the indexing work is fully determined bysessionsDirtyFiles, which the engine populated from transcript events before triggering the sync.On networked filesystems these scans are expensive: an NFSv4.1 profile of a real deployment measured full listings of
agents/main/sessionsat 3.4–3.8s each (~100 entries; the client requests a broad READDIR attribute set, which the backing server satisfies with per-entry metadata reads). Trace-level attribution later showed the specific scans that profile captured came from per-save archive-retention cleanup — fixed separately in #91957 — not from memory delta syncs (that deployment does not run memory session sync). The per-scan cost numbers still stand, and they are what every delta sync pays on such storage whenever memory session sync is enabled.This PR makes delta syncs use the targeted path that already exists (
runMemoryTargetedSessionSync):processSessionDeltaBatchnow passes the accumulated dirty set as explicitsessionFilestargets → the sessions-dir enumeration is skipped entirely. The targeted path clears only the synced files from the dirty set (files dirtied mid-sync survive) and retains targets as dirty on failure.SESSION_PRUNE_RECONCILE_INTERVAL_MS(15 min), a delta sync falls back to the full enumeration so index rows for files deleted out-of-band (another process/node on shared storage) still get pruned. The timestamp only advances when an authoritative prune actually completed, so a failed scan retries on the next batch.Behavior notes
{ ok, files }), so a failed reconcile scan cannot prune and dirty state is retained for retry.Identified while decomposing the NFS readdir-load report behind the #91081 discussion (its "avoid full scans when a targeted dirty-file set exists" recommendation). With this change, deployments running memory session sync stop paying a full directory scan per delta batch; the remaining enumeration cost (boot catch-up, periodic reconcile, qmd export, and the store-save retention sweeps addressed by #91957) is the much smaller surface the coalescing/TTL discussion in #91081 addresses separately.
Verification
node scripts/run-vitest.mjs extensions/memory-core/src/memory/manager-sync-ops.targeted-delta.test.ts extensions/memory-core/src/memory/manager-sync-ops.prune-guard.test.ts extensions/memory-core/src/memory/manager-targeted-sync.test.ts extensions/memory-core/src/memory/manager-session-sync-state.test.ts— 19/19 (5 new).node scripts/run-vitest.mjs extensions/memory-core/src/memory/manager-sync-ops.archive-delta-bypass.test.ts— 4/4.pnpm tsgo:extensionsandpnpm tsgo:extensions:test— clean.node scripts/run-oxlint.mjson touched files — clean.scandirevents on the sessions dir during steady-state activity; expected: one per 15-min reconcile window instead of one per 5s delta batch.