fix(memory): do not prune session index from a failed directory scan#91091
fix(memory): do not prune session index from a failed directory scan#91091amknight wants to merge 3 commits into
Conversation
|
Codex review: found issues before merge. Reviewed June 28, 2026, 1:47 AM ET / 05:47 UTC. Summary PR surface: Source +225, Tests +693, Config -1, Generated 0, Other +6. Total +923 across 22 files. Reproducibility: yes. Source inspection shows current main can turn failed or incomplete memory/session enumerations into empty listings that destructive prune/export paths treat as authoritative; I did not execute tests because this review is read-only. Review metrics: 2 noteworthy metrics.
Stored data model Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Security Review findings
Review detailsBest possible solution: Keep the scan-authority guard, repair missing state-tree classification, regenerate the lockfile on current main, resolve conflicts, and make an explicit maintainer decision on the SDK surface before merge. Do we have a high-confidence way to reproduce the issue? Yes. Source inspection shows current main can turn failed or incomplete memory/session enumerations into empty listings that destructive prune/export paths treat as authoritative; I did not execute tests because this review is read-only. Is this the best way to solve the issue? No, not in its current form. Carrying scan authority to destructive consumers is the right fix shape, but the missing whole-tree ENOENT case and inconsistent lockfile need repair before this is the best merge candidate. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 119dc4bd821e. Label changesLabel justifications:
Evidence reviewedPR surface: Source +225, Tests +693, Config -1, Generated 0, Other +6. Total +923 across 22 files. View PR surface stats
Security concerns:
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
92684c4 to
31c7719
Compare
|
/clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
/clawsweeper re-review |
|
🦞👀 Command router queued. I will update this comment with the next step. |
|
/clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
This is awaiting the new version of fs-safe being released |
39d8647 to
aa94a36
Compare
…91091) Rebased onto main's corpus-based session enumeration. A failed sessions or memory scan (transient NFS EIO/ESTALE) no longer masquerades as an empty listing that wipes the session/memory index or the exported QMD corpus. - scanSessionFilesForAgent() returns {ok, files}; resolveMemorySessionSyncPlan gates the stale-row prune on scanOk (pruneStaleRows). Indexing stays corpus-driven (archive artifacts + dreaming/cron classification); only the ok-flag gates destructive prunes. - runInPlaceReindex aborts via assertReindexScanOk before publishing an empty rebuild; exportSessions throws before reconciling QMD deletes.
0.4.0 adds `failedDirs` to WalkDirectoryResult so walkDirectory surfaces readdir failures instead of swallowing them (consumed by the memory scan in the next commit). 0.4.0 is well past the 48h minimumReleaseAge window, so it no longer needs the release-age exclude.
Replace the manual fs.readdir root-probes in scanMemoryFiles with walkDirectory's failedDirs (fs-safe 0.4.0). collectMemoryFilesFromDir now reports whether the walk was complete; a failed readdir at any depth (not just the probed root) flags the scan not-ok so destructive prunes skip an incomplete listing rather than wiping unseen files.
aa94a36 to
75a1da0
Compare
Summary
A full memory sync treats source enumerations as the source of truth for destructive maintenance: stale-row pruning of the session and memory-file indexes (
files+chunks+ vectors + FTS), full reindex rebuilds, and exported-session reconciliation. These enumerations swallowed filesystem errors and returned an empty listing, so a single transient scan failure (NFSEIO/ESTALE, permission blip) was indistinguishable from an empty directory and wiped the corresponding index — followed by an expensive full re-read/re-embed after the next successful scan. The failed-scan prune test in this PR fails without the guard (the indexed row gets deleted), confirming the hazard is live.Sessions
packages/memory-host-sdk: addscanSessionFilesForAgent(agentId): { ok, files }.ok: truemeans the directory was actually read or authoritatively does not exist (see below);ok: falsemeans the scan failed.listSessionFilesForAgentdelegates to it and is unchanged for its non-destructive callers.resolveMemorySessionSyncPlantakes a requiredscanOkand returnspruneStaleRows, true only for a full enumeration that authoritatively read the directory.syncSessionFilesgates the stale-row prune onpruneStaleRows. A failed scan skips the prune (with a warning) and leaves the index intact for the next successful enumeration.MemorySourceSyncPlancarriesscanOk;runSyncno longer clearsdirty/sessionsDirty/sessionsDirtyFilesafter a failed scan, so the next eligible sync retries instead of reporting a clean sync that indexed nothing.scanMemoryFiles(): { ok, files }(exported viaengine-storageand the plugin SDK barrel) with explicit rootreaddirprobes — fs-safewalkDirectoryswallows readdir errors even at the walk root, so an lstat-only check would still reportokon the exact failure class this PR targets.syncMemoryFilesskips its stale-row prune on a failed enumeration.assertReindexScanOk) instead of swapping in an empty index built from a failed scan; the safe path restores the original DB on throw, the unsafe path skipswriteMetaso the next sync retries the rebuild.exportSessionsskips its reconcile-delete pass on a failed listing so one blip cannot delete the exported session-markdown corpus.cli.runtime.ts(scanMemoryFiles→diagnoseMemoryFiles,scanSessionFiles→diagnoseSessionFiles) to avoid colliding with the new SDK export surfaced through the same barrel.Why skip on failed, not on empty
Deletions/resets archive to usage-counted
.deleted./.reset.artifacts that still appear in the listing, anddisk-budgetcan physically remove the last archive — so an authoritatively empty directory with a non-empty index is legitimately reachable. In that case we must still prune, or deleted-session content would linger in search forever. Skipping only on a failed scan prevents the wipe without introducing persistent orphan rows.Known residual gaps (follow-ups)
@openclaw/fs-safewalkDirectoryreports no errors); blast radius is the affected subtree, not the whole index. A clean fix needs an additive fs-safe API (e.g.WalkDirectoryResult.failedDirCount), which would also remove the rootreaddirprobes.resolveCanonicalRootMemoryFileswallows workspace-root readdir errors; bounded to the single root memory file.Verification
Rebased on
origin/main(e3a6da0f51). The rebase dropped the previously includedmanager-sync-ops.startup-catchup.test.tsassertion fix — it landed onmainindependently and the commit was skipped as already applied.pnpm test extensions/memory-core— 905/905 passing (66 files), including the newmanager-sync-ops.prune-guard.test.ts,manager-sync-ops.targeted-delta.test.ts,manager-session-sync-state.test.tsmatrix, andmain's atomic-reindex suites added since the original base.pnpm test packages/memory-host-sdk/src/host/session-files.test.ts packages/memory-host-sdk/src/host/internal.test.ts— 31/31 passing (scan ok / authoritatively-empty / absent-dir-with-parent / unreachable-tree / failed readdir-after-lstat / failed extra path;listSessionFilesForAgentandlistMemoryFilesunchanged on failure).pnpm tsgo:core,pnpm tsgo:extensions,pnpm tsgo:test:packages,pnpm tsgo:test:extensions— clean.main's new swap machinery (865fdab075, "preserve live SQLite index during swaps"):runMemoryAtomicReindexpublishes viaswapMemoryIndexFilesonly afterbuild()resolves;assertReindexScanOkthrows insidebuild()before any dirty-flag clearing, so the temp DB is closed and removed, the original DB handle and FTS/vector state are restored, and dirty state is retained for retry.expected ['sessions/main/thread.jsonl'] to deeply equal []).Notes
{ ok, files }scan result and must never serve a failed scan as authoritative, or it reintroduces the wipe hazard this PR closes.