fix(memory-wiki): retry transient existing-page reads to preserve human Notes blocks#98555
fix(memory-wiki): retry transient existing-page reads to preserve human Notes blocks#98555zhangqueping wants to merge 1 commit into
Conversation
…an Notes blocks Two memory-wiki source-page write paths used .catch(() => "") to read the existing vault page before re-ingest. A transient I/O failure (EIO, fs-safe path-mismatch concurrent-rewrite race) turns the read result into an empty string, which routes past preserveHumanNotesBlock and silently overwrites the user's hand-written ## Notes block with an empty template. Replace the swallow with a one-shot retry (100ms delay) in both paths: ingest.ts (source ingest) and source-page-shared.ts (shared source-page writer used by wiki syncs). If the retry still fails, throw rather than silently producing an empty string that would cause data loss. Regression tests added for both paths: - ingest.test.ts: transientEIO → retry preserves notes - source-page-shared.test.ts: re-ingest preserves notes Fixes openclaw#98345
|
Thanks for the context here. I did a careful shell check against current Current main already contains the central memory-wiki fix from the merged canonical PR, including retry/fail-closed existing-page reads for both affected writer paths and a dedicated regression test. This PR is now a conflicting duplicate landing path rather than necessary remaining work. So I’m closing this as already implemented rather than keeping a duplicate issue open. Review detailsBest possible solution: Keep the merged current-main implementation from #98360 as the canonical fix and close this conflicting duplicate branch. Do we have a high-confidence way to reproduce the issue? Yes. Current main includes source-level regression coverage that injects one existing-page read failure in the direct ingest path and one Is this the best way to solve the issue? Yes for the current main solution, not this branch. The merged canonical PR already fixed both implicated read sites and added stronger combined coverage, so this PR is no longer the best landing path. Security review: Security review cleared: The diff is confined to memory-wiki read handling and tests, with no dependency, workflow, secret, package-resolution, permission, or code-execution surface changes. AGENTS.md: found and applied where relevant. What I checked:
Likely related people:
Codex review notes: model internal, reasoning high; reviewed against 76038c3a625b; fix evidence: commit aa91c9d4a950, main fix timestamp 2026-07-01T02:55:05-07:00. |
|
ClawSweeper applied the proposed close for this PR.
|
Summary
Problem: Two memory-wiki source-page write paths (
ingest.ts:90andsource-page-shared.ts:55) read the existing vault page before re-ingest with.catch(() => ""). A transient I/O failure (EIO, fs-safepath-mismatchconcurrent-rewrite race) turns the read result into an empty string, which routes pastpreserveHumanNotesBlockand silently overwrites the user's hand-written## Notesblock with an empty template. The sync fingerprint advances, so the loss never self-heals.Solution: Replace the swallowed
.catch(() => "")with a one-shot retry (100ms delay) in both paths. If the retry still fails, throw rather than silently producing an empty string that would cause data loss. The writer already retries the exact same transientpath-mismatchrace (vault-page-write.ts:24-32) — this fix makes the reader match the writer's resilience.What changed:
extensions/memory-wiki/src/ingest.ts— +25/-2 (addreadVaultPageWithRetryhelper, replace.catch(() => "")with retry-then-throw);extensions/memory-wiki/src/source-page-shared.ts— +24/-2 (addreadExistingVaultPageTexthelper, same pattern);extensions/memory-wiki/src/ingest.test.ts— +56/-1 (regression test: transient EIO → retry preserves notes);extensions/memory-wiki/src/source-page-shared.test.ts— +42 (regression test: re-ingest preserves notes)What did NOT change:
preserveHumanNotesBlock, the writer'swriteGuardedVaultPageretry loop, the fs-safe vault layer, any other memory-wiki module, config, or protocol surface.Fixes #98345
What Problem This Solves
A P0 data-loss bug: when a memory-wiki source is re-ingested while a transient I/O failure hits the existing-page re-read, the user's hand-written
## Notesblock is permanently lost. The generated content still lands, the page persists, and the sync fingerprint advances, so the loss never self-heals on a later sync.This is the same fail-open-on-read anti-pattern the repo already fixed for config (#96469) and doctor (#98245): a swallowed read error must not be treated as "no prior data" before an overwrite.
Real behavior proof
Behavior addressed: Transient existing-page read failure during memory-wiki source re-ingest silently empties the user's
## Notesblock.Real setup tested:
fix/issue-98345-memory-wiki-notes-wipeat90670fd7c3scripts/run-vitest.mjsRoot cause: Both
ingest.ts:90andsource-page-shared.ts:55use.catch(() => "")to read the existing page. A transient I/O failure (EIO or the fs-safepath-mismatchconcurrent-rewrite race) returns empty string →preserveHumanNotesBlockis bypassed → the regenerated template (with empty## Notes) overwrites the user's hand-written notes. The writer already retries this exact race (vault-page-write.ts:31-32viaisConcurrentRewriteRace), but the reader swallows it.Exact steps or command run after this patch:
node scripts/run-vitest.mjs run extensions/memory-wiki/src/ingest.test.ts extensions/memory-wiki/src/source-page-shared.test.tsAfter-fix evidence:
Regression test: transient EIO in ingest.ts (#98345):
vi.spyOn(fs, "readFile")injects a single EIO failure on the existing-page re-readpreserveHumanNotesBlockrunsRegression test: re-ingest in source-page-shared.ts (#98345):
Observed result after the fix: A transient I/O failure on the existing-page read triggers one retry (100ms delay). If the retry succeeds,
preserveHumanNotesBlockpreserves the user's notes as intended. If the retry also fails, the error propagates (fail-closed) rather than silently producing an empty string.What was not tested: Racing two real concurrent syncs to trigger the fs-safe path-mismatch race. The EIO injection stands in for the transient failure; the retry logic is source-identical across EIO and path-mismatch. The
source-page-shared.tspath usesvault.readText(fs-safe layer), which cannot be cleanly mocked in a unit test without importing the security runtime.Evidence provenance:
scripts/run-vitest.mjson the PR head at Node 24.13 / Linux x86_64, 2026-07-01.Evidence
Vitest (6/6)
Before/after behavior
Tests and validation
Code walkthrough
The fix adds a small retry helper to each affected file and replaces the
.catch(() => "")pattern:ingest.ts —
readVaultPageWithRetry:source-page-shared.ts —
readExistingVaultPageText:Both helpers follow the same pattern: attempt the read, retry once after 100ms, throw on second failure. The 100ms delay matches the retry window used by the writer's
writeGuardedVaultPage(25ms–50ms delays, 3 attempts).Design decisions
Why retry-once instead of multi-retry like the writer? The writer retries 3 times with 25–50ms delays because the transient rewrite race resolves sub-ms. The reader faces the same race at a different point in the lifecycle — a single retry with a generous 100ms window is sufficient. If the file is persistently unreadable (permission error, disk failure), failing fast is better than blocking the ingest pipeline with retries.
Why throw on second failure instead of fall back to empty? Failing closed (throw) ensures the caller's error handling can retry the entire ingest operation. Failing open (return "") permanently loses user data with no detectable error. The existing callers already handle thrown errors — the ingest falls through to catch blocks, the source-page-shared path returns a result with error context.
Why not share the helper between the two files?
ingest.tsuses barefs.readFile(the page path is a direct filesystem path within the vault).source-page-shared.tsusesvault.readText(which goes through the fs-safe security layer). The two read methods have different signatures and dependency chains, so a shared helper would need to abstract over both — adding complexity for a 6-line retry loop.Risk checklist
What is the highest-risk area of this change? The throw-on-failure path. If an existing page is genuinely unreadable (permission error, disk corruption), the ingest/sync will now fail with an error instead of silently producing a blank page. This is the correct behavior — a detectible error is always better than silent data loss — but it may surface pre-existing filesystem issues that were previously hidden by the swallow.
Risk level: Low — the fix only changes the behavior of a
.catch(() => "")to.catch(() => { throw }). The 100ms retry covers the transient case. Persistent failures become visible errors instead of silent data loss. All existing tests pass unchanged. The writer already retries the same transient race, confirming the retry approach is correct for this codebase.Mitigation: One-shot retry covers the transient case. Persistent failure surfaces immediately as an error (no silent data loss). Callers already handle thrown errors.
Merge risk declaration: No merge risk. Pure fix of a fail-open read pattern in two memory-wiki helper functions. No config, protocol, API, or SDK surface changes. The sibling
unsafe-local.tspath (#97523) is distinct — this fix does not change its behavior.Architecture context
The memory-wiki plugin writes vault pages through two source page writer paths. Both share the same Notes-preservation contract (
preserveHumanNotesBlock) but read the existing page through different I/O layers:ingest.ts:90fs.readFile(pagePath)readVaultPageWithRetrysource-page-shared.ts:55vault.readText(pagePath)readExistingVaultPageTextBoth paths follow the same flow:
The Notes preservation relies on step 1 returning the real existing content. When step 1 returns
""(because.catch(() => "")swallowed a transient error), step 3 is skipped (theexisting ? preserve : newternary), and step 4 writes the template with an empty Notes block.The fs-safe path-mismatch race
The writer at
vault-page-write.ts:24-32documents and retries a specific transient race condition:FsSafeErrorwith codepath-mismatchThe same race can occur between step 1 (read existing page) and step 4 (write merged page): the file is replaced by a concurrent sync between the read and the write. Before this fix, if the concurrent replacement happened between step 1's read and the fs-safe guard, the read would fail with
path-mismatch,.catch(() => "")would return"", and the user's Notes would be wiped in step 4 — even though the writer would successfully retry its own operation.The fix adds parity: the reader now retries the same transient race, so the subsequent Notes preservation runs on the correct existing content.
Relation to #95614
PR #95614 added
preserveHumanNotesBlockto protect human notes on successful re-ingest. This fix closes the gap where transient read failures bypassed that protection. The fix complements #95614 rather than replacing it.Relation to #97523
Issue #97523 covers a distinct bug: unsafe-local sync deletes imported pages when the configured source path is unavailable. This fix only addresses the existing-page re-read path — it does not change the unsafe-local prune/recreate behavior.
How to test manually
Reproducing the bug requires injecting a transient I/O failure at the existing-page read site after adding human notes. The ingest.test.ts regression test does this by:
fs.readFileto inject one EIO failure on the specific page pathThe same pattern applies to the vault.readText path used by source-page-shared.ts, though that path is tested through the normal re-ingest flow (without mock injection, since vault.readText goes through the fs-safe security layer).
Linked context
preserveHumanNotesBlockon successful re-ingest — this fix covers the unreadable-existing-page path that bypassed itCurrent review state
This is a narrow data-loss fix: two
.catch(() => "")replaced by retry-once-then-throw. The approach matches the writer's existing retry pattern for the same transient race. Two regression tests added. No config, protocol, or API changes. Ready for maintainer review.Notes
The fix scope is deliberately minimal. The
.catch(() => "")fail-open pattern exists in specific code sites where read → write coherence matters. Expanding the fix to a general-purpose retry helper across all memory-wiki reads would touch callers whose failure semantics differ (log vs skip vs replace). The two affected sites are the only ones where a swallowed read can cause permanent user data loss. Any future sites added underextensions/memory-wiki/src/that read a page before overwriting it should adopt the same retry-once pattern.The 100ms delay between retries is intentionally generous. Most transient races (EIO from concurrent rename, path-mismatch from fs-safe guard) resolve in under 1ms. The 100ms window provides headroom for slower filesystems (NFS, FUSE) without meaningfully blocking the ingest pipeline — a single ingest already takes hundreds of milliseconds for compilation and indexing.