Skip to content

fix(memory-wiki): retry transient existing-page reads in wiki_apply and chatgpt import#98787

Merged
vincentkoc merged 2 commits into
openclaw:mainfrom
yetval:fix/memory-wiki-synthesis-transient-read
Jul 2, 2026
Merged

fix(memory-wiki): retry transient existing-page reads in wiki_apply and chatgpt import#98787
vincentkoc merged 2 commits into
openclaw:mainfrom
yetval:fix/memory-wiki-synthesis-transient-read

Conversation

@yetval

@yetval yetval commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Re-running wiki_apply create_synthesis for an existing synthesis page, or re-running the ChatGPT conversations import over already-imported pages, silently wipes the user's hand-written ## Notes block and drops hand-added frontmatter when the existing-page read fails transiently (EBUSY/EMFILE/ENFILE style faults). The page is treated as brand-new and the regenerated content replaces the user's edits without any error surfacing.

PR #98360 fixed this defect class for ingest.ts and source-page-shared.ts (issue #98345) but did not cover these two surfaces.

Root cause

Both surfaces swallow every existing-page read error and continue with an empty page.

Before, extensions/memory-wiki/src/apply.ts:231 in applyCreateSynthesisMutation:

const existing = await root.readText(pagePath).catch(() => "");

Before, extensions/memory-wiki/src/chatgpt-import.ts:762 in importChatGptConversations:

const existing = await fs.readFile(absolutePath, "utf8").catch(() => "");

With an empty existing, applyCreateSynthesisMutation keeps none of the frontmatter that only lived on disk (for example a hand-added privacyTier) and buildSynthesisBody regenerates an empty ## Notes scaffold, which writeWikiPage then persists. In the importer, preserveExistingPageBlocks receives an empty existing, so the human notes and ## Related blocks are not carried over, the write is misclassified as create instead of update or skip, and the rollback record snapshots an empty existing for that page.

Fix

Route both reads through a local retry-once helper that treats only a genuinely missing page as new and propagates a failure that persists across the retry, the same shape #98360 shipped for ingest and imported-source pages. Unlike those two call sites, these reads have no existence gate (a missing page is the normal create path), so the helpers translate missing-page errors to an empty string instead of rethrowing them.

After, extensions/memory-wiki/src/apply.ts:

async function readExistingWikiPage(root: VaultRoot, pagePath: string): Promise<string> {
  try {
    return await root.readText(pagePath);
  } catch (error) {
    if (isMissingWikiPageError(error)) {
      return "";
    }
    try {
      return await root.readText(pagePath);
    } catch (retryError) {
      if (isMissingWikiPageError(retryError)) {
        return "";
      }
      throw retryError;
    }
  }
}

isMissingWikiPageError matches FsSafeError codes not-found and path-alias, the same missing-page classification okf.ts already uses for vault stat calls. chatgpt-import.ts gets the equivalent readExistingConversationPage keyed to plain fs ENOENT.

writeWikiPage's change-detection read at apply.ts:207 is routed through the same helper so the file has one canonical existing-page read; a transient failure there previously caused only a spurious identical rewrite, never data loss.

Why this is the right boundary

  • apply.ts owns the synthesis page lifecycle and chatgpt-import.ts owns imported conversation pages, so each gets the same local helper pattern fix(memory-wiki): preserve notes after transient page reads #98360 established in ingest.ts and source-page-shared.ts, keyed to its own missing-page error shape (FsSafeError for the vault root, ENOENT for plain fs).
  • These were the last two paths that merge preserved human content into a rewrite from a swallowed existing-page read. The remaining .catch(() => "") reads in the plugin are not user-content-lossy: okf.ts:461 and compile.ts:1377 use the read only for change detection before writing fully caller-computed content, and okf.ts:496 fails safe by skipping removal.
  • compile.ts writeManagedMarkdownFile and writeDashboardPage fall back to a fresh scaffold for plugin-generated index/dashboard/report pages; a transient read failure there can reset plugin-generated scaffolding but does not touch user-authored notes pages. Flagging as a candidate follow-up if the same retry treatment is wanted there.
  • Fail-closed on persistent read errors matches the direction fix(memory-wiki): preserve notes after transient page reads #98360 shipped: the mutation or import errors out and can be retried instead of silently destroying user edits.

Verification

  • node scripts/run-vitest.mjs extensions/memory-wiki/src/wiki-notes-read-retry.test.ts extensions/memory-wiki/src/apply.test.ts extensions/memory-wiki/src/import-runs.test.ts: 3 files, 13 tests passed with the patch.
  • The two new regression tests fail on pristine main 17482a4 (notes and frontmatter wiped) and pass with the patch; the two pre-existing fix(memory-wiki): preserve notes after transient page reads #98360 tests in the same suite stay green.
  • node scripts/run-oxlint.mjs on the three changed files: clean.
  • oxfmt --check on the three changed files: clean.
  • node scripts/run-tsgo.mjs -p tsconfig.extensions.json and -p test/tsconfig/tsconfig.extensions.test.json: clean.

Real behavior proof

Behavior addressed: a single transient read failure of the existing page during a create_synthesis re-run or a ChatGPT re-import silently empties the user's ## Notes block, drops hand-added frontmatter, and misclassifies the import operation.
Real environment tested: drove applyMemoryWikiMutation (the production wiki_apply mutation entry point) and importChatGptConversations (the production ChatGPT import entry point) against a real on-disk vault, on pristine main 17482a4 and on the patched tree with identical inputs; vault initialization, markdown render/parse, compile, and all page writes were the real runtime on disk; the one injected fault was a single EBUSY error on the existing-page read.
Exact steps or command run after this patch: create the synthesis page via applyMemoryWikiMutation create_synthesis "Release Plan", hand-edit its notes block and add privacyTier frontmatter on disk, re-run the same mutation with one EBUSY injected on the syntheses/release-plan.md read; separately import a ChatGPT export via importChatGptConversations, hand-edit the conversation page notes, re-run the identical import with one EBUSY injected on that page read; then print the resulting page files.
Evidence after fix:

# BEFORE (pristine main 17482a4026)
$ wiki_apply create_synthesis 'Release Plan' (re-run with one EBUSY on the existing-page read)
syntheses/release-plan.md frontmatter privacyTier line: MISSING (hand-added frontmatter dropped)
syntheses/release-plan.md notes section:
## Notes
<!-- openclaw:human:start -->
<!-- openclaw:human:end -->

$ openclaw wiki chatgpt-import (re-run of same export with one EBUSY on the existing-page read)
import result: created=1 updated=0 skipped=0
chatgpt-2024-04-06-123456781234123412341234567890ab.md notes section:
## Notes
<!-- openclaw:human:start -->
<!-- openclaw:human:end -->

# AFTER (this patch, identical inputs)
$ wiki_apply create_synthesis 'Release Plan' (re-run with one EBUSY on the existing-page read)
syntheses/release-plan.md frontmatter privacyTier line: privacyTier: sensitive
syntheses/release-plan.md notes section:
## Notes
<!-- openclaw:human:start -->
Ship gate: legal sign-off required before GA.
<!-- openclaw:human:end -->

$ openclaw wiki chatgpt-import (re-run of same export with one EBUSY on the existing-page read)
import result: created=0 updated=0 skipped=1
chatgpt-2024-04-06-123456781234123412341234567890ab.md notes section:
## Notes
<!-- openclaw:human:start -->
HUMAN NOTE: verified against the airline booking.
<!-- openclaw:human:end -->

Observed result after fix: after one transient EBUSY on the existing-page read, the re-synthesized page keeps the user's notes line and hand-added privacyTier frontmatter, and the re-imported conversation page keeps its notes and is classified skipped instead of being recreated with an empty notes block.
What was not tested: no live gateway session drove wiki_apply over RPC; the EBUSY fault was injected at the read seam rather than produced by a real file lock; full build not run on this machine.

Related

@clawsweeper

clawsweeper Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 1, 2026, 9:18 PM ET / 01:18 UTC.

Summary
The PR changes memory-wiki synthesis and ChatGPT import existing-page reads to retry once, treat only missing pages as new, and adds regression tests for both paths.

PR surface: Source +40, Tests +206. Total +246 across 3 files.

Reproducibility: yes. Current main still has catch-to-empty reads at the synthesis and ChatGPT import sites, and a one-shot read fault has a direct source-level path to losing Notes/frontmatter; I did not execute the fault injection because this review is read-only.

Review metrics: 1 noteworthy metric.

  • Existing-page read behavior: 3 changed from swallow-as-empty to retry/fail-closed. Persistent non-missing read failures now surface instead of treating existing wiki pages as brand-new before a write.

Stored data model
Persistent data-model change detected: serialized state: extensions/memory-wiki/src/chatgpt-import.ts, unknown-data-model-change: extensions/memory-wiki/src/apply.ts, unknown-data-model-change: extensions/memory-wiki/src/wiki-notes-read-retry.test.ts, vector/embedding metadata: extensions/memory-wiki/src/apply.ts, vector/embedding metadata: extensions/memory-wiki/src/chatgpt-import.ts, vector/embedding metadata: extensions/memory-wiki/src/wiki-notes-read-retry.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: canonical
Canonical: #98787
Summary: This PR is the current canonical item for the remaining memory-wiki apply and ChatGPT existing-page read sites; related work fixed sibling ingest/source-page paths, while unsafe-local prune and bridge-concurrency behavior are adjacent and distinct.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Risk before merge

  • [P1] Persistent non-missing existing-page read failures now abort wiki_apply synthesis creation or ChatGPT import instead of silently continuing as create/update; that protects user-authored content but can stop existing workflows until file access recovers.

Maintainer options:

  1. Accept fail-closed existing-page reads (recommended)
    Land the PR as written if maintainers agree that repeated non-missing read failures should surface an error rather than risk overwriting user-authored wiki content.
  2. Tune the read-error contract first
    If maintainers want a different missing/transient classification or retry policy, update both helpers and regression tests before merge so memory-wiki write paths stay consistent.

Next step before merge

  • No automated repair is needed; the remaining action is maintainer acceptance of the compatibility-sensitive fail-closed read behavior before merge.

Security
Cleared: The diff is limited to memory-wiki read handling and tests, with no dependency, workflow, permission, secret, package-resolution, or new code-execution surface changes.

Review details

Best possible solution:

Land the focused retry/fail-closed fix once maintainers explicitly accept the persistent-read-error behavior for memory-wiki workflows.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main still has catch-to-empty reads at the synthesis and ChatGPT import sites, and a one-shot read fault has a direct source-level path to losing Notes/frontmatter; I did not execute the fault injection because this review is read-only.

Is this the best way to solve the issue?

Yes, with maintainer acceptance. The patch fixes the owner-local read sites directly, follows the merged sibling retry/fail-closed pattern, and avoids a shared abstraction across different I/O error shapes.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against a7a444e7ef1e.

Label changes

Label justifications:

  • P0: The PR fixes a concrete memory-wiki data-loss path where transient existing-page reads can wipe hand-authored Notes blocks and manual frontmatter.
  • merge-risk: 🚨 compatibility: The patch intentionally changes repeated non-missing existing-page read failures from silent create/update behavior to surfaced errors for existing wiki_apply and ChatGPT import workflows.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes copied before/after output from real on-disk memory-wiki vault runs for both affected entry points with one injected transient read failure.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes copied before/after output from real on-disk memory-wiki vault runs for both affected entry points with one injected transient read failure.
Evidence reviewed

PR surface:

Source +40, Tests +206. Total +246 across 3 files.

View PR surface stats
Area Files Added Removed Net
Source 2 44 4 +40
Tests 1 207 1 +206
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 3 251 5 +246

What I checked:

  • Repository policy applied: Root and scoped extension policy require review beyond the diff and treat fail-closed behavior changes as compatibility-sensitive merge risk. (AGENTS.md:31, a7a444e7ef1e)
  • Current main synthesis read can fail open: Current main reads existing synthesis pages and write-change-detection content with catch-to-empty, so a transient read failure can make existing frontmatter/body look absent before rewrite. (extensions/memory-wiki/src/apply.ts:207, a7a444e7ef1e)
  • Current main ChatGPT import read can fail open: Current main reads each existing ChatGPT import destination with fs.readFile(...).catch(() => "") before preserveExistingPageBlocks, which treats an unreadable existing page as new. (extensions/memory-wiki/src/chatgpt-import.ts:740, a7a444e7ef1e)
  • PR head synthesis retry/fail-closed implementation: The PR head adds readExistingWikiPage, retries once, returns empty only when the retry is an FsSafe not-found, and uses it for both synthesis existing-page reads. (extensions/memory-wiki/src/apply.ts:200, ea1263adb831)
  • PR head ChatGPT retry/fail-closed implementation: The PR head adds readExistingConversationPage, retries once, returns empty only for ENOENT after the retry, and uses it before preserving existing page blocks. (extensions/memory-wiki/src/chatgpt-import.ts:143, ea1263adb831)
  • Regression coverage: The PR head tests synthesis notes/frontmatter preservation, path-alias failure preservation, ChatGPT notes preservation, and persistent ChatGPT read failure leaving the page unchanged. (extensions/memory-wiki/src/wiki-notes-read-retry.test.ts:241, ea1263adb831)

Likely related people:

  • vincentkoc: Commit d624ec3 added the wiki apply mutation tool and the synthesis write path changed by this PR; this person also authored the latest PR-head policy-failure adjustment. (role: feature introducer and recent area contributor; confidence: high; commits: d624ec3a0bab, ea1263adb831; files: extensions/memory-wiki/src/apply.ts, extensions/memory-wiki/src/tool.ts)
  • mbelinky: PR [codex] Dreaming: surface memory wiki imports and palace #64505 introduced the ChatGPT import implementation that owns the conversation-page rewrite path changed here. (role: feature introducer; confidence: high; commits: 64693d2e96ab; files: extensions/memory-wiki/src/chatgpt-import.ts, extensions/memory-wiki/src/cli.ts)
  • qingminglong: Merged PR fix(memory-wiki): preserve notes after transient page reads #98360 established the sibling retry/fail-closed pattern for transient existing-page reads in ingest and source-page-shared. (role: recent sibling-fix author; confidence: medium; commits: aa91c9d4a950; files: extensions/memory-wiki/src/ingest.ts, extensions/memory-wiki/src/source-page-shared.ts, extensions/memory-wiki/src/wiki-notes-read-retry.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P0 Emergency: data loss, security bypass, crash loop, or unusable core runtime. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. rating: 🦀 challenger crab Exceptional PR readiness: strong proof, clean patch, and convincing validation. and removed rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. labels Jul 1, 2026
@yetval
yetval force-pushed the fix/memory-wiki-synthesis-transient-read branch from 448269a to 8db6560 Compare July 1, 2026 23:40
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🦀 challenger crab Exceptional PR readiness: strong proof, clean patch, and convincing validation. labels Jul 1, 2026
yetval and others added 2 commits July 1, 2026 18:06
…nd chatgpt import

A create_synthesis re-run and a ChatGPT conversations re-import swallowed
every existing-page read error and treated the page as brand-new, so one
transient read failure silently emptied the user's ## Notes block and
dropped hand-added frontmatter. Route both reads through a retry-once
helper that treats only a missing page as new and propagates persistent
failures, matching the ingest and imported-source fix from openclaw#98360.
@vincentkoc
vincentkoc force-pushed the fix/memory-wiki-synthesis-transient-read branch from 8db6560 to ea1263a Compare July 2, 2026 01:09
@vincentkoc vincentkoc self-assigned this Jul 2, 2026
@vincentkoc

Copy link
Copy Markdown
Member

Maintainer repair and final verification are complete on ea1263adb8315577d809d881f70914f717a2217c.

  • Existing synthesis and ChatGPT-import page reads now retry once before treating a page as missing.
  • Persistent read failures and fs-safe path-alias policy errors abort before rewriting existing content.
  • Focused proof: node scripts/run-vitest.mjs extensions/memory-wiki/src/wiki-notes-read-retry.test.ts extensions/memory-wiki/src/apply.test.ts extensions/memory-wiki/src/import-runs.test.ts (15/15 passed).
  • Direct @openclaw/[email protected] contract inspection confirmed not-found is absence while path-alias is a policy failure.
  • Fresh structured autoreview: clean, no actionable findings, confidence 0.93.
  • git diff --check origin/main...HEAD: passed.
  • Exact-head hosted CI: all required checks passed.
  • Native scripts/pr prepare-run 98787: passed with matching prep and PR heads.

No remaining proof gaps.

@vincentkoc
vincentkoc merged commit db6a3c2 into openclaw:main Jul 2, 2026
96 of 98 checks passed
@vincentkoc

Copy link
Copy Markdown
Member

Merged via squash.

vincentkoc added a commit that referenced this pull request Jul 2, 2026
* origin/main:
  test(plugins): align release fixtures with runtime contracts
  Require explicit iOS release versions
  fix(ci): recover incomplete Swift build caches (#98818)
  fix(memory-wiki): retry transient existing-page reads in wiki_apply and chatgpt import (#98787)
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 2, 2026
…nd chatgpt import (openclaw#98787)

* fix(memory-wiki): retry transient existing-page reads in wiki_apply and chatgpt import

A create_synthesis re-run and a ChatGPT conversations re-import swallowed
every existing-page read error and treated the page as brand-new, so one
transient read failure silently emptied the user's ## Notes block and
dropped hand-added frontmatter. Route both reads through a retry-once
helper that treats only a missing page as new and propagates persistent
failures, matching the ingest and imported-source fix from openclaw#98360.

* fix(memory-wiki): preserve fs-safe policy failures

---------

Co-authored-by: Vincent Koc <[email protected]>
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 3, 2026
…nd chatgpt import (openclaw#98787)

* fix(memory-wiki): retry transient existing-page reads in wiki_apply and chatgpt import

A create_synthesis re-run and a ChatGPT conversations re-import swallowed
every existing-page read error and treated the page as brand-new, so one
transient read failure silently emptied the user's ## Notes block and
dropped hand-added frontmatter. Route both reads through a retry-once
helper that treats only a missing page as new and propagates persistent
failures, matching the ingest and imported-source fix from openclaw#98360.

* fix(memory-wiki): preserve fs-safe policy failures

---------

Co-authored-by: Vincent Koc <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: memory-wiki merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P0 Emergency: data loss, security bypass, crash loop, or unusable core runtime. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: M status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants