Skip to content

memory-wiki: wiki.palace fails with (path-mismatch) and wiki.importInsights takes 100s+ under load #91154

Description

@mlaihk

Summary

Two related defects in the memory-wiki bridge import path:

  1. wiki.palace (and the bridge write inside wiki.importInsights) hard-fails with Refusing to write imported source page (path-mismatch): <path>: path changed during read when the vault file at the target path is replaced between the fs.lstat pre-check and the fs.open/handle.stat/fs.stat re-check in @openclaw/fs-safe's openVerifiedLocalFile. No retry, the write is silently dropped on that call, and the gateway surfaces a misleading error.
  2. wiki.importInsights is consistently slow (30-100+ seconds) whenever any bridge source is dirty, because every imported source triggers three fs-safe stat-and-verify operations and the function does not coalesce concurrent invocations.

When the import takes 100+ seconds the gateway's RSS climbs past the 1.5 GB memory-warning threshold and enters a V8 GC death spiral — observed pushing RSS from 1.9 GB to 2.13 GB and CPU to 40-110% sustained before a manual restart.

Related issues

Steps to reproduce

  1. Install OpenClaw 2026.6.1 with the memory-wiki bundled plugin enabled and bridge mode importing from one or more workspaces.
  2. Have several bridge sources per workspace (in this report: 1914 memory markdown files across 7+ workspaces, producing 1955 vault pages).
  3. Open the OpenClaw web UI (or any client that polls wiki.palace).
  4. Trigger a wiki.importInsights call (the frontend will, on its own).
  5. Modify a memory file in any workspace's memory/ dir to invalidate its import entry, or just wait for a periodic refresh.
  6. Observe: a single wiki.palace call returns a (path-mismatch) error after ~13s; the next wiki.importInsights call succeeds but takes 30-100+ seconds.

Observed timeline (2026-06-07 18:30-18:38 HKT, +08:00)

18:25:02  [diagnostics/memory] memory pressure: warning rss=1.94 GB heap=0.75 GB
18:30:07  [diagnostics/memory] memory pressure: warning rss=1.90 GB heap=0.88 GB
18:30:11  [reload] config hot reload applied (agents.list)         (frontend activity)
18:31:32  [ws] skills.proposals.list 74ms
18:31:34  [ws] node.list 121ms
18:31:36  [ws] sessions.list, chat.history, doctor.memory.*       (frontend poll)
18:31:45  [ws] cron.list 75ms
18:31:49  [ws] ✗ wiki.palace 13187ms
                errorCode=internal_error
                errorMessage=Refusing to write imported source page
                  (path-mismatch): sources/bridge-jordi-6080193a-memory-2d62c497.md:
                  path changed during read
18:33:17  [ws] ✓ wiki.importInsights 101321ms (101.3s)
18:35:13  [diagnostics/memory] memory pressure: warning rss=2.06 GB heap=1.07 GB
18:37:28  [hooks/session-memory] Session context saved to
                ~/.openclaw/agents/jordi/memory/2026-06-07-1837.md
18:38:03  [plugins] active-memory: ollama/kimi-k2.6:cloud timeout 4.3s
18:38     CPU pinned, load avg 1.90 on 10 cores
18:41:34  Gateway restart (manual)
18:41:57  New gateway PID, RSS 537 MB, load avg 0.76, healthy

After the restart, the bridge source page wiki/main/sources/bridge-jordi-6080193a-memory-2d62c497.md exists, mode 0600, last modified 18:32:44 — confirming the next import run did successfully re-write the file the failed palace call was complaining about. The 101s import covered 20+ vault pages written in the 18:30-18:35 window.

Root cause analysis

A) The path-mismatch race in @openclaw/fs-safe's openVerifiedLocalFile

In node_modules/@openclaw/fs-safe/dist/root-impl.js (lines 75-141), the function does:

const preStat = await fs.lstat(filePath);          // [1] stat the path
...
handle = await fs.open(filePath, openFlags);        // [2] open (gets inode X)
const stat = await handle.stat();                   // [3] stat the OPENED handle (inode X)
...
const pathStat = await fs.lstat(filePath);          // [4] stat the path AGAIN
if (!sameFileIdentity(stat, pathStat)) {            // [5] compare X vs pathStat
  throw new FsSafeError("path-mismatch", "path changed during read");
}

The race window is between [1]/[2] and [4]. If the file at filePath is atomically replaced (different inode) in that window — which is what fs.writeFile/rename style atomic writes do — the comparison fails and the write is rejected.

The wrapper at dist/cli-BC1g2VJh.js:3471:

if (error.code !== "symlink" && error.code !== "path-alias")
  throw new Error(`Refusing to write imported source page (${error.code}): ${params.pagePath}: ${error.message}`, { cause: error });

…propagates this to the gateway's WS error. No retry, no in-process recovery, no log of the race cause. A retry on path-mismatch would almost certainly succeed (the file at that path is now the new inode, which is the right one to read).

B) The 100s+ import duration

syncMemoryWikiBridgeSources in dist/cli-BC1g2VJh.js:3611-3676 iterates every public memory artifact in every workspace (listActiveMemoryPublicArtifacts) and calls writeBridgeSourcePage for each. For each artifact it does, in writeImportedSourcePage:

  1. vault.stat(pagePath) (fs-safe: preStat + open + handle.stat + pathStat + realStat = 5 stats)
  2. fs.readFile(sourcePath) (raw fs)
  3. vault.readText(pagePath) (fs-safe: 5 more stats if file exists)
  4. vault.write(pagePath, rendered) (fs-safe: 5 more stats on the open+write path)

That's up to 15 stat-equivalent calls per bridge source. With ~20 newly-dirty sources per import run, this is 300 stats per wiki.importInsights call. Each stat on a system with the workboard plugin holding 46 sqlite handles is contending with sqlite I/O.

The shouldSkipImportedSourceWrite short-circuit (line 3447) helps when the source hasn't changed, but it requires a DB hit per source. There's no per-call coalescing, no per-vault-file mutex, and no batching of consecutive source-page writes.

C) Why it tipped into a memory spiral

The 101s wall-clock + the 46 workboard file handles + the wiki.palace path-mismatch failure all happening in the same minute pushed V8 over its 1.5 GB heap warning threshold. V8 then spent the next 5+ minutes doing GC, RSS climbed 1.94 → 2.13 GB, CPU stayed at 40-110%, and the only recovery was a manual restart. None of the individual observations are fatal; their combination at this scale is.

Suggested fix (sketch)

For (A), the minimal fix in fs-safe:

// In openVerifiedLocalFile, after the path-mismatch check fails:
//   - On path-mismatch, treat the new path's inode as the canonical one
//     and retry the open once. The racing writer has finished; the new
//     file is the one the caller wants.

The cleaner architectural fix is to push the retry into the caller (writeImportedSourcePage and writeBridgeSourcePage): catch (path-mismatch) from vault.write, re-stat the page path once, and retry the write. One retry resolves ~all observed cases because the racing writer has already finished.

For (B), three options, in increasing order of effort:

  1. Coalesce in-flight wiki.importInsights calls. Add a per-process promise cache keyed on {vaultRoot, syncKey}; if a call is in flight, await it instead of starting a new one. The frontend polling cadence (Frontend repeatedly polls wiki.palace causing unknown method WebSocket error logs on backend #65088) means this alone could cut the import volume by 5-10x.
  2. Batch the stat calls in writeImportedSourcePage. Currently we do vault.stat + vault.readText + vault.write — three separate fs-safe round-trips. The existing read is only needed to decide whether to write. If we just always write, we save 5 stats per source.
  3. Pre-filter artifacts whose stats.mtimeMs hasn't changed since the last successful import (we already have state in readMemoryWikiSourceSyncState). The code already does this per-source via shouldSkipImportedSourceWrite, but the check is gated on a DB hit. A cheap in-memory Map<syncKey, {mtimeMs, size}> in the sync state would skip the DB hit.

For the V8 spiral specifically, a process-level RSS limit + auto-recycle (or a hard backpressure on wiki.importInsights when memory pressure is warning) would prevent the 100s import from being the first domino.

Environment

  • OpenClaw: 2026.6.1 (npm global)
  • memory-wiki: bundled plugin, default config
  • Vault: /home/mlaih/.openclaw/wiki/main (~1955 pages)
  • Bridge sources: ~1914 markdown files across 7+ workspaces in ~/.openclaw/agents/<id>/memory/
  • Workboard plugin: enabled (unrelated but compounds: 46 sqlite file handles held)
  • Frontend: OpenClaw web UI open during the incident (drives the wiki.palace poll)
  • Obsidian CLI: not configured for this vault
  • OS: Ubuntu 24.04 in WSL2, kernel 6.6.114, 10 cores, 32 GB RAM

Sample error payload (from gateway journal)

[ws] ⇄ res ✗ wiki.palace 13187ms
  errorCode=internal_error
  errorMessage=Refusing to write imported source page (path-mismatch):
    sources/bridge-jordi-6080193a-memory-2d62c497.md: path changed during read
    | path changed during read

The doubled path changed during read is the wrapper message + cause.message from the FsSafeError. Useful for grep in existing log captures.

Happy to send a PR

If maintainers confirm the preferred direction (caller-side retry on path-mismatch vs. fs-safe-level retry, and coalesce-only vs. coalesce+batch+prefilter for the import speed), I'm happy to send a PR with the chosen scope.

Metadata

Metadata

Assignees

Labels

P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

Type

No type

Fields

Priority

None yet

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions