memory: gateway never self-heals a missing index-identity, and the only repair (out-of-process CLI reindex) orphans the running gateway's DB handle

Summary

A long-running gateway whose memory index-identity becomes invalid with chunks already indexed never recovers on its own — memory_search returns { disabled: true, unavailable: true, error: "index metadata is missing" } indefinitely. The only repair path is an out-of-process openclaw memory index --force / openclaw memory status --index, but that performs an atomic file swap that leaves the running gateway reading a deleted inode, so the CLI "fix" never reaches the live process. Net effect: memory search (and any plugin supplement that rides on it) stays dead until the gateway is restarted, and re-running the CLI just re-orphans it.

Observed on [email protected] (Linux, embeddings = ollama / bge-m3). Source refs below are against the current tree.

Impact

• memory_search returns no results and reports disabled/unavailable on every call.
• Because the disabled branch returns before corpus supplements are queried, bundled/registered supplements (e.g. compiled-wiki) are never searched either — so the failure looks like "wiki injection broke" when the real cause is the core index.
• Recovery requires a manual gateway restart. Running the documented openclaw memory status --index / index --force against a live gateway does not fix the running process (and silently makes things worse — see Defect B).

Two defects that combine into a trap

Defect A — gateway cannot self-heal an invalid index-identity

extensions/memory-core/src/memory/manager-sync-ops.ts:1785

const needsInitialIndex = indexIdentity.status !== "valid" && !hasIndexedChunks;
const needsExplicitIdentityReindex =
  params?.reason === "cli" && indexIdentity.status !== "valid" && !hasTargetSessionFiles;
const needsFullReindex =
  (params?.force && !hasTargetSessionFiles) ||
  needsInitialIndex ||
  needsExplicitIdentityReindex;
if (indexIdentity.status !== "valid" && !needsFullReindex) {
  this.dirty = true;
  // ...marks dirty and returns without rebuilding
  return;
}

When the identity is invalid (resolveMemoryIndexIdentityState → status: "missing", e.g. the memory_index_meta_v1 row is absent after a meta-schema bump) but chunks exist, a full reindex only runs for params.force or params.reason === "cli". An ordinary in-gateway sync (watcher / interval / search-bootstrap / session-start) hits the return and never rebuilds. search() then gates on a valid identity and returns [] (the disabled result). So the gateway is permanently stuck: the only thing that writes a fresh meta row is a full reindex, and the gateway won't run one on its own.

Defect B — atomic reindex orphans a concurrent reader; the existing reopen safety net doesn't cover it

The repair the operator is told to run (openclaw memory status --index / index --force) is a separate process. Its reindex swaps the DB file atomically:

extensions/memory-core/src/memory/manager-atomic-reindex.ts:115

const backupPath = `${targetPath}.backup-${randomUUID()}`;
await moveMemoryIndexFiles(targetPath, backupPath);   // rename main.sqlite -> backup
// ...move temp -> target...
await removeMemoryIndexFiles(backupPath);             // unlink backup

The long-running gateway still holds an open file descriptor on the original inode, which is now the renamed-then-unlinked *.backup-<uuid>. On Linux that deleted inode stays fully readable and writable, so the gateway keeps serving stale data and never notices the swap. readMeta() re-reads per search, but from the wrong (deleted) inode.

The manager already has a reopen seam, but it only triggers on a readonly write error:

extensions/memory-core/src/memory/manager-sync-control.ts:36

/attempt to write a readonly database|database is read-only|SQLITE_READONLY/i

extensions/memory-core/src/memory/manager-sync-control.ts:106 → state.db = state.openDatabase();

A silent inode swap throws no such error, so the readonly-recovery path never fires and the connection is never reopened.

How they combine

1. Index-identity goes invalid in the running gateway (meta-schema bump on update, or the meta row otherwise lost) → Defect A: gateway can't self-heal, search is dead.
2. Operator follows the in-tool guidance and runs openclaw memory status --index / index --force → Defect B: the CLI rebuilds the on-disk DB (a fresh main.sqlite) but atomically swaps it under the running gateway, which keeps reading the deleted backup inode.
3. The CLI process exits reporting success; a fresh openclaw memory search works. The running gateway still returns index metadata is missing.
4. Re-running the CLI just re-orphans it. Only a gateway restart reopens the live main.sqlite.

Reproduction

Defect A (self-heal gap):

1. Start the gateway with memory search enabled (any embedding provider).
2. Make the on-disk identity invalid while the gateway runs — e.g. simulate a meta-schema loss:
   sqlite3 ~/.openclaw/memory/main.sqlite "DELETE FROM meta WHERE key='memory_index_meta_v1';"
3. Call memory_search (any query). It returns { disabled: true, unavailable: true, error: "index metadata is missing" } and keeps doing so — the gateway never rebuilds, despite chunks being present.

Defect B (orphaned handle):

1. With the gateway running and healthy, run openclaw memory index --force (or memory status --index) as a separate process.
2. lsof -p "$(systemctl --user show -p MainPID --value openclaw-gateway)" | grep main.sqlite
   shows the gateway holding …/memory/main.sqlite.backup-<uuid> (deleted) while the live main.sqlite is a different inode.
3. A fresh openclaw memory search … returns hits, but memory_search inside the running gateway still reports index metadata is missing until restart.

Observed lsof (sanitized) from a 2026.6.1 box that had been broken for ~2 days:

node … 34ur REG … (deleted) /home/<user>/.openclaw/memory/main.sqlite.backup-6e607510-… (inode 8934582)
# live on disk: /home/<user>/.openclaw/memory/main.sqlite (inode 8936913)

Suggested direction (details + draft in linked PR)

• Self-heal (Defect A): let an in-gateway sync rebuild a chunk-backed index when the identity is "missing" (lost/corrupt metadata), not only for reason === "cli". This reuses runSafeReindex, which closes and reopens the DB handle via openDatabase() — so it also recovers the orphaned-handle case whenever the stale inode reports "missing". Whether "mismatched" (a deliberate provider/model change) should also auto-rebuild or stay operator-gated is a maintainer call.
• Reopen-on-swap (Defect B): detect that the underlying DB file was replaced (inode changed) and reopen through the existing openDatabase() seam at a sync boundary (not on the per-search hot path, per the no-freshness-polling rule), so a stale-but-"valid" swapped-in identity is also recovered without restart. This touches the multi-process file-swap contract and likely warrants its own change + Crabbox proof.

Happy to scope a PR to Defect A first (smallest, clearly correct) and track Defect B as follow-up.