Skip to content

fix(memory-core): keep durable index identity visible during safe reindex#90422

Closed
Alix-007 wants to merge 2 commits into
openclaw:mainfrom
Alix-007:alix/fix-90361
Closed

fix(memory-core): keep durable index identity visible during safe reindex#90422
Alix-007 wants to merge 2 commits into
openclaw:mainfrom
Alix-007:alix/fix-90361

Conversation

@Alix-007

@Alix-007 Alix-007 commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #90361.

memory_search intermittently returned disabled/unavailable with index metadata is missing during parallel or closely overlapping calls, even though the durable builtin index was valid (CLI deep status, direct DB inspection, and fresh-manager checks all showed valid metadata). The cause is a race: MemoryIndexManager.search() can kick off an async sync, and runSafeReindex() briefly swaps the live this.db to a fresh temp db (resetting vector/FTS state before metadata is written). A concurrent status() / tool-wrapper read during that window refreshes index identity from the temp db, finds no metadata, and fails closed with index metadata is missing — while the real index is fine.

What changed

extensions/memory-core/src/memory/manager-sync-ops.ts — add a reindexInProgress flag, set for the duration of runSafeReindex's atomic temp-db swap and always cleared in a finally.

extensions/memory-core/src/memory/manager.ts — in refreshIndexIdentityDirty, while reindexInProgress is set, a transient missing no longer overwrites a previously valid identity; the last stable identity is returned instead. Real mismatches — and missing outside a rebuild — still propagate, so genuine provider/model/settings drift keeps failing closed.

extensions/memory-core/src/memory/index.test.ts — deterministic regression test: it drives a real safe (atomic) reindex and reads status() from inside the reindex build, after the temp db has been swapped in, asserting the durable identity stays valid.

Real behavior proof

Behavior addressed: A status() read concurrent with a safe reindex's temp-db swap keeps returning the durable valid identity instead of a transient missing, so memory_search no longer fails closed mid-rebuild. Real provider/model/settings mismatches and durable-missing states still fail closed.

Real environment tested: Local worktree of origin/main at base 118060157d, Node v22.22.0. Drove the real MemoryIndexManager over a real on-disk SQLite index via node scripts/run-vitest.mjs (the actual production reindex + status paths, no manager mocks). The concurrent status() is read from inside the real reindex build, while the real temp db is swapped in.

Exact steps or command run after this patch:

# Concurrent status() during a real safe (atomic) reindex:
node scripts/run-vitest.mjs run extensions/memory-core/src/memory/index.test.ts -t "concurrent status"
# Baseline measured by git-stashing the manager.ts guard and re-running the same test.

Evidence after fix: With the guard reverted (origin manager.ts), the same test fails — the concurrent read sees the temp db, reproducing the issue:

AssertionError: expected { status: 'missing', … } to deeply equal { status: 'valid' }
+   "reason": "index metadata is missing",
+   "status": "missing",

With the fix applied:

Test Files  1 passed (1)
     Tests  1 passed | 34 skipped (35)

Observed result after fix: During a real safe reindex (temp db swapped in), status().custom.indexIdentity stays { status: "valid" } instead of { status: "missing", reason: "index metadata is missing" }. The transient missing reported by the issue is gone; durable-missing and real mismatch still surface.

What was not tested: No live OpenClaw v2026.6.1 end-to-end reproduction across multiple OS/providers; the change is in the deterministic manager identity-refresh path, exercised directly against the real manager and SQLite index. The fix is scoped to the manager identity layer (not a tool-wrapper-only mitigation), preserving real mismatch fail-closed as requested in the issue review.

Validation (supplemental)

  • index.test.ts + tools.test.ts + manager-reindex-state.test.ts → 59 passed; manager.async-search.test.ts → 3 passed
  • tsgo:extensions:test → no new errors in changed files; oxlint → clean

…ndex

memory_search intermittently returned disabled/unavailable with 'index
metadata is missing' during overlapping calls while the durable index was
valid (openclaw#90361). runSafeReindex briefly swaps the live db to a fresh temp db,
and a concurrent status()/tool read refreshed index identity from that temp
db and failed closed. Track a reindexInProgress flag across the atomic swap
and, while it is set, keep returning the last valid identity instead of a
transient missing. Real mismatches and missing outside a rebuild still
propagate, preserving genuine fail-closed behavior.
@openclaw-barnacle openclaw-barnacle Bot added extensions: memory-core Extension: memory-core size: S proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 4, 2026
@clawsweeper

clawsweeper Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Close as superseded: the useful status-identity guard is covered by the broader open fix at #90453, which also protects search readers and active DB reads during safe reindex.

Canonical path: Close this narrower branch and use #90453 as the canonical landing path for the safe-reindex race.

So I’m closing this here and keeping the remaining discussion on #90453.

Review details

Best possible solution:

Close this narrower branch and use #90453 as the canonical landing path for the safe-reindex race.

Do we have a high-confidence way to reproduce the issue?

Yes, source inspection shows the race window: safe reindex swaps this.db to a temp DB while status/search can read identity and index state. I did not run the regression locally because this review was read-only.

Is this the best way to solve the issue?

No, this is not the best fix because it only preserves the identity state and does not wait search/vector/FTS readers through the temp-DB window. The broader open PR is the safer owner-boundary fix.

Security review:

Security review cleared: The diff only changes memory-core TypeScript and tests; no CI, dependency, secret, install, or package-publishing surface was touched.

AGENTS.md: found and applied where relevant.

What I checked:

Likely related people:

  • steipete: Central memory manager, status, and safe-reindex lines are attributed to Peter Steinberger in blame, and shortlog shows the highest contribution count across the affected memory-core files. (role: recent area contributor; confidence: medium; commits: 82710b4f1f10, f4d8393bf4c1, cad83db8b2f7; files: extensions/memory-core/src/memory/manager.ts, extensions/memory-core/src/memory/manager-sync-ops.ts, extensions/memory-core/src/tools.ts)
  • vincentkoc: Recent history shows multiple memory-core fixes and test changes by Vincent Koc across the same manager/tool surface. (role: adjacent area contributor; confidence: medium; commits: 37f449d7e189, ca26489fe882, 7e3b48c2546f; files: extensions/memory-core/src/memory/manager.ts, extensions/memory-core/src/memory/manager-sync-ops.ts, extensions/memory-core/src/memory/index.test.ts)

Codex review notes: model gpt-5.5, reasoning high; reviewed against 1a3ce7c2a8da.

@clawsweeper clawsweeper Bot added the rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. label Jun 4, 2026
@myles-mworks

Copy link
Copy Markdown

@clawsweeper re-review

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. and removed rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. labels Jun 5, 2026
@clawsweeper

clawsweeper Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper applied the proposed close for this PR.

  • Action: closed this PR.
  • Close reason: duplicate or superseded.
  • Evidence: durable ClawSweeper review.
  • Coverage proof: PR B clearly carries PR A's useful status-identity intent and also covers the material missing behavior identified in PR A's report: guarding search/readers during the same safe-reindex race. Any remaining PR A details are incidental or reviewable on PR B. Covering PR: fix(memory-core): guard searches during safe reindex #90453.

@clawsweeper clawsweeper Bot closed this Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: memory-core Extension: memory-core merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]:Intermittent memory_search "index metadata is missing" despite valid builtin memory index; likely search/reindex race. Locally hotfixed.

2 participants