Skip to content

Refresh memory index state after external reindex#92943

Closed
mushuiyu886 wants to merge 5 commits into
openclaw:mainfrom
mushuiyu886:feat/issue-91167-90508
Closed

Refresh memory index state after external reindex#92943
mushuiyu886 wants to merge 5 commits into
openclaw:mainfrom
mushuiyu886:feat/issue-91167-90508

Conversation

@mushuiyu886

@mushuiyu886 mushuiyu886 commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Problem: memory gateway/default managers could keep using an old sqlite handle after a CLI/status atomic reindex replaced the index file, so missing identity or dirty state could remain stuck until process restart. Search-first recovery could read the new DB but still keep stale dirty flags.
  • Solution: cached managers now detect when the on-disk sqlite file has been atomically replaced, reopen the DB handle, refresh schema/meta/vector state, and carry the replacement-refresh signal into identity dirty cleanup even when search() is the first observer.
  • ClawSweeper follow-up: replacement refresh now clears identity-only stale dirty state without clearing real pending memory/session work signaled by memory full-retry, pending watch paths, session full-retry, or session dirty files.
  • Scope note: stale sqlite temp cleanup is intentionally left to fix(memory): clean stale reindex temp files #92891's guarded pid-lock cleanup path; this PR only handles cached-manager refresh after atomic replacement.
  • What did NOT change: semantic indexes are still protected when the configured embedding provider is unavailable; this does not force a semantic rebuild, change provider identity rules, or delete temp sqlite files.
  • Why it matters / User impact: a gateway should recover from memory index repair/rebuild work performed by CLI/status flows without requiring a restart.
  • Fixes bug(memory): gateway cannot self-heal a missing index identity when chunks are already indexed #91167
  • Refs memory-core main reindex thrashes, leaks main.sqlite.tmp files, and leaves memory_search paused after repair #90508

Real behavior proof

  • Behavior or issue addressed: This patch covers local memory-index lifecycle failures: a cached live/default manager continuing to report missing identity after a CLI-style manager atomically replaces the same sqlite index with a valid FTS-only index; the search-first path where manager.search() is the first call after replacement; and the patch-quality edge where replacement refresh must clear identity-only dirty state without dropping unrelated pending memory/session dirty work.
  • Real environment tested: Local Linux worktree /media/vdc/code/ai/aispace/openclaw-worktrees/issue-91167-90508, branch feat/issue-91167-90508.
  • Exact steps or command run after this patch:
cd /media/vdc/code/ai/aispace/openclaw-worktrees/issue-91167-90508 && OPENCLAW_REPO=/media/vdc/code/ai/aispace/openclaw-worktrees/issue-91167-90508 node --import tsx /media/vdc/code/ai/aispace/openclaw-issue-91167-90508-evidence/memory-external-reindex-real-proof.mjs > /media/vdc/code/ai/aispace/openclaw-issue-91167-90508-evidence/memory-external-reindex-real-proof.out.json
cd /media/vdc/code/ai/aispace/openclaw-worktrees/issue-91167-90508 && OPENCLAW_TEST_FAST=1 node scripts/run-vitest.mjs extensions/memory-core/src/memory/manager.self-heal-missing-identity.test.ts -t "clears stale dirty state when search first observes a cli atomic reindex replacement"
cd /media/vdc/code/ai/aispace/openclaw-worktrees/issue-91167-90508 && OPENCLAW_TEST_FAST=1 node scripts/run-vitest.mjs extensions/memory-core/src/memory/manager.self-heal-missing-identity.test.ts -t "preserves pending memory and session dirty work"
cd /media/vdc/code/ai/aispace/openclaw-worktrees/issue-91167-90508 && OPENCLAW_TEST_FAST=1 node scripts/run-vitest.mjs extensions/memory-core/src/memory/manager.self-heal-missing-identity.test.ts
cd /media/vdc/code/ai/aispace/openclaw-worktrees/issue-91167-90508 && ./node_modules/.bin/oxfmt --check --threads=1 extensions/memory-core/src/memory/manager.ts extensions/memory-core/src/memory/manager.self-heal-missing-identity.test.ts
cd /media/vdc/code/ai/aispace/openclaw-worktrees/issue-91167-90508 && node scripts/run-tsgo.mjs -p tsconfig.extensions.json --noEmit
git -C /media/vdc/code/ai/aispace/openclaw-worktrees/issue-91167-90508 diff --check
  • Evidence after fix: The independent Node proof below runs outside Vitest, imports the production memory manager, uses real SQLite files, warms a live/default manager, performs an external CLI-purpose force sync, and then verifies the same live manager object recovers without restart.
{
  "proof": "memory-external-cli-reindex-live-manager-refresh",
  "head": "f007c10b7cc47bda43951ee5f5fedac75eed590d",
  "runtime": "node --import tsx; production memory manager; real sqlite files; no Vitest",
  "scenarios": [
    {
      "scenario": "status-first",
      "firstObserver": "status",
      "sameLiveManagerObject": true,
      "before": { "identity": "missing", "dirty": true, "chunks": 1 },
      "after": { "identity": "valid", "dirty": false, "chunks": 1 },
      "recoveredWithoutRestart": true
    },
    {
      "scenario": "search-first",
      "firstObserver": "search",
      "sameLiveManagerObject": true,
      "before": { "identity": "missing", "dirty": true, "chunks": 1 },
      "after": { "identity": "valid", "dirty": false, "chunks": 1, "searchHits": 1 },
      "recoveredWithoutRestart": true
    }
  ],
  "allRecoveredWithoutRestart": true
}
Test Files  1 passed (1)
Tests  1 passed | 3 skipped (4)
Test Files  1 passed (1)
Tests  1 passed | 4 skipped (5)
Test Files  1 passed (1)
Tests  5 passed (5)
[test] passed 1 Vitest shard in 17.71s
All matched files use the correct format.
Finished in 22ms on 2 files using 1 threads.
node scripts/run-tsgo.mjs -p tsconfig.extensions.json --noEmit exited 0 with no stdout/stderr output.
git diff --check exited 0 with no whitespace errors.
  • Observed result after fix: The cached manager no longer stays stuck on the old missing-identity sqlite handle after a CLI atomic reindex publishes a valid replacement. The real runtime proof covers both first observers with no unrelated pending work: status() refreshes to indexIdentity: valid and dirty: false, and search("Alpha topic") returns one hit before the subsequent status reports valid/dirty=false. The focused pending-dirty regression then seeds memory full-retry state plus session dirty files/full-retry state before the replacement; after refresh, identity becomes valid while dirty, memoryFullRetryDirty, sessionsDirty, sessionsDirtyFiles, and sessionsFullRetryDirty remain set. In both runtime-proof scenarios the same warmed live manager object is reused, so recovery happens without process restart. Stale temp cleanup is left to fix(memory): clean stale reindex temp files #92891's guarded cleanup path rather than duplicated here.
  • What was not tested: No remote hosted gateway, desktop app, QMD daemon, or separate OS process gateway was run. The added proof uses a long-lived in-process default manager plus a separate CLI-purpose manager against real SQLite files to exercise the same cached manager / external atomic replacement boundary without mocking the memory manager.

Regression Test Plan

  • Target test file: extensions/memory-core/src/memory/manager.self-heal-missing-identity.test.ts.
  • Scenario locked in: A default manager with missing identity refreshes to valid after a CLI manager atomically replaces the sqlite file; the same replacement also clears identity-only stale dirty state when search() is the first post-replacement call; pending memory and session dirty work survives the replacement refresh.
  • Why this is the smallest reliable guardrail: The tests use real sqlite files, real manager cache semantics, real search, and the existing atomic reindex path rather than mocking the state transition.

Merge risk

Root Cause

  • Root cause: Safe reindex publishes by replacing the sqlite file, but an already-cached manager keeps its existing sqlite handle and in-memory identity/dirty flags. search() could refresh the handle before refreshIndexIdentityDirty() ran, consuming the one-shot replacement signal and leaving stale dirty/retry state in memory. The first cleanup pass also cleared all dirty/session retry state after a valid replacement, which fixed identity-only stale dirty flags but could drop unrelated pending memory/session work.
  • Why this is root-cause fix: The cached manager now treats the on-disk sqlite file identity as the source-of-truth for whether its handle is stale, reopens when that identity changes, and carries that replacement signal until identity cleanup recomputes state from the replacement DB. Cleanup now recomputes only identity-derived stale dirtiness while preserving independently queued memory/session retry work.
  • Fix classification: Root-cause reliability fix for memory index lifecycle refresh after external atomic replacement.
  • Maintainer-ready confidence: High for the local sqlite/index lifecycle paths covered here. The patch is limited to manager open/refresh behavior and deterministic memory-core tests.
  • Patch quality notes: The diff adds a dev/ino based DB replacement check, a pending dirty-cleanup marker, and regression tests for stale identity cleanup plus pending memory/session dirty preservation. It does not add sleeps, broaden provider compatibility, duplicate fix(memory): clean stale reindex temp files #92891's guarded temp cleanup, or relax semantic-index protection.
  • Architecture / source-of-truth check: The sqlite index file identity and memory_index_meta_v1 remain the source of truth; the live manager now refreshes its in-memory view from that source instead of trusting stale process-local state.

Reopen cached memory index handles when another manager atomically publishes a replacement sqlite file, and clean stale temp index files on open.
@openclaw-barnacle openclaw-barnacle Bot added extensions: memory-core Extension: memory-core size: S proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 14, 2026
@clawsweeper

clawsweeper Bot commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 14, 2026, 12:20 PM ET / 16:20 UTC.

Summary
The PR adds SQLite file-signature detection to the builtin memory manager so cached managers reopen after external atomic reindex and preserve unrelated dirty memory/session work.

PR surface: Source +65, Tests +103. Total +168 across 2 files.

Reproducibility: yes. source-reproducible: current main has cached default managers, transient CLI/status managers, and atomic SQLite replacement without file-signature refresh. Linked reports add Linux and macOS live evidence, though I did not run a live gateway in this read-only review.

Review metrics: 1 noteworthy metric.

  • Hot-Path Refresh Observers: 3 observer paths affected. Search, sync, and status-observed identity refresh can now reopen the SQLite handle when file identity changes, which is the main merge-risk surface.

Stored data model
Persistent data-model change detected: migration/backfill/repair: extensions/memory-core/src/memory/manager.self-heal-missing-identity.test.ts, unknown-data-model-change: extensions/memory-core/src/memory/manager.self-heal-missing-identity.test.ts, unknown-data-model-change: extensions/memory-core/src/memory/manager.ts, vector/embedding metadata: extensions/memory-core/src/memory/manager.self-heal-missing-identity.test.ts, vector/embedding metadata: extensions/memory-core/src/memory/manager.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Maintainers should decide whether the local real-SQLite proof is enough or require a live gateway proof before merge.

Risk before merge

  • [P1] Merging changes cached memory/session state refresh after external SQLite replacement, so maintainers should explicitly accept the search/sync/status file-signature freshness exception.
  • [P1] The strongest proof is local real-SQLite runtime proof with a same-process default manager plus CLI-purpose manager; it does not include a remote hosted gateway or long-running desktop/gateway proof.

Maintainer options:

  1. Accept The Bounded Refresh Path (recommended)
    Merge after maintainer acceptance that the targeted file-signature check is the right exception for recovering cached memory managers after external reindex.
  2. Request Live Gateway Proof
    Ask for a live gateway-style proof showing memory_search recovery without restart after CLI reindex if local SQLite proof is not enough for the session-state risk.
  3. Pause For Broader Lifecycle Coordination
    Pause this PR only if maintainers want to coordinate it with the separate stale-temp cleanup work before changing memory index lifecycle behavior.

Next step before merge

  • No automated repair is indicated; the next action is maintainer review of the session-state hot-path risk and proof boundary.

Security
Cleared: No concrete security or supply-chain concern found; the diff is confined to local memory-core SQLite lifecycle code and tests.

Review details

Best possible solution:

Land the manager-local file-signature refresh with dirty-state preservation once maintainers accept the session-state hot-path risk; keep stale temp cleanup separate in #92891.

Do we have a high-confidence way to reproduce the issue?

Yes, source-reproducible: current main has cached default managers, transient CLI/status managers, and atomic SQLite replacement without file-signature refresh. Linked reports add Linux and macOS live evidence, though I did not run a live gateway in this read-only review.

Is this the best way to solve the issue?

Yes, this is the best bounded fix I found: the manager-local reopen path covers external atomic replacement without adding config or duplicating temp cleanup. The safer alternative, broad cache invalidation from CLI/status, would miss other replacement producers and create more owner coupling.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against c67dc59b02b0.

Label changes

Label justifications:

  • P1: The PR targets a shipped memory_search recovery failure that can keep long-running gateways stuck until restart.
  • merge-risk: 🚨 session-state: Merging changes how cached memory/session index state is refreshed and cleaned after external SQLite replacement.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body now includes current-head terminal/log proof for real SQLite files, production memory manager use, status-first and search-first refresh, and dirty-state preservation.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body now includes current-head terminal/log proof for real SQLite files, production memory manager use, status-first and search-first refresh, and dirty-state preservation.
Evidence reviewed

PR surface:

Source +65, Tests +103. Total +168 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 65 0 +65
Tests 1 108 5 +103
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 173 5 +168

What I checked:

Likely related people:

  • xydt-tanshanshan: Authored merged work for missing-identity self-heal and atomic SQLite swap preservation in the same memory-core manager/reindex paths. (role: recent area contributor; confidence: high; commits: 1a2eb74b9df6, 865fdab07501; files: extensions/memory-core/src/memory/manager-sync-ops.ts, extensions/memory-core/src/memory/manager-atomic-reindex.ts, extensions/memory-core/src/memory/manager-db.ts)
  • mushuiyu886: Shows prior merged memory-core indexing history through fix #88009: [Feature]: batched memory embedding should batch over files #89138, beyond being the current PR author. (role: adjacent memory-core contributor; confidence: medium; commits: a36e05050a9d, d29a921a7a6c, 4ad0113b6144; files: extensions/memory-core/src/memory/manager-sync-ops.ts, extensions/memory-core/src/memory/manager-embedding-ops.ts, extensions/memory-core/src/memory/index.test.ts)
  • shakkernerd: Carried follow-up fixes in the merged source-wide memory batching PR that touched memory sync and session cleanup behavior adjacent to this change. (role: recent area contributor; confidence: medium; commits: 5f6fc0280d77, a021efb7deb3, 9a39ab9b5d65; files: extensions/memory-core/src/memory/manager-sync-ops.ts, extensions/openai/embedding-batch.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jun 14, 2026
Keep this PR focused on cached manager refresh after external atomic reindex replacement, while leaving stale temp sqlite cleanup to the guarded cleanup path tracked separately.
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. proof: sufficient ClawSweeper judged the real behavior proof convincing. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 14, 2026
Keep real pending memory and session dirty work when an external index replacement clears stale identity state, while still cleaning identity-only dirty flags.
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 14, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jun 14, 2026
@clawsweeper clawsweeper Bot added status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 14, 2026
@mushuiyu886

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 14, 2026
@vincentkoc vincentkoc self-assigned this Jun 14, 2026

@vincentkoc vincentkoc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep review found two merge blockers in extensions/memory-core/src/memory/manager.ts:

  • The refresh cleanup can clear dirty while broad watcher dirtiness is pending with no path, causing changes to remain stale indefinitely when interval sync is disabled.
  • Closing the shared SQLite handle while searches/vector probes are active invalidates their statements; a resumed search can silently degrade to empty vector results.

Best fix: preserve broad watcher dirtiness separately and make database-handle replacement lifecycle-owned, keeping the old handle alive until active readers drain.

@vincentkoc

Copy link
Copy Markdown
Member

Closing this rather than landing the proposed cached-manager freshness path. The stale SQLite-handle report in #91167/#90508 is valid, and the contributor proof is useful, but polling filesystem identity from search/status/sync observers conflicts with the memory hot-path ownership rule and makes each observer a second lifecycle authority.

I tried the maintainer repair path in an isolated worktree. Making publication, reader draining, recovery, watcher generations, and cleanup coherent expanded the change from the PR's +173/-5 across two files to roughly +1.5k/-311 across nine files. Fresh autoreview still found a temp-index cleanup failure that the new path hid; fixing that does not change the underlying blast-radius problem.

The next fix should use one explicit manager-owned publication/reload contract, with live gateway proof, rather than per-observer file freshness polling or a contributor PR that silently becomes a broad SQLite lifecycle rewrite. #91167 and #90508 should remain open for that canonical fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: memory-core Extension: memory-core merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(memory): gateway cannot self-heal a missing index identity when chunks are already indexed

2 participants