Refresh memory index state after external reindex#92943
Conversation
Reopen cached memory index handles when another manager atomically publishes a replacement sqlite file, and clean stale temp index files on open.
|
Codex review: needs maintainer review before merge. Reviewed June 14, 2026, 12:20 PM ET / 16:20 UTC. Summary PR surface: Source +65, Tests +103. Total +168 across 2 files. Reproducibility: yes. source-reproducible: current main has cached default managers, transient CLI/status managers, and atomic SQLite replacement without file-signature refresh. Linked reports add Linux and macOS live evidence, though I did not run a live gateway in this read-only review. Review metrics: 1 noteworthy metric.
Stored data model Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Land the manager-local file-signature refresh with dirty-state preservation once maintainers accept the session-state hot-path risk; keep stale temp cleanup separate in #92891. Do we have a high-confidence way to reproduce the issue? Yes, source-reproducible: current main has cached default managers, transient CLI/status managers, and atomic SQLite replacement without file-signature refresh. Linked reports add Linux and macOS live evidence, though I did not run a live gateway in this read-only review. Is this the best way to solve the issue? Yes, this is the best bounded fix I found: the manager-local reopen path covers external atomic replacement without adding config or duplicating temp cleanup. The safer alternative, broad cache invalidation from CLI/status, would miss other replacement producers and create more owner coupling. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against c67dc59b02b0. Label changesLabel justifications:
Evidence reviewedPR surface: Source +65, Tests +103. Total +168 across 2 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
Keep this PR focused on cached manager refresh after external atomic reindex replacement, while leaving stale temp sqlite cleanup to the guarded cleanup path tracked separately.
Keep real pending memory and session dirty work when an external index replacement clears stale identity state, while still cleaning identity-only dirty flags.
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
vincentkoc
left a comment
There was a problem hiding this comment.
Deep review found two merge blockers in extensions/memory-core/src/memory/manager.ts:
- The refresh cleanup can clear
dirtywhile broad watcher dirtiness is pending with no path, causing changes to remain stale indefinitely when interval sync is disabled. - Closing the shared SQLite handle while searches/vector probes are active invalidates their statements; a resumed search can silently degrade to empty vector results.
Best fix: preserve broad watcher dirtiness separately and make database-handle replacement lifecycle-owned, keeping the old handle alive until active readers drain.
|
Closing this rather than landing the proposed cached-manager freshness path. The stale SQLite-handle report in #91167/#90508 is valid, and the contributor proof is useful, but polling filesystem identity from search/status/sync observers conflicts with the memory hot-path ownership rule and makes each observer a second lifecycle authority. I tried the maintainer repair path in an isolated worktree. Making publication, reader draining, recovery, watcher generations, and cleanup coherent expanded the change from the PR's +173/-5 across two files to roughly +1.5k/-311 across nine files. Fresh autoreview still found a temp-index cleanup failure that the new path hid; fixing that does not change the underlying blast-radius problem. The next fix should use one explicit manager-owned publication/reload contract, with live gateway proof, rather than per-observer file freshness polling or a contributor PR that silently becomes a broad SQLite lifecycle rewrite. #91167 and #90508 should remain open for that canonical fix. |
Summary
search()is the first observer.Real behavior proof
manager.search()is the first call after replacement; and the patch-quality edge where replacement refresh must clear identity-only dirty state without dropping unrelated pending memory/session dirty work./media/vdc/code/ai/aispace/openclaw-worktrees/issue-91167-90508, branchfeat/issue-91167-90508.{ "proof": "memory-external-cli-reindex-live-manager-refresh", "head": "f007c10b7cc47bda43951ee5f5fedac75eed590d", "runtime": "node --import tsx; production memory manager; real sqlite files; no Vitest", "scenarios": [ { "scenario": "status-first", "firstObserver": "status", "sameLiveManagerObject": true, "before": { "identity": "missing", "dirty": true, "chunks": 1 }, "after": { "identity": "valid", "dirty": false, "chunks": 1 }, "recoveredWithoutRestart": true }, { "scenario": "search-first", "firstObserver": "search", "sameLiveManagerObject": true, "before": { "identity": "missing", "dirty": true, "chunks": 1 }, "after": { "identity": "valid", "dirty": false, "chunks": 1, "searchHits": 1 }, "recoveredWithoutRestart": true } ], "allRecoveredWithoutRestart": true }status()refreshes toindexIdentity: validanddirty: false, andsearch("Alpha topic")returns one hit before the subsequent status reportsvalid/dirty=false. The focused pending-dirty regression then seeds memory full-retry state plus session dirty files/full-retry state before the replacement; after refresh, identity becomes valid whiledirty,memoryFullRetryDirty,sessionsDirty,sessionsDirtyFiles, andsessionsFullRetryDirtyremain set. In both runtime-proof scenarios the same warmed live manager object is reused, so recovery happens without process restart. Stale temp cleanup is left to fix(memory): clean stale reindex temp files #92891's guarded cleanup path rather than duplicated here.Regression Test Plan
extensions/memory-core/src/memory/manager.self-heal-missing-identity.test.ts.search()is the first post-replacement call; pending memory and session dirty work survives the replacement refresh.Merge risk
Root Cause
search()could refresh the handle beforerefreshIndexIdentityDirty()ran, consuming the one-shot replacement signal and leaving stale dirty/retry state in memory. The first cleanup pass also cleared all dirty/session retry state after a valid replacement, which fixed identity-only stale dirty flags but could drop unrelated pending memory/session work.memory_index_meta_v1remain the source of truth; the live manager now refreshes its in-memory view from that source instead of trusting stale process-local state.