fix(memory-core): checkpoint WAL after publishing index metadata#92509
fix(memory-core): checkpoint WAL after publishing index metadata#92509Dazlarus wants to merge 7 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Hardening the memory-core SQLite index lifecycle to prevent “index metadata is missing” after interrupted reindex/sync by forcing WAL checkpointing after meta writes and cleaning up leftover swap artifacts.
Changes:
- Add best-effort cleanup of stale
.backup-*/.tmp-*SQLite files at the start ofrunSync() - Force a WAL checkpoint immediately after
writeMeta()to persist meta into the main DB file - Add vitest coverage to validate meta visibility from a fresh read-only connection and stale file cleanup
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| extensions/memory-core/src/memory/manager-sync-ops.ts | Adds stale swap-file cleanup during sync and forces WAL checkpoint after meta writes |
| extensions/memory-core/src/memory/manager.wal-checkpoint.test.ts | Adds regression tests for WAL checkpoint behavior and stale file cleanup |
| PR_BODY.md | Documents the underlying failure modes, fixes, and new test coverage |
| if ( | ||
| (entry.startsWith(`${baseName}.backup-`) || | ||
| entry.startsWith(`${baseName}.tmp-`)) && | ||
| !entry.endsWith("-wal") && | ||
| !entry.endsWith("-shm") | ||
| ) { |
| function cleanupStaleIndexFiles(dbPath: string): void { | ||
| const dir = path.dirname(dbPath); | ||
| const baseName = path.basename(dbPath); | ||
| try { | ||
| const entries = fsSync.readdirSync(dir); |
| // Force WAL checkpoint so meta survives process crashes. | ||
| // Without this, the meta row may live only in the WAL file; if the | ||
| // process is killed before closeMemoryDatabase() checkpoints, the | ||
| // next startup reads no meta and declares the index missing. | ||
| try { | ||
| this.db.exec("PRAGMA wal_checkpoint(TRUNCATE)"); | ||
| } catch { |
| expect(metaRow).toBeDefined(); | ||
| const parsed = JSON.parse(metaRow!.value); | ||
| expect(parsed.model).toBeDefined(); | ||
| expect(parsed.provider).toBeDefined(); |
|
Codex review: needs real behavior proof before merge. Reviewed July 4, 2026, 1:41 PM ET / 17:41 UTC. Summary PR surface: Source +12, Tests +176. Total +188 across 2 files. Reproducibility: yes. at source level: current main publishes metadata to the live per-agent DB and then relies on later close-time WAL maintenance, leaving a plausible crash window before close. I did not run an OpenClaw hard-kill reproduction for the current post-publish path. Review metrics: 1 noteworthy metric.
Stored data model Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Proof guidance:
Risk before merge
Maintainer options:
Next step before merge
Security Review findings
Review detailsBest possible solution: Keep the live post-publish checkpoint idea, but implement it as a checked helper that reads the wal_checkpoint result, handles busy/incomplete frames explicitly, and proves live per-agent DB durability before close. Do we have a high-confidence way to reproduce the issue? Yes, at source level: current main publishes metadata to the live per-agent DB and then relies on later close-time WAL maintenance, leaving a plausible crash window before close. I did not run an OpenClaw hard-kill reproduction for the current post-publish path. Is this the best way to solve the issue? No, not yet. The checkpoint belongs after live DB publish, but it must inspect the SQLite result row and the regression test must prove the live DB checkpoint rather than any temp-DB close checkpoint. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 0869edc23b0d. Label changesLabel changes:
Label justifications:
Evidence reviewedPR surface: Source +12, Tests +176. Total +188 across 2 files. View PR surface stats
Acceptance criteria:
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
Reproduction evidencePer review feedback requesting real behavior proof, here is a minimal reproduction using Node's native SQLite (WAL mode) on Windows, demonstrating that meta written without a checkpoint does NOT survive a hard process kill. Script outputMethodEach phase:
After each phase, the main Findings
This directly corresponds to the production failure path: gateway process killed (OOM, Ctrl-C, power loss) between Why this matters on current mainThe Environment: Node v24.16.0, Windows 11, native |
|
Updated PR body with real behavior proof section (after-fix evidence from hard kill reproduction). The lint/extension-bundling CI failures appear to be from the sparse checkout not including workspace tooling config. @clawsweeper re-review |
|
@clawsweeper re-review Moved WAL checkpoint from writeMeta (shadow DB) to the live per-agent database after publishMemoryDatabaseTables completes. The checkpoint now uses prepare().get() to inspect the result row and logs a warning if busy frames remain (P2 addressed). writeMeta checkpoint removed because the shadow DB is closed+checkpointed immediately after writeMeta returns — the real vulnerability is between publish and live DB close. Key changes:
|
…ables The WAL checkpoint was inside writeMeta() which runs against the shadow/temp database during full reindex. The meta row gets copied to the live per-agent DB by publishMemoryDatabaseTables, so the checkpoint must happen on originalDb (the live DB) after publish returns — not on the shadow DB where writeMeta wrote it. This ensures the meta row is durable on disk before the process moves on. Without it, a crash between publish and close can leave meta only in the WAL file, causing the next startup to declare the index missing. Updates test comments to reflect the new checkpoint location.
|
Closing as no longer justified after the current per-agent SQLite and transactional table-publication changes. SQLite recovers committed transactions from the WAL after an ordinary process crash. This PR's reproduction intentionally copies only the main database file without its WAL, so it proves an unsupported live-database copy scenario rather than a current OpenClaw crash-loss path. I could not identify a current memory-core path that drops the live WAL; current cleanup is scoped to closed shadow databases. The remaining search/reindex reader race is tracked separately in #90361 and #90453. Happy to revisit this durability change if a current OpenClaw reproduction demonstrates loss of the live WAL or a supported operation that copies the database without its sidecars. |
Summary
One focused fix: after publishing the shadow index tables into the live per-agent database, sync immediately checkpoints the existing WAL-maintenance handle so the published metadata reaches the main database file.
What Problem This Solves
When the gateway process is killed after the shadow index is published but before scheduled or close-time WAL maintenance, the published metadata can remain only in the SQLite WAL. If that sidecar is lost, the next startup reads no metadata and declares the index "missing" even though chunks are present in the main database.
This puts the system into a permanent degraded state:
memory_searchreturns"index metadata is missing", normal syncs cannot self-heal (depending on provider availability), and the only recovery isopenclaw memory index --force.Real behavior proof
Minimal reproduction using Node v24.16.0 native SQLite in WAL mode on Windows 11, with child processes killed via
taskkill /F(SIGKILL equivalent):Method: Each phase creates a fresh WAL-mode SQLite DB, creates schema, inserts a chunk, and writes
memory_index_meta_v1. After each phase, the main.sqlitefile is copied to an isolated directory (no-walor-shmsidecar) and opened read-only — this ensures we read the checkpointed main file, not replayed WAL.Phase 2 proves the bug: after a hard kill without checkpoint, the main DB file has NO
metatable — the entire schema was WAL-only. On next startup, the manager sees an empty DB, auto-creates it, and declares"index metadata is missing".Phase 3 proves the fix: after publishing the rebuilt tables and successfully checkpointing the live database, the metadata is in the main file and survives the kill.
This corresponds to the production failure path: the gateway is killed after publishing the rebuilt index but before the next checkpoint, leaving the index identity dependent on the WAL sidecar.
Fix
After
publishMemoryDatabaseTables()commits the rebuilt tables, invoke the already configured SQLite WAL-maintenance checkpoint on the live database. A busy or failed checkpoint now fails the full-reindex attempt instead of silently claiming durability; the existing retry state schedules the work again.What this PR does NOT do
manager-atomic-reindex.tslifecycle — filed as follow-up)Related issues
The
needsMissingIdentityReindexauto-recovery path (already on main) can eventually heal this when the provider is available. This checkpoint fix prevents the meta loss in the first place, avoiding unnecessary full reindex cycles after crashes.Evidence
.sqlitefile (no WAL/SHM sidecars) and verifies the exactfts-only/nonemetadata from a fresh read-only connection.exec()call as success.