Skip to content

fix(memory-core): checkpoint WAL after publishing index metadata#92509

Closed
Dazlarus wants to merge 7 commits into
openclaw:mainfrom
Dazlarus:fix/memory-meta-recovery
Closed

fix(memory-core): checkpoint WAL after publishing index metadata#92509
Dazlarus wants to merge 7 commits into
openclaw:mainfrom
Dazlarus:fix/memory-meta-recovery

Conversation

@Dazlarus

@Dazlarus Dazlarus commented Jun 12, 2026

Copy link
Copy Markdown

Summary

One focused fix: after publishing the shadow index tables into the live per-agent database, sync immediately checkpoints the existing WAL-maintenance handle so the published metadata reaches the main database file.

What Problem This Solves

When the gateway process is killed after the shadow index is published but before scheduled or close-time WAL maintenance, the published metadata can remain only in the SQLite WAL. If that sidecar is lost, the next startup reads no metadata and declares the index "missing" even though chunks are present in the main database.

This puts the system into a permanent degraded state: memory_search returns "index metadata is missing", normal syncs cannot self-heal (depending on provider availability), and the only recovery is openclaw memory index --force.

Real behavior proof

Minimal reproduction using Node v24.16.0 native SQLite in WAL mode on Windows 11, with child processes killed via taskkill /F (SIGKILL equivalent):

=== WAL Meta Durability Reproduction ===

Phase 1: Normal close (baseline)
  WAL size after writes: 0 bytes
  Meta in main file: PRESENT

Phase 2: Hard kill WITHOUT fix
  WAL size: 0 bytes
  Meta in main file: MISSING (no meta table — data was WAL-only)

Phase 3: Hard kill WITH wal_checkpoint fix
  WAL size: 0 bytes
  Meta in main file: PRESENT

Method: Each phase creates a fresh WAL-mode SQLite DB, creates schema, inserts a chunk, and writes memory_index_meta_v1. After each phase, the main .sqlite file is copied to an isolated directory (no -wal or -shm sidecar) and opened read-only — this ensures we read the checkpointed main file, not replayed WAL.

Phase 2 proves the bug: after a hard kill without checkpoint, the main DB file has NO meta table — the entire schema was WAL-only. On next startup, the manager sees an empty DB, auto-creates it, and declares "index metadata is missing".

Phase 3 proves the fix: after publishing the rebuilt tables and successfully checkpointing the live database, the metadata is in the main file and survives the kill.

This corresponds to the production failure path: the gateway is killed after publishing the rebuilt index but before the next checkpoint, leaving the index identity dependent on the WAL sidecar.

Fix

After publishMemoryDatabaseTables() commits the rebuilt tables, invoke the already configured SQLite WAL-maintenance checkpoint on the live database. A busy or failed checkpoint now fails the full-reindex attempt instead of silently claiming durability; the existing retry state schedules the work again.

What this PR does NOT do

  • ❌ No stale file cleanup (belongs in manager-atomic-reindex.ts lifecycle — filed as follow-up)
  • ❌ No changes to the atomic reindex swap protocol
  • ❌ No new dependencies or schema migrations

Related issues

The needsMissingIdentityReindex auto-recovery path (already on main) can eventually heal this when the provider is available. This checkpoint fix prevents the meta loss in the first place, avoiding unnecessary full reindex cycles after crashes.

Evidence

  • Main-file integration test: while the manager remains open, copies only the canonical .sqlite file (no WAL/SHM sidecars) and verifies the exact fts-only / none metadata from a fresh read-only connection.
  • Checkpoint contract: uses the existing WAL-maintenance helper, which reads SQLite's checkpoint result and rejects a busy checkpoint instead of treating a result-less exec() call as success.

Copilot AI review requested due to automatic review settings June 12, 2026 18:26
@openclaw-barnacle openclaw-barnacle Bot added extensions: memory-core Extension: memory-core size: M triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 12, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Hardening the memory-core SQLite index lifecycle to prevent “index metadata is missing” after interrupted reindex/sync by forcing WAL checkpointing after meta writes and cleaning up leftover swap artifacts.

Changes:

  • Add best-effort cleanup of stale .backup-* / .tmp-* SQLite files at the start of runSync()
  • Force a WAL checkpoint immediately after writeMeta() to persist meta into the main DB file
  • Add vitest coverage to validate meta visibility from a fresh read-only connection and stale file cleanup

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File Description
extensions/memory-core/src/memory/manager-sync-ops.ts Adds stale swap-file cleanup during sync and forces WAL checkpoint after meta writes
extensions/memory-core/src/memory/manager.wal-checkpoint.test.ts Adds regression tests for WAL checkpoint behavior and stale file cleanup
PR_BODY.md Documents the underlying failure modes, fixes, and new test coverage

Comment on lines +132 to +137
if (
(entry.startsWith(`${baseName}.backup-`) ||
entry.startsWith(`${baseName}.tmp-`)) &&
!entry.endsWith("-wal") &&
!entry.endsWith("-shm")
) {
Comment on lines +126 to +130
function cleanupStaleIndexFiles(dbPath: string): void {
const dir = path.dirname(dbPath);
const baseName = path.basename(dbPath);
try {
const entries = fsSync.readdirSync(dir);
Comment on lines +2618 to +2624
// Force WAL checkpoint so meta survives process crashes.
// Without this, the meta row may live only in the WAL file; if the
// process is killed before closeMemoryDatabase() checkpoints, the
// next startup reads no meta and declares the index missing.
try {
this.db.exec("PRAGMA wal_checkpoint(TRUNCATE)");
} catch {
Comment on lines +100 to +103
expect(metaRow).toBeDefined();
const parsed = JSON.parse(metaRow!.value);
expect(parsed.model).toBeDefined();
expect(parsed.provider).toBeDefined();
@clawsweeper

clawsweeper Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 4, 2026, 1:41 PM ET / 17:41 UTC.

Summary
The PR adds a live per-agent SQLite WAL checkpoint after memory table publish and adds WAL checkpoint regression tests for memory-core.

PR surface: Source +12, Tests +176. Total +188 across 2 files.

Reproducibility: yes. at source level: current main publishes metadata to the live per-agent DB and then relies on later close-time WAL maintenance, leaving a plausible crash window before close. I did not run an OpenClaw hard-kill reproduction for the current post-publish path.

Review metrics: 1 noteworthy metric.

  • SQLite checkpoint surface: 1 synchronous live-DB checkpoint added. The new checkpoint is small but sits on the memory reindex publish path, so maintainers should review both durability and latency/lock behavior before merge.

Stored data model
Persistent data-model change detected: unknown-data-model-change: extensions/memory-core/src/memory/manager.wal-checkpoint.test.ts, vector/embedding metadata: extensions/memory-core/src/memory/manager.wal-checkpoint.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🦪 silver shellfish
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Read and handle the live DB wal_checkpoint(TRUNCATE) result row, including busy/incomplete frames.
  • Replace the spy-only test with proof that fails on current main and verifies live per-agent DB durability before close.
  • [P1] Add redacted real OpenClaw proof for the current post-publish path, such as a hard-kill or equivalent transcript that shows metadata survives without relying on close-time replay.

Proof guidance:

  • [P1] Needs stronger real behavior proof before merge: The PR includes copied terminal output for a minimal Node SQLite WAL reproduction, but it does not prove the current OpenClaw post-publish live-DB checkpoint path before close after the latest patch shape. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

  • [P1] The synchronous TRUNCATE checkpoint can add lock contention or latency after full reindex, which green CI does not fully settle for large live memory stores.
  • [P1] Because the patch ignores SQLite's checkpoint result row, a busy or incomplete checkpoint can leave the original crash window open while the code proceeds as if durability was achieved.
  • [P1] The supplied proof demonstrates the general SQLite WAL behavior, but not the current OpenClaw post-publish live-DB path before close.

Maintainer options:

  1. Fix Checked Checkpoint Semantics (recommended)
    Read the wal_checkpoint result row on the live DB, treat busy or uncheckpointed frames as an explicit warning/retry/failure path, and add a regression test that fails on current main.
  2. Accept Best-Effort Durability
    Maintainers may intentionally accept a best-effort checkpoint, but the PR body and tests should stop claiming the metadata is guaranteed durable before close.
  3. Coordinate With Safe-Reindex PR
    Pause this branch if maintainers want the broader safe-reindex read-latch work in fix(memory-core): guard searches during safe reindex #90453 to land first and then rebase this narrower durability fix on top.

Next step before merge

  • [P1] The branch needs contributor changes and new real behavior proof; ClawSweeper should not request an automated repair while the external proof gate remains insufficient.

Security
Cleared: Security review cleared: the diff only changes memory-core SQLite checkpoint behavior and tests, with no dependency, workflow, secret, permission, package-resolution, or external code execution changes.

Review findings

  • [P2] Check the checkpoint result before continuing — extensions/memory-core/src/memory/manager-sync-ops.ts:2802
  • [P2] Make the checkpoint test fail without the live checkpoint — extensions/memory-core/src/memory/manager.wal-checkpoint.test.ts:132-139
Review details

Best possible solution:

Keep the live post-publish checkpoint idea, but implement it as a checked helper that reads the wal_checkpoint result, handles busy/incomplete frames explicitly, and proves live per-agent DB durability before close.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: current main publishes metadata to the live per-agent DB and then relies on later close-time WAL maintenance, leaving a plausible crash window before close. I did not run an OpenClaw hard-kill reproduction for the current post-publish path.

Is this the best way to solve the issue?

No, not yet. The checkpoint belongs after live DB publish, but it must inspect the SQLite result row and the regression test must prove the live DB checkpoint rather than any temp-DB close checkpoint.

Full review comments:

  • [P2] Check the checkpoint result before continuing — extensions/memory-core/src/memory/manager-sync-ops.ts:2802
    wal_checkpoint(TRUNCATE) can complete as a SQL statement while reporting busy=1 or uncheckpointed frames in its result row, and DatabaseSync.exec() discards that row. If a reader keeps the WAL busy, this code continues with the same pre-close crash window this PR is trying to close; use prepare(...).get() on the live DB and handle busy/incomplete results with a warning, retry, or failure that the test covers. (sqlite.org) (nodejs.org)
    Confidence: 0.89
  • [P2] Make the checkpoint test fail without the live checkpoint — extensions/memory-core/src/memory/manager.wal-checkpoint.test.ts:132-139
    This spy counts any PRAGMA wal_checkpoint(TRUNCATE) during sync(), but current main already runs one when closeMemoryDatabase(tempDb) closes the shadow DB inside the same sync. That means the test can pass without the new live originalDb checkpoint; assert the checkpoint happens on the live per-agent DB or read the main DB before close with WAL sidecars removed so the regression is pinned.
    Confidence: 0.86

Overall correctness: patch is incorrect
Overall confidence: 0.87

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 0869edc23b0d.

Label changes

Label changes:

  • add merge-risk: 🚨 session-state: The diff changes how durable memory index metadata is flushed, which controls whether agent memory/session recall state is considered valid after restart.
  • add merge-risk: 🚨 availability: A synchronous TRUNCATE checkpoint on the live per-agent DB can block or contend with other SQLite users during full reindex.
  • add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦪 silver shellfish.
  • add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR includes copied terminal output for a minimal Node SQLite WAL reproduction, but it does not prove the current OpenClaw post-publish live-DB checkpoint path before close after the latest patch shape. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Label justifications:

  • P1: The PR targets a first-class memory_search regression family that can make agent memory recall unavailable after reindex or crash-adjacent metadata loss.
  • merge-risk: 🚨 session-state: The diff changes how durable memory index metadata is flushed, which controls whether agent memory/session recall state is considered valid after restart.
  • merge-risk: 🚨 availability: A synchronous TRUNCATE checkpoint on the live per-agent DB can block or contend with other SQLite users during full reindex.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦪 silver shellfish.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR includes copied terminal output for a minimal Node SQLite WAL reproduction, but it does not prove the current OpenClaw post-publish live-DB checkpoint path before close after the latest patch shape. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +12, Tests +176. Total +188 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 12 0 +12
Tests 1 176 0 +176
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 188 0 +188

Acceptance criteria:

  • [P1] A focused test that fails on current main and proves the live per-agent DB checkpoint, not the shadow DB close checkpoint.
  • [P1] A test or source proof for busy/incomplete wal_checkpoint result handling.
  • [P1] Redacted real OpenClaw after-fix proof for the current post-publish live-DB path before close.

What I checked:

  • Repository policy read: Read the full root AGENTS.md and scoped extensions/AGENTS.md; the deep review, plugin boundary, SQLite/storage, proof, and merge-risk guidance applied to this memory-core PR. (AGENTS.md:1, 0869edc23b0d)
  • Current main publish path: Current main writes metadata to the shadow DB, closes/checkpoints that temp DB, publishes memory-owned tables into the live per-agent DB, and then restores this.db to originalDb without a live post-publish checkpoint. (extensions/memory-core/src/memory/manager-sync-ops.ts:2779, 0869edc23b0d)
  • PR implementation: The PR head adds originalDb.exec("PRAGMA wal_checkpoint(TRUNCATE)") immediately after publishMemoryDatabaseTables, which targets the live DB but does not inspect the checkpoint result row. (extensions/memory-core/src/memory/manager-sync-ops.ts:2802, ee66b808412a)
  • SQLite checkpoint contract: SQLite documents that wal_checkpoint returns a single row and that the first column is 1 when a FULL/RESTART/TRUNCATE checkpoint was blocked from completing; Node's DatabaseSync.exec executes SQL without returning results. (sqlite.org) (nodejs.org)
  • Test gap: The PR test only counts any PRAGMA wal_checkpoint(TRUNCATE) during sync, but current main already checkpoints the temp DB when closeMemoryDatabase(tempDb) runs inside sync, so this does not distinguish the new live-DB checkpoint. (extensions/memory-core/src/memory/manager.wal-checkpoint.test.ts:132, ee66b808412a)
  • Related PR 90453: The broader safe-reindex PR remains open and viable for the in-process read race, but it rewrites metadata on the live DB without adding a post-write WAL checkpoint, so it only partially overlaps this PR rather than fully superseding it. (extensions/memory-core/src/memory/manager-sync-ops.ts:2864, a820c14e4da6)

Likely related people:

  • vincentkoc: Recent memory manager and per-agent SQLite lifecycle history touches manager-sync-ops.ts and manager-db.ts, the central files for this post-publish checkpoint path. (role: recent area contributor; confidence: high; commits: 2caa4e2bd748, 8bd32a790b9f, e085fa1a3ffd; files: extensions/memory-core/src/memory/manager-sync-ops.ts, extensions/memory-core/src/memory/manager-db.ts)
  • steipete: History and prior review context show substantial memory manager extraction and atomic reindex refactor work that frames the current owner boundary. (role: historical memory-core refactor contributor; confidence: medium; commits: c9f288ceafbf, d2a03eca1a5e, f4d8393bf4c1; files: extensions/memory-core/src/memory/manager-sync-ops.ts, extensions/memory-core/src/memory/manager.ts)
  • 849261680: Authored the open safe-reindex read-race PR that overlaps the same missing-metadata cluster and changes the same manager lifecycle paths. (role: adjacent candidate-fix author; confidence: medium; commits: a820c14e4da6; files: extensions/memory-core/src/memory/manager.ts, extensions/memory-core/src/memory/manager-sync-ops.ts, extensions/memory-core/src/memory/manager.reindex-read-race.test.ts)
  • xydt-tanshanshan: Merged work preserving live SQLite indexes during swaps overlaps the same atomic reindex and stale-handle lifecycle area. (role: adjacent reindex lifecycle contributor; confidence: medium; commits: 865fdab07501; files: extensions/memory-core/src/memory/manager-sync-ops.ts, extensions/memory-core/src/memory/manager-db.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels Jun 12, 2026
@Dazlarus

Copy link
Copy Markdown
Author

Reproduction evidence

Per review feedback requesting real behavior proof, here is a minimal reproduction using Node's native SQLite (WAL mode) on Windows, demonstrating that meta written without a checkpoint does NOT survive a hard process kill.

Script output

=== WAL Meta Durability Reproduction ===

Phase 1: Normal close (baseline)
  WAL size after writes: 0 bytes
  Meta in main file: PRESENT

Phase 2: Hard kill WITHOUT fix
  WAL size: 0 bytes
  Meta in main file: MISSING (no meta table — data was WAL-only)

Phase 3: Hard kill WITH wal_checkpoint fix
  WAL size: 0 bytes
  Meta in main file: PRESENT

Method

Each phase:

  1. Creates a fresh WAL-mode SQLite DB
  2. Creates schema, inserts a chunk, writes memory_index_meta_v1 to the meta table
  3. Phase 1: calls db.close() (normal shutdown — checkpoints WAL)
  4. Phase 2: taskkill /F (SIGKILL equivalent) — no checkpoint, no clean close
  5. Phase 3: taskkill /F AFTER running PRAGMA wal_checkpoint(TRUNCATE)

After each phase, the main .sqlite file is copied to an isolated directory (no -wal or -shm sidecar) and opened read-only. This ensures we are reading the checkpointed main file, not replaying WAL.

Findings

  • Phase 2 proves the bug: after a hard kill without checkpoint, the main DB file has NO meta table at all — the entire schema was WAL-only. On next startup, the manager sees an empty/missing DB, auto-creates it, and declares "index metadata is missing".
  • Phase 3 proves the fix: with PRAGMA wal_checkpoint(TRUNCATE) after writeMeta, the meta row is in the main file and survives the kill.

This directly corresponds to the production failure path: gateway process killed (OOM, Ctrl-C, power loss) between writeMeta() and closeMemoryDatabase() → meta lost → memory_search returns "index metadata is missing" permanently.

Why this matters on current main

The needsMissingIdentityReindex auto-recovery path (already on main) can eventually heal this — but only when the provider is available and no target session files block it. The checkpoint fix prevents the meta loss in the first place, avoiding an unnecessary full reindex cycle on every restart after a crash.

Environment: Node v24.16.0, Windows 11, native node:sqlite with WAL mode.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jun 12, 2026
@Dazlarus

Copy link
Copy Markdown
Author

Updated PR body with real behavior proof section (after-fix evidence from hard kill reproduction). The lint/extension-bundling CI failures appear to be from the sparse checkout not including workspace tooling config. @clawsweeper re-review

@clawsweeper clawsweeper Bot added rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 12, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. labels Jun 15, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. label Jun 15, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Jun 15, 2026
@clawsweeper clawsweeper Bot added the rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. label Jul 2, 2026
@Dazlarus

Dazlarus commented Jul 3, 2026

Copy link
Copy Markdown
Author

@clawsweeper re-review

Moved WAL checkpoint from writeMeta (shadow DB) to the live per-agent database after publishMemoryDatabaseTables completes. The checkpoint now uses prepare().get() to inspect the result row and logs a warning if busy frames remain (P2 addressed). writeMeta checkpoint removed because the shadow DB is closed+checkpointed immediately after writeMeta returns — the real vulnerability is between publish and live DB close.

Key changes:

  • Checkpoint targets originalDb (live) after publish, not shadow DB
  • Result row inspected for busy frames (P2)
  • Shadow DB gets its own close-time checkpoint via closeMemoryDatabase()

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 3, 2026
…ables

The WAL checkpoint was inside writeMeta() which runs against the
shadow/temp database during full reindex. The meta row gets copied
to the live per-agent DB by publishMemoryDatabaseTables, so the
checkpoint must happen on originalDb (the live DB) after publish
returns — not on the shadow DB where writeMeta wrote it.

This ensures the meta row is durable on disk before the process
moves on. Without it, a crash between publish and close can leave
meta only in the WAL file, causing the next startup to declare the
index missing.

Updates test comments to reflect the new checkpoint location.
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 4, 2026
@clawsweeper clawsweeper Bot removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 4, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels Jul 6, 2026
@Dazlarus Dazlarus changed the title fix(memory-core): WAL checkpoint after writeMeta + stale index file cleanup fix(memory-core): checkpoint WAL after publishing index metadata Jul 14, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jul 14, 2026

Copy link
Copy Markdown
Author

Closing as no longer justified after the current per-agent SQLite and transactional table-publication changes.

SQLite recovers committed transactions from the WAL after an ordinary process crash. This PR's reproduction intentionally copies only the main database file without its WAL, so it proves an unsupported live-database copy scenario rather than a current OpenClaw crash-loss path. I could not identify a current memory-core path that drops the live WAL; current cleanup is scoped to closed shadow databases.

The remaining search/reindex reader race is tracked separately in #90361 and #90453. Happy to revisit this durability change if a current OpenClaw reproduction demonstrates loss of the live WAL or a supported operation that copies the database without its sidecars.

@Dazlarus Dazlarus closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: memory-core Extension: memory-core merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants