Summary

• Problem: when a safe (temp-DB) or unsafe (in-place) full memory reindex aborts mid-run, the committed index is preserved but the in-memory sync state is not — the next sync can silently converge without retrying the rolled-back work, and any successful embedding batches are thrown away with the temp DB.
• Why it matters: transient provider failures during memory index --force should degrade to "retry the same work next sync" and should not waste the embedding batches that already succeeded before the failure. Today they do both.
• What changed: snapshot the sync state (dirty flags, session dirty files, session deltas) before a full reindex and restore the union on rollback; add a sessionFullRetryPending sentinel so a failed full session rebuild still forces the next sync onto the full path even if sessionsDirtyFiles was cleared; during a safe reindex, mirror every successful embedding cache batch back to the original DB so the cache survives a later-batch rollback; and have seedEmbeddingCache check for the embedding_cache table on the source DB so enabling cache on an older index no longer crashes the reindex.
• What did NOT change (scope boundary): no new config knobs, no provider API changes, no query behavior changes, no schema migrations. The committed index swap remains atomic and transport-layer retry work stays out of scope (see memory reindex aborts on transient embedding transport errors instead of retrying or splitting the batch #44166).

AI-assisted: yes. I used Claude Opus 4.7 to port the original fix onto the current manager-sync-ops / manager-embedding-ops layout and reviewed the diff and validation output.

Change Type (select all)

• Bug fix
• Feature
• Refactor required for the fix
• Docs
• Security hardening
• Chore/infra

Scope (select all touched areas)

• Gateway / orchestration
• Skills / tool execution
• Auth / tokens
• Memory / storage
• Integrations
• API / contracts
• UI / DX
• CI/CD / infra

Linked Issue/PR

• Closes: none
• Related: Bug: Memory Index Fails for Main Agent with 'fetch failed' Error #56901, memory index --force fails with 'fetch failed' on large workspace (1400+ .md files) #56427, memory reindex aborts on transient embedding transport errors instead of retrying or splitting the batch #44166, [Bug]: Session memory index silently dropped on every gateway restart (shouldSync Sessions priority bug) #45981, builtin memory_search can return empty results until builtin index is forced to reindex #56862
• Supersedes: fix(memory): harden builtin sync and reindex semantics #55497
• This PR fixes a bug or regression

Real behavior proof (required for external PRs)

• Behavior or issue addressed: an offline force-reindex that aborts mid-run used to leave Embedding cache at 0 on retry, throwing away the embedding work the provider had already returned before the failure.
• Real environment tested: macOS, Node 24.13.0, pnpm + OpenClaw checkout; two isolated $OPENCLAW_HOME dirs with identical starting DB (14408 chunks · 0 cache rows) and identical workspace (45 local markdown files). Remote provider: real SiliconFlow openai / BAAI/bge-m3, memorySearch.cache.enabled: true, cap = 500000. Git worktrees at upstream/main and fix/memory-reindex-recovery.
• Exact steps or command run after this patch:
  1. sqlite3 $OPENCLAW_HOME/.openclaw/memory/main.sqlite 'DELETE FROM embedding_cache;' (baseline confirmed: 0 rows).
  2. OPENCLAW_HOME=... pnpm openclaw memory index --force (against real provider).
  3. Wi-Fi off ~5 s in (macOS airport disable), mid-reindex.
  4. OPENCLAW_HOME=... pnpm openclaw memory status.
  5. sqlite3 $OPENCLAW_HOME/.openclaw/memory/main.sqlite 'SELECT COUNT(*) FROM embedding_cache;'.
• Evidence after fix: terminal capture from the isolated home-fix run — Memory index failed (main): fetch failed, then openclaw memory status prints Embedding cache: enabled (109 entries), and sqlite3 ... 'SELECT COUNT(*) FROM embedding_cache' prints 109. Row breakdown SELECT provider, model, COUNT(*) ... returns openai|BAAI/bge-m3|109 with MIN/MAX(updated_at) both landing inside the interrupted reindex window (2026-05-09 10:01:34..10:01:36). Full A/B terminal transcript is pasted under ## Evidence below.
• Observed result after fix: on upstream/main the same offline force-reindex leaves embedding_cache at 0 rows; on fix/memory-reindex-recovery the same run preserves 109 rows written through the mirror write, which survive the temp-DB rollback. The committed chunks / files table counts are unchanged on both sides (2384 files · 14408 chunks), confirming the atomic swap is still atomic.
• What was not tested: a live CLI repro for the session sessionFullRetryPending state machine (covered by focused manager-level tests instead); multi-process concurrent indexing; running the full pre-fix flow against a remote provider that rate-limits instead of failing DNS.
• Before evidence: on upstream/main (64514a6) the same offline force-reindex emits Memory index failed (main): getaddrinfo ENOTFOUND api.siliconflow.cn, then openclaw memory status prints Embedding cache: enabled (0 entries) and sqlite3 ... 'SELECT COUNT(*) FROM embedding_cache' prints 0 — captured in this same evidence window.

Root Cause (if applicable)

• Root cause: runSafeReindex / runUnsafeReindex preserved storage/vector state on rollback but did not snapshot or restore the in-memory sync state (dirty, sessionsDirty, sessionsDirtyFiles, sessionDeltas). A failed full session rebuild could clear the per-file dirty filter mid-run and leave no sentinel for the next sync to see, so shouldSyncSessionsForReindex fell back to sessionsDirty && dirtySessionFileCount > 0 and silently downgraded to an incremental sync. Separately, embedChunksInBatches accumulated cache entries until all batches finished and wrote them only to the temp DB, so a later-batch failure threw away every successful batch along with the temp DB.
• Missing detection / guardrail: existing atomic-reindex coverage asserted that the previous index survives a failed temp reindex, but did not exercise retry-state restoration, post-compaction targeted refresh after a pending full retry, or cache preservation across temp-DB rollback.
• Contributing context: after the memory engine moved into extensions/memory-core/src/memory/* with the helper-split layout, the old reliability work in fix(memory): harden builtin sync and reindex semantics #55497 no longer applied cleanly. This PR replays the narrow fix on top of the current layout.

Regression Test Plan (if applicable)

• Coverage level that should have caught this:
  • Unit test
  • Seam / integration test
  • End-to-end test
  • Existing coverage already sufficient
• Target test or file:
  • extensions/memory-core/src/memory/manager-sync-recovery.test.ts (new)
  • extensions/memory-core/src/memory/manager-session-reindex.test.ts (new)
  • extensions/memory-core/src/memory/manager.reindex-recovery.test.ts (new)
• Scenario the test should lock in:
  • snapshotSyncState / computeRestoredSyncState union sticky dirty flags and dirty file sets; mergeSessionDeltas takes the later lastSize and the conservative pendingBytes / pendingMessages; restoreRetryStateAfterReindexRollback flips the sentinel only when the failed attempt actually began a full session rebuild.
  • shouldSyncSessionsForReindex forces a full sync when sessionFullRetryPending is set, even with zero dirty files.
  • A real manager where batch fix: add @lid format support and allowFrom wildcard handling #1 succeeds and batch Login fails with 'WebSocket Error (socket hang up)' ECONNRESET #2 throws: the committed index is preserved, the embedding_cache on the original DB has rows after the failure, a subsequent sync({ force: true }) succeeds, and enabling cache on an index that was built without the embedding_cache table no longer crashes.
• Why this is the smallest reliable guardrail: the invariants live in local SQLite-backed state and per-method sync bookkeeping, so focused unit + seam tests exercise the contract without requiring live providers or gateway E2E.
• Existing test that already covers this (if any): manager.atomic-reindex.test.ts covered "previous index survives failed temp reindex" but not retry state or cache preservation.
• If no new test is added, why not: N/A.

User-visible / Behavior Changes

• Full memory reindex continues to keep the previous committed index on failure, and now also keeps enough dirty/retry state for the next sync to finish the rolled-back work.
• A failed full session rebuild sets sessionFullRetryPending so the next normal sync still performs a full session retry instead of silently downgrading to an incremental pass.
• Successful embedding batches survive a later safe-reindex rollback via cache mirroring, reducing repeated provider calls on retry.
• Indexes built before the embedding_cache table existed can enable cache without failing reindex.

Diagram (if applicable)

Before:
[force reindex] -> [temp DB builds some state] -> [later batch failure]
               -> [committed DB restored]
               -> [dirty flags lost / full-retry intent lost / temp cache discarded]

After:
[force reindex] -> [snapshot sync state] -> [temp DB builds some state]
               -> [each successful cache batch mirrored to committed DB]
               -> [later batch failure]
               -> [committed DB restored + sync state restored + sessionFullRetryPending set]
               -> [next normal sync rebuilds the rolled-back work]

Security Impact (required)

• New permissions/capabilities? (No)
• Secrets/tokens handling changed? (No)
• New/changed network calls? (No)
• Command/tool execution surface changed? (No)
• Data access scope changed? (No)
• If any Yes, explain risk + mitigation: N/A.

Repro + Verification

Environment

• OS: macOS (Darwin 25.2.0)
• Runtime/container: Node 24.13.0 + pnpm + Vitest
• Model/provider: OpenAI-compatible remote embeddings (provider: openai, model: BAAI/bge-m3) for manual repro; mocked provider in targeted tests
• Integration/channel (if any): CLI / builtin memory index
• Relevant config (redacted):
  • memorySearch.provider: "openai"
  • memorySearch.model: "BAAI/bge-m3"
  • memorySearch.cache.enabled: true

Steps

1. Start from an existing committed index (14408 chunks, 0 cache rows) in an isolated $OPENCLAW_HOME.
2. sqlite3 $OPENCLAW_HOME/.openclaw/memory/main.sqlite 'DELETE FROM embedding_cache;'
3. OPENCLAW_HOME=... pnpm openclaw memory index --force (real SiliconFlow provider).
4. Turn Wi-Fi off ~5 s into the reindex.
5. Wait for the CLI to fail (on main) or cancel once retry backoff has spent the retry budget (on fix, see note below).
6. OPENCLAW_HOME=... pnpm openclaw memory status and sqlite3 ... 'SELECT COUNT(*) FROM embedding_cache;'

Note on fix-side cancellation: the retry classifier treats fetch failed-class wrapped errors as retryable, so per-item fallback after a failed batch keeps retrying for a while; on main the error surfaced as raw getaddrinfo ENOTFOUND and exited on its own. Early-cancelling the fix run after the retries are all exhausted does not affect the 0 vs 109 evidence — those rows were already mirrored to the original DB before cancellation. Tightening that retry classifier for obviously-permanent network failures is out of scope here (see "Known follow-ups" below).

Expected

• Force reindex fails with a transport error.
• Committed index is preserved.
• On main: Embedding cache count stays at 0 (successful batches are in the temp DB that gets thrown away).
• On this branch: Embedding cache count > 0 (mirrored to the original DB).

Actual

• On upstream/main (64514a6): Memory index failed (main): getaddrinfo ENOTFOUND api.siliconflow.cn; memory status → Embedding cache: enabled (0 entries); SELECT COUNT(*) → 0.
• On fix/memory-reindex-recovery-v2 (0ed55aa): memory status → Embedding cache: enabled (109 entries); SELECT COUNT(*) → 109, all rows provider=openai / model=BAAI/bge-m3, updated_at inside the reindex window.

Evidence

Attach at least one:

• Failing test/log before + passing after
• Trace/log snippets
• Screenshot/recording
• Perf numbers (if relevant)

Validation run:

pnpm test extensions/memory-core
  Test Files  56 passed (56)
  Tests  631 passed (631)

pnpm exec oxfmt --check --threads=1 \
  extensions/memory-core/src/memory/manager-sync-ops.ts \
  extensions/memory-core/src/memory/manager-sync-recovery.ts \
  extensions/memory-core/src/memory/manager-sync-recovery.test.ts \
  extensions/memory-core/src/memory/manager-session-reindex.ts \
  extensions/memory-core/src/memory/manager-session-reindex.test.ts \
  extensions/memory-core/src/memory/manager-embedding-ops.ts \
  extensions/memory-core/src/memory/manager.reindex-recovery.test.ts \
  CHANGELOG.md
  All matched files use the correct format.

pnpm tsgo:extensions && pnpm tsgo:test
  (both green)

pnpm check:changed
  EXIT=0 (typecheck all + lint + import cycles + changelog attributions)

Manual CLI A/B (same starting DB and workspace in both isolated homes, real SiliconFlow provider, Wi-Fi turned off mid-reindex):

# upstream/main (64514a6548)
$ OPENCLAW_HOME=/.../home-main pnpm openclaw memory index --force
[memory] embeddings rate limited; retrying in 591ms
[memory] embeddings rate limited; retrying in 502ms
[memory] embeddings rate limited; retrying in 524ms
[memory] embeddings rate limited; retrying in 550ms
Memory index failed (main): getaddrinfo ENOTFOUND api.siliconflow.cn

$ OPENCLAW_HOME=/.../home-main pnpm openclaw memory status
  Indexed: 2384/45 files · 14408 chunks
  Embedding cache: enabled (0 entries)       # <- committed index preserved but cache gone

$ sqlite3 .../home-main/.openclaw/memory/main.sqlite 'SELECT COUNT(*) FROM embedding_cache;'
0

# fix/memory-reindex-recovery-v2 (0ed55aac0c)
$ OPENCLAW_HOME=/.../home-fix pnpm openclaw memory index --force
[memory] embeddings rate limited; retrying in 540ms
[memory] embeddings rate limited; retrying in 593ms
[memory] embeddings rate limited; retrying in 586ms
^C                                           # <- cancelled after retry loop exhausted

$ OPENCLAW_HOME=/.../home-fix pnpm openclaw memory status
  Indexed: 2384/45 files · 14408 chunks
  Embedding cache: enabled (109 entries)     # <- mirrored to original DB before rollback

$ sqlite3 .../home-fix/.openclaw/memory/main.sqlite \
    "SELECT provider, model, COUNT(*) FROM embedding_cache GROUP BY provider, model;"
openai|BAAI/bge-m3|109
$ sqlite3 .../home-fix/.openclaw/memory/main.sqlite \
    "SELECT datetime(MIN(updated_at)/1000, 'unixepoch', 'localtime'),
            datetime(MAX(updated_at)/1000, 'unixepoch', 'localtime'),
            COUNT(*) FROM embedding_cache;"
2026-05-09 10:01:34|2026-05-09 10:01:36|109   # <- rows written inside the reindex window

Human Verification (required)

• Verified scenarios:
  • pnpm test extensions/memory-core — 56 files / 631 tests green, including the three new suites for recovery helpers, shouldSyncSessionsForReindex, and the manager-level rollback integration.
  • Traced the current code paths to confirm the three gaps (no sync-state snapshot on rollback, no full-retry sentinel, no mirror write) still exist on upstream/main before this PR.
  • Forced a second-batch failure in the manager-level test and asserted the original DB has embedding_cache rows after rollback.
  • Dropped embedding_cache from an existing index and reopened with cache enabled — reindex completes and the table is re-created.
  • Manual CLI A/B against real SiliconFlow (BAAI/bge-m3), Wi-Fi cut mid-reindex: main cache stays at 0 after rollback, this branch preserves 109 rows (provider+model+timestamp all from the interrupted reindex window).
• Edge cases checked:
  • Safe reindex rollback after the first batch succeeded and a later batch threw.
  • Cache table absent in the original DB when enabling cache.
  • Targeted post-compaction session refresh preserves sessionFullRetryPending (it only clears on a full rebuild success, not on targeted refresh).
  • mergeSessionDeltas when one side has pendingBytes: 0 and the other does not — the conservative min still does not silently wipe pending work.
• What you did not verify:
  • Live CLI repro for the session sessionFullRetryPending state machine (covered by focused manager-level tests).
  • Batch API fallback flows beyond the local regression tests.
  • Multi-process concurrent indexing.

Known follow-ups (out of scope for this PR)

• isRetryableMemoryEmbeddingError classifies any fetch failed-wrapped error as retryable, including permanent failures like getaddrinfo ENOTFOUND. Combined with per-item fallback after a failed batch, an offline reindex on some Node versions keeps looping for a long time before the process exits — surfaces as an apparent hang. Pre-existing on main, unrelated to rollback recovery. A separate narrow PR can add ENOTFOUND to the non-retryable set or put a total-elapsed budget on the retry loop.

Review Conversations

• I replied to or resolved every bot review conversation I addressed in this PR.
• I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

• Backward compatible? (Yes)
• Config/env changes? (No)
• Migration needed? (No)
• If yes, exact upgrade steps: N/A.

Risks and Mitigations

• Risk: sessionFullRetryPending adds one bit to the session sync state machine.
  • Mitigation: shouldSyncSessionsForReindex is the single point of consumption, unit-tested in isolation; targeted session refresh preserves the flag and only full-rebuild success clears it.
• Risk: cache mirroring during safe reindex writes successful embedding batches into the original DB before the temp DB is committed.
  • Mitigation: the mirror is scoped to embedding_cache rows only (reusable work), never the searchable index rows; the committed searchable index swap remains atomic; regression coverage locks in the intended rollback behavior.
• Risk: older indexes may not have embedding_cache.
  • Mitigation: seedEmbeddingCache probes for the table in sqlite_master before reading or mirroring; regression test exercises enabling cache on a pre-cache index.