Skip to content

memory-core main reindex thrashes, leaks main.sqlite.tmp files, and leaves memory_search paused after repair #90508

Description

@joeykrug

Summary

On OpenClaw 2026.6.1 (2e08f0f), the main memory index got into a reindex-thrash loop on a large active index. The live SQLite DB had chunks and cached embeddings, but no identity metadata row, so vector search failed closed with index metadata is missing. Repeated rebuild attempts left many abandoned main.sqlite.tmp-* files, and after a successful CLI rebuild the in-session dynamic memory_search tool still served the stale paused state until manager/gateway reload.

This looks like a combined memory-core failure mode:

  1. interrupted rebuilds do not clean up temporary SQLite files;
  2. large/active indexes can keep re-dirtying and spawning full rebuild attempts;
  3. a successful on-disk rebuild does not invalidate the live memory-search manager's cached disabled/identity state.

Live repro/evidence from local agent

Environment:

  • OpenClaw: 2026.6.1 (2e08f0f)
  • OS: Linux x64
  • Agent: main
  • Memory config: provider openai, model text-embedding-3-large, sources memory,sessions

Before repair:

  • Live DB: /home/exedev/.openclaw/memory/main.sqlite
  • Size: about 2.3G
  • PRAGMA integrity_check: ok
  • chunks: 5462
  • embedding_cache: 34255
  • meta rows: 0
  • openclaw memory status --agent main reported:
    • Indexed: 412/2360 files · 5462 chunks
    • Dirty: yes
    • Index identity: index metadata is missing
    • Vector search: paused until memory is rebuilt

Tmp leak/thrash evidence:

  • Around 2026-06-04 21:46 AST, there were 10 abandoned main.sqlite.tmp-* DB families totaling roughly 40GB before cleanup.
  • One Codex-spawned openclaw-memory process had open FDs to main.sqlite.tmp-457c93d7-46cc-49e2-87d7-e0812ce8119b{,-wal,-shm} but file mtimes/sizes stayed unchanged during a 30s observation window after about 20 minutes of runtime. It looked stalled, not active.
  • After deleting only tmp files not held open, terminating the stale process with SIGTERM, and deleting its leftover tmp family, memory dir usage fell to about 3.2G after the clean rebuild.

Manual repair that succeeded on disk:

openclaw memory index --agent main --force

Post-repair CLI verification:

  • Command printed Memory index updated (main).
  • No main.sqlite.tmp-* files remained.
  • Live DB had meta row memory_index_meta_v1:
    • provider openai
    • model text-embedding-3-large
    • vectorDims 3072
  • chunks: 5522
  • embedding_cache: 34318
  • openclaw memory status --agent main --index --json reported:
    • dirty: false
    • vector.enabled: true
    • vector.available: true
    • custom.indexIdentity.status: valid
    • embedding probe ok: true
  • openclaw memory search --agent main --query 'Morningstar NHYDY fair value' --max-results 3 returned results.

Still broken in live manager/tool path after repair:

The dynamic in-session memory_search tool still returned:

{
  "disabled": true,
  "unavailable": true,
  "error": "index metadata is missing",
  "action": "Tell the user to run: openclaw memory status --index or openclaw memory index --force."
}

That was after the CLI status/index probe already saw the index as valid and active. This suggests the memory-search manager caches the disabled identity state and does not re-read the repaired index metadata.

Expected behavior

  • A failed/interrupted rebuild should remove its main.sqlite.tmp-* files, or mark them recoverable and clean them on next startup/status/index operation.
  • Only one rebuild should run per agent/index at a time; concurrent or repeated full rebuild attempts should be coalesced/locked.
  • After a successful rebuild and atomic swap, the live search manager should invalidate any cached disabled state and immediately observe the new memory_index_meta_v1 identity.
  • Large session-backed indexes should not repeatedly dirty themselves into full rebuild thrash.

Related issues

This seems related but not identical to:

The distinct repro here is the combined loop: large active main index + missing meta + repeated abandoned tmp rebuilds + CLI repair succeeds while dynamic memory_search remains paused until manager reload.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions