Skip to content

[Bug]:Intermittent memory_search "index metadata is missing" despite valid builtin memory index; likely search/reindex race. Locally hotfixed. #90361

Description

@AyitLabs

Bug type

Regression (worked before, now fails)

Beta release blocker

No

Summary

The first-class memory_search tool intermittently returns:

{
  "disabled": true,
  "unavailable": true,
  "error": "index metadata is missing"
}

This happens even though the durable builtin SQLite memory index appears valid via CLI/deep status, direct DB inspection, and fresh-manager checks.

The failure appears concurrency-sensitive. Single memory_search calls can return real semantic/vector hits, while parallel or closely overlapping searches produced index metadata is missing and one memory_search timed out after 15s.

We applied a local two-file mitigation/hotfix against the installed dist. After gateway restart, a single first-class memory_search again returned valid semantic/vector hits without the missing-metadata error.

This report is not a claim that the upstream fix is complete. It is a bug report with a local mitigation and suspected root cause, submitted so maintainers can confirm and implement the correct upstream fix.

### Steps to reproduce

1. Configure builtin memory search with OpenAI embeddings:

```json
{
  "agents": {
    "defaults": {
      "memorySearch": {
        "provider": "openai",
        "model": "text-embedding-3-small"
      }
    }
  }
}

    Confirm the memory index is valid:

openclaw memory status --deep --index --json

In my case this reported builtin memory with provider: "openai", model: "text-embedding-3-small", vector dims 1536, semanticAvailable: true, dirty: false, and custom.indexIdentity.status: "valid".

    Confirm single semantic searches can work. A single first-class memory_search can return real vector hits, e.g. vectorScore > 0 and textScore: 0.

    Trigger multiple/parallel or closely overlapping first-class memory_search calls.
    Observe that one or more live tool calls can return: invalid search results and failed tool calls.

Later single memory_search calls may succeed again, suggesting transient live manager state rather than permanent index/config corruption.

### Expected behavior

If the durable builtin memory DB has valid `memory_index_meta_v1` metadata and CLI/fresh-manager status reports `indexIdentity.status: "valid"`, the first-class `memory_search` tool should not return `index metadata is missing` solely because of transient async sync/reindex state.

Expected behavior would be one of:

- return semantic/vector search results from the stable existing index;
- report an honest transient state like `index rebuild in progress`;
- wait/serialize appropriately during reindex;
- or return a true mismatch/unavailable state only when the durable index identity is actually missing/mismatched.


### Actual behavior

The live first-class `memory_search` tool intermittently reports:

```json
{
  "disabled": true,
  "unavailable": true,
  "error": "index metadata is missing"
}

This occurred even though:

    openclaw memory status --deep --index --json reported custom.indexIdentity.status: "valid";
    the DB contained the expected memory_index_meta_v1 metadata;
    a fresh Node process using the installed manager could search successfully and reported valid index identity;
    single first-class memory_search calls could return real semantic/vector hits.

The failure became easier to reproduce with parallel/overlapping memory_search calls. A later single search could succeed again.



### OpenClaw version

2026.6.1 (2e08f0f)

### Operating system

Ubuntu Linux x64

### Install method

Global npm install 

### Model

openai/gpt-5.5

### Provider / routing chain

OpenAI API key profile; default OpenClaw routing; memory embeddings via openai/text-embedding-3-small

### Additional provider/model setup details

Agent runtime model was `openai/gpt-5.5`.

Memory search was configured through:

```json
{
  "agents": {
    "defaults": {
      "memorySearch": {
        "provider": "openai",
        "model": "text-embedding-3-small"
      }
    }
  }
}

There was no per-agent main override.

Effective memory config resolved to:

  • backend: builtin
  • sources: ["memory"]
  • store path: /home/<user>/.openclaw/memory/main.sqlite
  • vector search enabled
  • hybrid query enabled
  • vector weight: 0.7
  • text weight: 0.3
  • cache enabled

models.providers.openai was not explicitly set, but the effective memory config had OpenAI remote auth material available/redacted, and real semantic/vector hits were observed.

Logs, screenshots, and evidence

### Durable index appeared valid

CLI/deep status previously reported healthy builtin memory:


backend: builtin
provider: openai
model: text-embedding-3-small
dirty: false
vector.semanticAvailable: true
vector.dims: 1536
custom.indexIdentity.status: valid
embeddingProbe.ok: true


Direct DB inspection found the expected metadata row/table:


memory_index_meta_v1


A fresh Node process using the same installed manager could search successfully and reported `indexIdentity: valid` before/after.

A single first-class `memory_search` could return genuine semantic/vector hits, e.g.:


vectorScore > 0
textScore: 0


### Live failure observed

During live tool testing, parallel/overlapping `memory_search` calls produced:


{
  "disabled": true,
  "unavailable": true,
  "error": "index metadata is missing"
}


and once:


{
  "disabled": true,
  "unavailable": true,
  "error": "memory_search timed out after 15s"
}


Then later single searches could succeed again.

### Suspected source-level race

In OpenClaw `2026.6.1` dist, source tracing found:


manager-dZw31DAG2.js
  ~903   function resolveMemoryIndexIdentityState(params)
  ~2202  async runSafeReindex(params)
  ~2230  this.db = tempDb\n  ~3564  refreshIndexIdentityDirty(params)
  ~3577  async search(query, opts)
  ~3763  async sync(params)
  ~3863  status() {


In `tools-CcIzM8pz.js`, the memory-search wrapper does roughly:


rawResults = await activeMemory.manager.search(query, searchOptions);
pausedIndexIdentityReason = resolvePausedMemoryIndexIdentityReason(activeMemory.manager.status());
if (pausedIndexIdentityReason) return;


There is a similar post-force-sync status check.

Hypothesis:

1. `memory_search` starts.
2. `MemoryIndexManager.search()` calls `startAsyncSearchSync()` before the current search finishes.
3. Async sync may enter `runSafeReindex()`.
4. `runSafeReindex()` temporarily assigns the shared live manager DB handle to a temp DB:


this.db = tempDb;
this.resetVectorState();
this.fts.available = false;
this.fts.loadError = void 0;
this.ensureSchema();


5. The temp DB has schema but may not yet have `memory_index_meta_v1`.
6. A concurrent search or post-search `status()` call reads identity from the temp DB rather than the durable main DB.
7. `status()` reports `index metadata is missing`.
8. The tool wrapper treats that as a hard unavailable state even though the durable main DB is valid.

### Local mitigation applied

I applied a local two-file dist hotfix, then restarted gateway.

Patched files:


/home/<user>/.npm-global/lib/node_modules/openclaw/dist/tools-CcIzM8pz.js
/home/<user>/.npm-global/lib/node_modules/openclaw/dist/manager-dZw31DAG2.js


Mitigation behavior:

1. Tool wrapper: if `manager.search()` returns nonempty hits and the only post-search pause reason is exactly `index metadata is missing`, preserve the hits and add debug warning instead of returning disabled/unavailable. True mismatches still block.

2. Memory manager: defer `startAsyncSearchSync()` until after the current search finishes, so a search is less likely to trigger reindex under its own feet.

`node --check` passed on both patched files before restart.

After gateway restart, a single first-class `memory_search` returned semantic/vector hits again with no missing-metadata error:


provider: openai
model: text-embedding-3-small
backend: builtin
hits: 5
vectorScore: 0.6357942223548889
textScore: 0


### Beta check

I inspected `[email protected]` from npm without installing it. The same suspicious patterns appear present:


package/dist/manager-B5STWg-k.js
  2229: async runSafeReindex(params)
  2257: this.db = tempDb
  3605: async search(query, opts)
  3626: startAsyncSearchSync({


and:


package/dist/tools-DUysl7tx.js
  rawResults = await activeMemory.manager.search(...)
  pausedIndexIdentityReason = resolvePausedMemoryIndexIdentityReason(activeMemory.manager.status())


So I do not see evidence that `2026.6.2-beta.1` has fixed this specific suspected race. So I'm premptively offering this fix to the openclaw team to improve the reliability of the native memory search function.

Impact and severity

Affects semantic memory reliability for builtin memory search.

Observed impact:

  • first-class memory_search can become unavailable even when the durable index is valid;
  • tool output suggests reindexing despite CLI/DB evidence that the index is valid;
  • memory recall becomes intermittent and confusing;
  • agents may incorrectly believe memory retrieval is disabled/unavailable;
  • parallel or overlapping searches appear to increase failure likelihood.

Severity: moderate to high for users relying on semantic memory. It does not appear to corrupt the durable DB in my case, but it can make live memory search unreliable, misleading and to silently fail. This issue may have been happening for several versions and no one noticed because it's not a noisy failure. The search just quietly fails.

Additional information

Proposed upstream fix:

Minimal mitigation

In the memory-search tool wrapper:

  • do not discard nonempty search hits solely because the post-search status check reports index metadata is missing;
  • preserve true mismatch blocking for provider/model/settings mismatch;
  • include a debug warning when preserving hits under transient missing identity.

In the manager:

  • defer startAsyncSearchSync() until after the current search finishes;
  • keep search from triggering async reindex before its own provider/identity check and query are complete.

Better structural fix

Do not expose the safe-reindex temp DB through the shared live manager.

Instead of having runSafeReindex() assign:

this.db = tempDb;

build the temp index in isolated state/context while the live manager continues using the old stable DB. Only after the temp index is fully built and atomically swapped into place should the live manager reopen/switch to the new durable DB.

This would allow searches to continue using the old stable index during reindex, rather than seeing half-built temp metadata state.

Defensive status improvement

Expose explicit reindex state in status, e.g.:

{
  "custom": {
    "reindex": {
      "active": true,
      "reason": "search"
    }
  }
}

and avoid overwriting a previously valid indexIdentityState with a transient temp-DB missing state while reindex is active.

Better user-facing state would be:

index rebuild in progress

rather than:

index metadata is missing

when the durable main DB metadata is valid.

Suggested tests:

  1. Unit test: preserve nonempty hits when only post-search identity says index metadata is missing.
  2. Unit test: true provider/model/settings mismatch still hard-fails.
  3. Integration test: pause runSafeReindex() after this.db = tempDb; this.ensureSchema();, then call search/status concurrently.
  4. Integration test: parallel memory_search calls do not produce false disabled: true when durable DB metadata exists.
  5. Manual smoke test: after restart, single memory_search returns semantic hits with vectorScore > 0 and no false missing-metadata error.

In short, we've been able to fix this issue local and we wanted to offer those fixes to the openclaw team to help build a more reliable product. We've been deploying internal fixes like this to keep our installation running.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.bugSomething isn't workingclawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.regressionBehavior that previously worked and now fails

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions