fix(memory): populate FTS index when no embedding provider is configured by opriz · Pull Request #56473 · openclaw/openclaw

opriz · 2026-03-28T15:36:38Z

Summary

Problem: When no embedding provider is configured (no API key, no local model), syncMemoryFiles() and syncSessionFiles() returned early unconditionally — skipping not just embedding generation but also FTS indexing. The FTS5 table (chunks_fts) was never populated, even though FTS5 is a built-in SQLite feature that requires no external dependency whatsoever. The default search path already includes a FTS-only branch that calls searchKeyword() when no provider is available, but because the index was never built, it always returned empty results.
Why it matters: This is the default state for most new users. The Memory Recall instruction is injected into the system prompt regardless, so the Agent calls memory_search and gets nothing back — effective amnesia with no warning.
What changed: Removed the provider-null early returns from the sync path. indexFile() now has a dedicated FTS-only branch that chunks files and writes to chunks + chunks_fts (model sentinel "fts-only", empty embeddings, no vector writes). Also fixed needsFullReindex to detect provider→FTS-only transitions so orphaned old-model FTS rows are cleaned up on the next sync. Extracted a writeChunks() helper to eliminate duplicated three-table write logic between the two paths.
What did NOT change: Configuration semantics, hybrid/FTS enablement flags, vector indexing behavior, session transcript handling, multimodal file handling (still requires a provider).

Change Type (select all)

Bug fix
Refactor required for the fix

Scope (select all touched areas)

Memory / storage

Linked Issue/PR

Closes [Bug]: FTS-only memory search is completely broken — 0 chunks indexed when no embedding provider is configured #26792
Closes memory search returns 0 results in FTS-only mode (no embedding provider) #51782
Related Support FTS-only memory indexing (skip embedding provider) #43957
Related fix(memory): support FTS-only indexing without embedding provider #44094
This PR fixes a bug or regression

Root Cause / Regression History (if applicable)

Root cause: Three if (!this.provider) return; guards in syncMemoryFiles(), syncSessionFiles(), and indexFile() were intended to skip embedding generation but inadvertently skipped FTS writes too. The comment said "FTS-only mode" but the code skipped FTS as well.
Missing detection / guardrail: No test covered the provider-null sync path end-to-end.
Prior context: The search path (manager.ts:346-391) already had a correct FTS-only branch calling searchKeyword() — but the index was always empty, so it returned nothing.
Why this regressed now: The guards appear to have been introduced when embedding providers were first added, with the assumption that FTS was only useful alongside embeddings.

Regression Test Plan (if applicable)

Coverage level:
- Unit test
- Seam / integration test
Target test: extensions/memory-core/src/memory/index.test.ts
Scenarios locked in:
1. "builds FTS index and returns search results when no embedding provider is available" — verifies chunks > 0, embedBatchCalls === 0, keyword search returns results, unknown terms return empty.
2. "triggers full reindex and cleans up old-model FTS rows when switching from provider to FTS-only" — verifies old model = "mock-embed" FTS rows are removed and new model = "fts-only" rows exist after transition.

User-visible / Behavior Changes

Users without an embedding provider now get working keyword search via FTS5 instead of silent empty results.
First sync after removing a provider triggers a full reindex (cleans up old embedding-model FTS rows).
openclaw memory status will show chunks > 0 and FTS: ready in provider-less setups.

Diagram (if applicable)

Before (no provider):
  sync() → syncMemoryFiles() → !this.provider → return  (chunks_fts: never written)
  search("keyword") → FTS-only branch → searchKeyword() → 0 results (index empty)

After (no provider):
  sync() → syncMemoryFiles() → indexFile() → FTS-only path
         → chunkMarkdown() → writeChunks("fts-only", [], false)
         → chunks_fts populated
  search("keyword") → FTS-only branch → searchKeyword() → results returned

Security Impact (required)

New permissions/capabilities? No
Secrets/tokens handling changed? No
New/changed network calls? No
Command/tool execution surface changed? No
Data access scope changed? No

Repro + Verification

Environment

OS: macOS 15.3
Runtime: Node 22 / Bun
Model/provider: none (openrouter configured, no active API key)
Relevant config: agents.defaults.memorySearch.provider = "auto", query.hybrid.enabled = true

Steps

Ensure no embedding provider API key is set (provider: "auto" with no keys)
Run openclaw memory index --force
Run openclaw memory status
Run openclaw memory search "<keyword>" --min-score 0

Expected

status shows Indexed: N files · M chunks, FTS: ready
search returns matching results

Actual (before fix)

status shows Indexed: 0/N files · 0 chunks
search returns No matches

Actual (after fix)

status shows Indexed: 6/6 files · 35 chunks, FTS: ready
search --min-score 0 returns matching chunks

Evidence

Failing test/log before + passing after

# After fix
npx vitest run extensions/memory-core/src/memory/index.test.ts
Test Files  1 passed (1)
Tests       24 passed | 3 skipped (27)

Human Verification (required)

Verified scenarios: ran openclaw memory index + openclaw memory status + openclaw memory search on a local instance with Provider: none (requested: auto) — confirmed 35 chunks indexed and keyword search returning results with --min-score 0.
Edge cases checked: provider→FTS-only transition (old rows cleaned up), FTS disabled case (fast-exit preserved), multimodal files (still skipped without provider).
What I did not verify: session transcript FTS-only indexing, Windows FTS5 availability, QMD backend interaction.

Review Conversations

I replied to or resolved every bot review conversation I addressed in this PR.
I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

Backward compatible? Yes
Config/env changes? No
Migration needed? No

Risks and Mitigations

Risk: First sync after provider removal triggers a full reindex, which may be slow for large workspaces.
- Mitigation: Full reindex only fires once (meta is updated after); subsequent syncs are incremental.
Risk: FTS-only BM25 scores are very small (~0.000002) and get filtered by the default minScore = 0.35.
- Mitigation: Known follow-up issue; workaround is --min-score 0. Not introduced by this PR — the search path already existed, just had no data.

greptile-apps · 2026-03-28T15:46:42Z

Greptile Summary

This PR correctly fixes the core bug where FTS indexing was entirely skipped when no embedding provider was configured. It removes the provider-null early returns from syncMemoryFiles and syncSessionFiles, adds a dedicated FTS-only branch in indexFile, extracts a writeChunks helper to eliminate duplicated write logic, and extends needsFullReindex to detect provider→FTS-only transitions. The overall approach is sound and the two new integration tests validate the main scenarios.

One P1 regression was found:

clearIndexedFileData (line 556) still guards FTS row deletion with && this.provider. In FTS-only mode during an incremental sync, when a file's content changes, the old chunks_fts rows for that file are never deleted before new ones are inserted. The chunks table is cleared correctly (line 563), but chunks_fts is left with orphaned rows from the old content. Over time (or after a single file edit) this causes stale FTS rows to accumulate, and keyword searches can return results pointing at deleted or outdated text. The needsFullReindex guard only protects the very first sync after a provider transition — subsequent incremental syncs will exhibit this issue.

The stale-file deletion in syncMemoryFiles/syncSessionFiles already uses the correct this.provider?.model ?? \"fts-only\" pattern; clearIndexedFileData needs the same treatment.

Confidence Score: 4/5

Core FTS-only indexing fix is correct, but a P1 regression causes stale FTS rows to accumulate on incremental file re-indexing in FTS-only mode — should be addressed before merging.

The primary fix is well-reasoned and the new reindex-transition detection is correct. However, clearIndexedFileData retains a && this.provider guard that prevents FTS cleanup during incremental re-indexing in FTS-only mode, resulting in stale rows in chunks_fts that will cause incorrect search results after any file modification in a provider-less setup. This is a real defect on the changed path that is not covered by the new tests.

extensions/memory-core/src/memory/manager-embedding-ops.ts — specifically the clearIndexedFileData method at line 556.

Important Files Changed

Filename	Overview
extensions/memory-core/src/memory/manager-embedding-ops.ts	Extracts a writeChunks helper and adds a correct FTS-only branch to indexFile. However, clearIndexedFileData still guards FTS row deletion with && this.provider, which will cause stale FTS rows to accumulate when files are modified and re-indexed incrementally in FTS-only mode.
extensions/memory-core/src/memory/manager-sync-ops.ts	Removes the early-return provider guards from syncMemoryFiles and syncSessionFiles, fixes stale-file FTS deletion to use this.provider?.model ?? 'fts-only', and adds needsFullReindex detection for provider→FTS-only transitions. Logic looks correct.
extensions/memory-core/src/memory/index.test.ts	Adds two well-structured integration tests covering the FTS-only indexing path and the provider→FTS-only reindex transition. Coverage is good, though the incremental-sync-after-file-modification path in FTS-only mode is not covered.

Comments Outside Diff (1)

extensions/memory-core/src/memory/manager-embedding-ops.ts, line 556-562 (link)

Stale FTS rows not deleted on incremental re-index in FTS-only mode

clearIndexedFileData guards FTS deletion with && this.provider. In FTS-only mode (this.provider === null), when a file's content changes and is re-indexed incrementally (needsFullReindex = false), the old FTS rows are silently left in chunks_fts before new ones are inserted.

The sequence:

File indexed → FTS rows written with chunk IDs derived from the old content hash.
File modified → next incremental sync calls writeChunks → clearIndexedFileData is called.
clearIndexedFileData skips FTS deletion (guard && this.provider is false), but does delete from the chunks table.
New chunks + new FTS rows are inserted.
chunks_fts now contains both the orphaned old rows (no matching chunks entry) and the new rows — subsequent keyword searches can return results pointing at deleted or outdated content.

The fix is to drop the this.provider guard and use this.provider?.model ?? "fts-only" as the model value, consistent with how stale-file deletion already works in syncMemoryFiles/syncSessionFiles:

if (this.fts.enabled && this.fts.available) {
  const ftsModel = this.provider?.model ?? "fts-only";
  try {
    this.db
      .prepare(`DELETE FROM ${FTS_TABLE} WHERE path = ? AND source = ? AND model = ?`)
      .run(pathname, source, ftsModel);
  } catch {}
}

Prompt To Fix With AI

This is a comment left during a code review.
Path: extensions/memory-core/src/memory/manager-embedding-ops.ts
Line: 556-562

Comment:
**Stale FTS rows not deleted on incremental re-index in FTS-only mode**

`clearIndexedFileData` guards FTS deletion with `&& this.provider`. In FTS-only mode (`this.provider === null`), when a file's content changes and is re-indexed incrementally (`needsFullReindex = false`), the old FTS rows are silently left in `chunks_fts` before new ones are inserted.

The sequence:
1. File indexed → FTS rows written with chunk IDs derived from the old content hash.
2. File modified → next incremental sync calls `writeChunks` → `clearIndexedFileData` is called.
3. `clearIndexedFileData` **skips** FTS deletion (guard `&& this.provider` is false), but **does** delete from the `chunks` table.
4. New chunks + new FTS rows are inserted.
5. `chunks_fts` now contains both the orphaned old rows (no matching `chunks` entry) and the new rows — subsequent keyword searches can return results pointing at deleted or outdated content.

The fix is to drop the `this.provider` guard and use `this.provider?.model ?? "fts-only"` as the model value, consistent with how stale-file deletion already works in `syncMemoryFiles`/`syncSessionFiles`:

```typescript
if (this.fts.enabled && this.fts.available) {
  const ftsModel = this.provider?.model ?? "fts-only";
  try {
    this.db
      .prepare(`DELETE FROM ${FTS_TABLE} WHERE path = ? AND source = ? AND model = ?`)
      .run(pathname, source, ftsModel);
  } catch {}
}
```

How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

This is a comment left during a code review.
Path: extensions/memory-core/src/memory/manager-embedding-ops.ts
Line: 556-562

Comment:
**Stale FTS rows not deleted on incremental re-index in FTS-only mode**

`clearIndexedFileData` guards FTS deletion with `&& this.provider`. In FTS-only mode (`this.provider === null`), when a file's content changes and is re-indexed incrementally (`needsFullReindex = false`), the old FTS rows are silently left in `chunks_fts` before new ones are inserted.

The sequence:
1. File indexed → FTS rows written with chunk IDs derived from the old content hash.
2. File modified → next incremental sync calls `writeChunks` → `clearIndexedFileData` is called.
3. `clearIndexedFileData` **skips** FTS deletion (guard `&& this.provider` is false), but **does** delete from the `chunks` table.
4. New chunks + new FTS rows are inserted.
5. `chunks_fts` now contains both the orphaned old rows (no matching `chunks` entry) and the new rows — subsequent keyword searches can return results pointing at deleted or outdated content.

The fix is to drop the `this.provider` guard and use `this.provider?.model ?? "fts-only"` as the model value, consistent with how stale-file deletion already works in `syncMemoryFiles`/`syncSessionFiles`:

```typescript
if (this.fts.enabled && this.fts.available) {
  const ftsModel = this.provider?.model ?? "fts-only";
  try {
    this.db
      .prepare(`DELETE FROM ${FTS_TABLE} WHERE path = ? AND source = ? AND model = ?`)
      .run(pathname, source, ftsModel);
  } catch {}
}
```

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (1): Last reviewed commit: "fix(memory): trigger full reindex on pro..." | Re-trigger Greptile}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 82a8626875

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2a42cbb3b2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

obviyus · 2026-03-29T06:31:02Z

Addressed the default-threshold gap in provider-less search.

Change:

FTS-only search now uses the same scored-result fallback helper as the hybrid keyword-only path, so exact BM25 hits are not dropped just because the default minScore (0.35) is tuned for vector/hybrid scores.
Added regression coverage in extensions/memory-core/src/memory/index.test.ts to exercise the no-provider path with minScore: 0.35 instead of 0.

Verification:

pnpm test -- extensions/memory-core/src/memory/index.test.ts

@opriz

…(thanks @opriz)

obviyus

Reviewed latest changes; landing now.

obviyus · 2026-03-29T06:52:53Z

Landed on main.

Landed source commit: c6eb4bf
Merge commit: 41c30f0

Thanks @opriz.

@opriz

… (thanks @opriz) * fix(memory): build FTS index when no embedding provider is available * fix(memory): trigger full reindex on provider→FTS-only transition * fix(memory): return FTS-only keyword hits at default threshold * fix: keep FTS-only memory hits at default threshold (openclaw#56473) (thanks @opriz) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

@opriz

… (thanks @opriz) * fix(memory): build FTS index when no embedding provider is available * fix(memory): trigger full reindex on provider→FTS-only transition * fix(memory): return FTS-only keyword hits at default threshold * fix: keep FTS-only memory hits at default threshold (openclaw#56473) (thanks @opriz) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

@opriz

… (thanks @opriz) * fix(memory): build FTS index when no embedding provider is available * fix(memory): trigger full reindex on provider→FTS-only transition * fix(memory): return FTS-only keyword hits at default threshold * fix: keep FTS-only memory hits at default threshold (openclaw#56473) (thanks @opriz) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

@opriz

… (thanks @opriz) * fix(memory): build FTS index when no embedding provider is available * fix(memory): trigger full reindex on provider→FTS-only transition * fix(memory): return FTS-only keyword hits at default threshold * fix: keep FTS-only memory hits at default threshold (openclaw#56473) (thanks @opriz) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

@opriz

… (thanks @opriz) * fix(memory): build FTS index when no embedding provider is available * fix(memory): trigger full reindex on provider→FTS-only transition * fix(memory): return FTS-only keyword hits at default threshold * fix: keep FTS-only memory hits at default threshold (openclaw#56473) (thanks @opriz) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

@opriz

… (thanks @opriz) * fix(memory): build FTS index when no embedding provider is available * fix(memory): trigger full reindex on provider→FTS-only transition * fix(memory): return FTS-only keyword hits at default threshold * fix: keep FTS-only memory hits at default threshold (openclaw#56473) (thanks @opriz) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

@opriz

… (thanks @opriz) * fix(memory): build FTS index when no embedding provider is available * fix(memory): trigger full reindex on provider→FTS-only transition * fix(memory): return FTS-only keyword hits at default threshold * fix: keep FTS-only memory hits at default threshold (openclaw#56473) (thanks @opriz) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

@opriz

… (thanks @opriz) * fix(memory): build FTS index when no embedding provider is available * fix(memory): trigger full reindex on provider→FTS-only transition * fix(memory): return FTS-only keyword hits at default threshold * fix: keep FTS-only memory hits at default threshold (openclaw#56473) (thanks @opriz) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

@opriz

… (thanks @opriz) * fix(memory): build FTS index when no embedding provider is available * fix(memory): trigger full reindex on provider→FTS-only transition * fix(memory): return FTS-only keyword hits at default threshold * fix: keep FTS-only memory hits at default threshold (openclaw#56473) (thanks @opriz) --------- Co-authored-by: Ayaan Zaidi <[email protected]>

openclaw-barnacle Bot added extensions: memory-core Extension: memory-core size: M labels Mar 28, 2026

opriz marked this pull request as ready for review March 28, 2026 15:43

chatgpt-codex-connector Bot reviewed Mar 28, 2026

View reviewed changes

Comment thread extensions/memory-core/src/memory/manager-embedding-ops.ts

opriz force-pushed the fix/memory-fts-sync-without-provider branch from 82a8626 to 2a42cbb Compare March 28, 2026 15:51

chatgpt-codex-connector Bot reviewed Mar 28, 2026

View reviewed changes

Comment thread extensions/memory-core/src/memory/manager-embedding-ops.ts Outdated

opriz force-pushed the fix/memory-fts-sync-without-provider branch 2 times, most recently from 8e4e7e7 to 2113520 Compare March 28, 2026 16:05

opriz changed the title ~~fix(memory): build FTS index and trigger full reindex on provider→FTS-only transition~~ fix(memory): populate FTS index when no embedding provider is configured Mar 28, 2026

obviyus force-pushed the fix/memory-fts-sync-without-provider branch from 2113520 to 896b655 Compare March 29, 2026 06:30

obviyus self-assigned this Mar 29, 2026

opriz and others added 4 commits March 29, 2026 12:19

fix(memory): build FTS index when no embedding provider is available

677e84c

fix(memory): trigger full reindex on provider→FTS-only transition

8fbf84f

fix(memory): return FTS-only keyword hits at default threshold

18b0f1b

fix: keep FTS-only memory hits at default threshold (openclaw#56473) …

c6eb4bf

…(thanks @opriz)

obviyus force-pushed the fix/memory-fts-sync-without-provider branch from 896b655 to c6eb4bf Compare March 29, 2026 06:50

obviyus approved these changes Mar 29, 2026

View reviewed changes

obviyus merged commit 41c30f0 into openclaw:main Mar 29, 2026
34 of 35 checks passed

github-actions Bot mentioned this pull request Mar 29, 2026

📡 Upstream Digest — 2026-03-29 06:55 UTC curtismercier/openclaw-mods#398

Open

This was referenced Mar 29, 2026

fix(memory): support FTS-only indexing without embedding provider #44094

Closed

fix(memory): index and search in FTS-only mode (provider=none) #39255

Closed

Support FTS-only memory indexing (skip embedding provider) #43957

Closed

opriz deleted the fix/memory-fts-sync-without-provider branch March 29, 2026 13:02

clawsweeper Bot mentioned this pull request May 1, 2026

Misleading "sqlite-vec unavailable" warning when sqlite-vec actually loads successfully #75624

Closed

This was referenced May 11, 2026

memory: preserve keyword fallback hits after merge-time scoring #48112

Closed

fix(memory): remove provider model filter from FTS keyword search #48328

Closed

Bug: memory_search hybrid mode not returning FTS matches #48300

Closed

clawsweeper Bot mentioned this pull request Jun 15, 2026

fix(memory): use per-keyword FTS search in hybrid mode #39484 #73976

Merged

clawsweeper Bot mentioned this pull request Jun 21, 2026

fix(memory-core): include path in FTS indexed text so filename tokens are searchable #94114

Closed

clawsweeper Bot mentioned this pull request Jun 29, 2026

fix(memory): stop chunkMarkdown injecting newlines into single-line chunks #96747

Closed

clawsweeper Bot mentioned this pull request Jul 16, 2026

fix(memory): use FTS-only when optional embeddings are unavailable #108229

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(memory): populate FTS index when no embedding provider is configured#56473

fix(memory): populate FTS index when no embedding provider is configured#56473
obviyus merged 4 commits into
openclaw:mainfrom
opriz:fix/memory-fts-sync-without-provider

opriz commented Mar 28, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Mar 28, 2026 •

edited

Loading

Important Files Changed

Comments Outside Diff (1)

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

obviyus commented Mar 29, 2026

Uh oh!

obviyus left a comment

Uh oh!

Uh oh!

obviyus commented Mar 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

opriz commented Mar 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Change Type (select all)

Scope (select all touched areas)

Linked Issue/PR

Root Cause / Regression History (if applicable)

Regression Test Plan (if applicable)

User-visible / Behavior Changes

Diagram (if applicable)

Security Impact (required)

Repro + Verification

Environment

Steps

Expected

Actual (before fix)

Actual (after fix)

Evidence

Human Verification (required)

Review Conversations

Compatibility / Migration

Risks and Mitigations

Uh oh!

greptile-apps Bot commented Mar 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Comments Outside Diff (1)

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

obviyus commented Mar 29, 2026

Uh oh!

obviyus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

obviyus commented Mar 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

opriz commented Mar 28, 2026 •

edited

Loading

greptile-apps Bot commented Mar 28, 2026 •

edited

Loading