Skip to content

fix(memory): support FTS-only indexing without embedding provider#44094

Closed
Haohao-end wants to merge 3 commits into
openclaw:mainfrom
Haohao-end:feat/43957-fts-only-memory-indexing
Closed

fix(memory): support FTS-only indexing without embedding provider#44094
Haohao-end wants to merge 3 commits into
openclaw:mainfrom
Haohao-end:feat/43957-fts-only-memory-indexing

Conversation

@Haohao-end

@Haohao-end Haohao-end commented Mar 12, 2026

Copy link
Copy Markdown

Summary

This PR fixes memory indexing when no embedding provider is configured.

Previously, the indexing pipeline bailed out early in provider-null mode, which prevented:

  • memory file scanning
  • session transcript scanning
  • chunk persistence
  • FTS row insertion

As a result, memory search could report FTS-only mode while still returning no indexed results.

This follow-up also fixes cleanup behavior when transitioning from provider-backed indexing to FTS-only mode, so old-model FTS rows do not remain behind and cause duplicate search results. It also skips provider-less sync work when FTS is disabled or unavailable.

What changed

  • removed provider-null early returns from:
    • syncMemoryFiles()
    • syncSessionFiles()
  • updated indexFile() so text chunking and persistence always happen
  • made embedding generation and vector writes conditional on provider availability
  • added a stable provider-less model sentinel: "fts-only"
  • made FTS cleanup path/source-based in:
    • clearIndexedFileData()
    • stale-row cleanup in syncMemoryFiles()
    • stale-row cleanup in syncSessionFiles()
  • added a fast-exit for provider-less sync when FTS is disabled or unavailable, to avoid unnecessary scanning and indexing work
  • added focused tests for:
    • markdown memory indexing in FTS-only mode
    • session transcript indexing in FTS-only mode
    • stale-row cleanup on resync
    • provider-backed regression sanity
    • provider-backed -> FTS-only transition cleanup, ensuring old-model FTS rows are removed and search results do not duplicate

Scope

This patch intentionally preserves the existing configuration semantics around FTS/hybrid enablement. It fixes the broken provider-null indexing path when FTS-backed search is enabled, but does not change whether FTS should be auto-enabled when query.hybrid.enabled is false.

Validation

Passed locally:

  • pnpm exec vitest run src/memory/manager.fts-only.test.ts
  • pnpm exec vitest run src/memory/index.test.ts
  • pnpm build
  • pnpm check

Closes #43957

@greptile-apps

greptile-apps Bot commented Mar 12, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes the FTS-only memory indexing pipeline by removing early !provider exits from syncMemoryFiles, syncSessionFiles, and indexFile, allowing file scanning, chunk persistence, and FTS row insertion to proceed even without an embedding provider. It introduces "fts-only" as a sentinel model name for chunks and FTS rows, and makes embedding generation and vector writes conditional on provider availability.

Key issues found:

  • Orphaned FTS rows on provider-to-FTS-only model transition (manager-embedding-ops.ts line 771, manager-sync-ops.ts lines ~741 and ~843): clearIndexedFileData and both stale-row cleanup loops delete chunks without a model filter (DELETE FROM chunks WHERE path = ? AND source = ?) but only delete FTS rows for currentModel ("fts-only" when no provider is set). If a file was previously indexed under a real provider model (e.g. "text-embedding-3-small") and then the provider is removed, re-indexing or stale cleanup will leave the old-model FTS rows in place. Since searchKeyword searches all FTS models when providerModel is undefined, these orphaned rows will surface as duplicate results. The fix is to drop the model predicate in the FTS DELETE statements, mirroring how chunks are cleaned up.

  • No test coverage for provider → FTS-only transition: The new test suite (manager.fts-only.test.ts) tests fresh FTS-only indexing and stale cleanup within FTS-only mode, but does not cover the scenario where an existing provider-backed index is reopened in FTS-only mode and then re-synced. Adding this test case would catch the orphaned-rows regression described above.

Confidence Score: 3/5

  • Safe to merge for users running FTS-only from a clean index; risk exists for users transitioning from provider-backed indexing to FTS-only mode.
  • The core fix is correct and well-tested for the happy path (fresh FTS-only index). However, the model-scoped FTS cleanup in clearIndexedFileData and the two stale-row loops creates a real data-consistency bug for users who previously had a provider configured: old FTS rows under the provider's model name will accumulate and pollute search results indefinitely when operating in FTS-only mode. This is not caught by the new tests, reducing confidence.
  • The FTS DELETE statements in clearIndexedFileData (manager-embedding-ops.ts ~line 775) and the stale cleanup loops in both syncMemoryFiles and syncSessionFiles (manager-sync-ops.ts ~lines 741 and 843) all need the model predicate removed.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: src/memory/manager-embedding-ops.ts
Line: 771-778

Comment:
**Orphaned FTS rows when model transitions across syncs**

`clearIndexedFileData` deletes `chunks` rows without any model filter (`DELETE FROM chunks WHERE path = ? AND source = ?`), but the FTS cleanup only removes rows matching `currentModel`. If a user previously indexed with a real embedding provider (say `model = "text-embedding-3-small"`) and then removes the provider, the next resync/re-index of a changed file will:

1. Delete ALL chunks for that path (no model filter) ✓
2. Try to delete FTS rows with `model = "fts-only"` — nothing to delete yet
3. Insert new chunks and FTS rows with `model = "fts-only"`4. **Leave the old `model = "text-embedding-3-small"` FTS rows untouched** ✗

Since `searchKeyword` searches ALL models when `providerModel` is `undefined` (FTS-only mode — see `manager-search.ts` line 155–156: `const modelClause = params.providerModel ? " AND model = ?" : ""`), those orphaned rows will be returned in future searches, causing duplicate results for the same content.

The same issue affects the stale-row cleanup loops in `syncMemoryFiles` (around line 741 in `manager-sync-ops.ts`) and `syncSessionFiles` (around line 843 in `manager-sync-ops.ts`), both of which also only delete FTS rows for `currentModel` while deleting chunks without a model filter.

The fix for all three sites is to drop the `model` predicate when deleting from the FTS table so it matches the model-agnostic `chunks` cleanup:

```suggestion
    const currentModel = this.provider?.model ?? "fts-only";
    if (this.fts.enabled && this.fts.available) {
      try {
        this.db
          .prepare(`DELETE FROM ${FTS_TABLE} WHERE path = ? AND source = ?`)
          .run(pathname, source);
      } catch {}
    }
```

The same one-liner change should be applied to the FTS `DELETE` in `syncMemoryFiles` (line ~741) and `syncSessionFiles` (line ~843) in `manager-sync-ops.ts`.

How can I resolve this? If you propose a fix, please make it concise.

Last reviewed commit: 4eae898

Comment thread src/memory/manager-embedding-ops.ts Outdated
Comment on lines 771 to 778
const currentModel = this.provider?.model ?? "fts-only";
if (this.fts.enabled && this.fts.available) {
try {
this.db
.prepare(`DELETE FROM ${FTS_TABLE} WHERE path = ? AND source = ? AND model = ?`)
.run(pathname, source, this.provider.model);
.run(pathname, source, currentModel);
} catch {}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orphaned FTS rows when model transitions across syncs

clearIndexedFileData deletes chunks rows without any model filter (DELETE FROM chunks WHERE path = ? AND source = ?), but the FTS cleanup only removes rows matching currentModel. If a user previously indexed with a real embedding provider (say model = "text-embedding-3-small") and then removes the provider, the next resync/re-index of a changed file will:

  1. Delete ALL chunks for that path (no model filter) ✓
  2. Try to delete FTS rows with model = "fts-only" — nothing to delete yet
  3. Insert new chunks and FTS rows with model = "fts-only"
  4. Leave the old model = "text-embedding-3-small" FTS rows untouched

Since searchKeyword searches ALL models when providerModel is undefined (FTS-only mode — see manager-search.ts line 155–156: const modelClause = params.providerModel ? " AND model = ?" : ""), those orphaned rows will be returned in future searches, causing duplicate results for the same content.

The same issue affects the stale-row cleanup loops in syncMemoryFiles (around line 741 in manager-sync-ops.ts) and syncSessionFiles (around line 843 in manager-sync-ops.ts), both of which also only delete FTS rows for currentModel while deleting chunks without a model filter.

The fix for all three sites is to drop the model predicate when deleting from the FTS table so it matches the model-agnostic chunks cleanup:

Suggested change
const currentModel = this.provider?.model ?? "fts-only";
if (this.fts.enabled && this.fts.available) {
try {
this.db
.prepare(`DELETE FROM ${FTS_TABLE} WHERE path = ? AND source = ? AND model = ?`)
.run(pathname, source, this.provider.model);
.run(pathname, source, currentModel);
} catch {}
}
const currentModel = this.provider?.model ?? "fts-only";
if (this.fts.enabled && this.fts.available) {
try {
this.db
.prepare(`DELETE FROM ${FTS_TABLE} WHERE path = ? AND source = ?`)
.run(pathname, source);
} catch {}
}

The same one-liner change should be applied to the FTS DELETE in syncMemoryFiles (line ~741) and syncSessionFiles (line ~843) in manager-sync-ops.ts.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/memory/manager-embedding-ops.ts
Line: 771-778

Comment:
**Orphaned FTS rows when model transitions across syncs**

`clearIndexedFileData` deletes `chunks` rows without any model filter (`DELETE FROM chunks WHERE path = ? AND source = ?`), but the FTS cleanup only removes rows matching `currentModel`. If a user previously indexed with a real embedding provider (say `model = "text-embedding-3-small"`) and then removes the provider, the next resync/re-index of a changed file will:

1. Delete ALL chunks for that path (no model filter) ✓
2. Try to delete FTS rows with `model = "fts-only"` — nothing to delete yet
3. Insert new chunks and FTS rows with `model = "fts-only"`4. **Leave the old `model = "text-embedding-3-small"` FTS rows untouched** ✗

Since `searchKeyword` searches ALL models when `providerModel` is `undefined` (FTS-only mode — see `manager-search.ts` line 155–156: `const modelClause = params.providerModel ? " AND model = ?" : ""`), those orphaned rows will be returned in future searches, causing duplicate results for the same content.

The same issue affects the stale-row cleanup loops in `syncMemoryFiles` (around line 741 in `manager-sync-ops.ts`) and `syncSessionFiles` (around line 843 in `manager-sync-ops.ts`), both of which also only delete FTS rows for `currentModel` while deleting chunks without a model filter.

The fix for all three sites is to drop the `model` predicate when deleting from the FTS table so it matches the model-agnostic `chunks` cleanup:

```suggestion
    const currentModel = this.provider?.model ?? "fts-only";
    if (this.fts.enabled && this.fts.available) {
      try {
        this.db
          .prepare(`DELETE FROM ${FTS_TABLE} WHERE path = ? AND source = ?`)
          .run(pathname, source);
      } catch {}
    }
```

The same one-liner change should be applied to the FTS `DELETE` in `syncMemoryFiles` (line ~741) and `syncSessionFiles` (line ~843) in `manager-sync-ops.ts`.

How can I resolve this? If you propose a fix, please make it concise.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4eae898336

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

return;
}

const files = await listMemoryFiles(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Skip providerless indexing when FTS is disabled

Add a guard for !this.provider && !this.fts.enabled before starting file scans here (and in syncSessionFiles). With this change, providerless setups still read/chunk/write chunks rows even when query.hybrid.enabled=false, but those rows are never searchable because search() returns early when no provider and FTS is disabled (src/memory/manager.ts:285-287). This creates unnecessary sync latency and disk growth for configurations that intentionally turned hybrid/FTS off.

Useful? React with 👍 / 👎.

@Haohao-end

Copy link
Copy Markdown
Author

Good catch — I pushed a follow-up fix.

Changes included:

  • made FTS cleanup model-agnostic in clearIndexedFileData()
  • made stale-row cleanup model-agnostic in both syncMemoryFiles() and syncSessionFiles()
  • added a provider-backed -> FTS-only transition regression test to verify old-model FTS rows are removed and search results do not duplicate
  • added a small fast-exit for provider-less sync when FTS is disabled

Validated locally with:

  • pnpm exec vitest run src/memory/manager.fts-only.test.ts
  • pnpm exec vitest run src/memory/index.test.ts
  • pnpm build
  • pnpm check

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5b046b2122

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/memory/manager-sync-ops.ts Outdated
Comment on lines 666 to 667
if (!this.provider && !this.fts.enabled) {
return;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Guard providerless sync on FTS availability

When no embedding provider is configured, this condition still runs indexing whenever query.hybrid.enabled is true, even if FTS failed to initialize (fts.available === false, e.g. a SQLite build without FTS5). In that state indexFile() writes chunks rows but never inserts chunks_fts rows, and providerless search() returns early with no results when FTS is unavailable, so sync work and disk usage grow for data that can never be queried; the same guard pattern is also present in syncSessionFiles.

Useful? React with 👍 / 👎.

@Haohao-end

Copy link
Copy Markdown
Author

Pushed a small follow-up update.

I tightened the provider-less sync fast-exit in syncMemoryFiles() and syncSessionFiles() to skip traversal when FTS is either disabled or unavailable, so provider-less setups do not scan/chunk/write rows that can never be queried.

Validated locally with:

  • pnpm exec vitest run src/memory/manager.fts-only.test.ts
  • pnpm exec vitest run src/memory/index.test.ts
  • pnpm build
  • pnpm check

@obviyus

obviyus commented Mar 29, 2026

Copy link
Copy Markdown
Contributor

Superseded by #56473, which is now merged and covers the same FTS-only provider-less indexing fix area. The landed work also includes the follow-up default-threshold fix for provider-less keyword hits plus regression coverage.

@obviyus obviyus closed this Mar 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support FTS-only memory indexing (skip embedding provider)

2 participants