fix(memory): support FTS-only indexing without embedding provider#44094
fix(memory): support FTS-only indexing without embedding provider#44094Haohao-end wants to merge 3 commits into
Conversation
Greptile SummaryThis PR fixes the FTS-only memory indexing pipeline by removing early Key issues found:
Confidence Score: 3/5
Prompt To Fix All With AIThis is a comment left during a code review.
Path: src/memory/manager-embedding-ops.ts
Line: 771-778
Comment:
**Orphaned FTS rows when model transitions across syncs**
`clearIndexedFileData` deletes `chunks` rows without any model filter (`DELETE FROM chunks WHERE path = ? AND source = ?`), but the FTS cleanup only removes rows matching `currentModel`. If a user previously indexed with a real embedding provider (say `model = "text-embedding-3-small"`) and then removes the provider, the next resync/re-index of a changed file will:
1. Delete ALL chunks for that path (no model filter) ✓
2. Try to delete FTS rows with `model = "fts-only"` — nothing to delete yet
3. Insert new chunks and FTS rows with `model = "fts-only"` ✓
4. **Leave the old `model = "text-embedding-3-small"` FTS rows untouched** ✗
Since `searchKeyword` searches ALL models when `providerModel` is `undefined` (FTS-only mode — see `manager-search.ts` line 155–156: `const modelClause = params.providerModel ? " AND model = ?" : ""`), those orphaned rows will be returned in future searches, causing duplicate results for the same content.
The same issue affects the stale-row cleanup loops in `syncMemoryFiles` (around line 741 in `manager-sync-ops.ts`) and `syncSessionFiles` (around line 843 in `manager-sync-ops.ts`), both of which also only delete FTS rows for `currentModel` while deleting chunks without a model filter.
The fix for all three sites is to drop the `model` predicate when deleting from the FTS table so it matches the model-agnostic `chunks` cleanup:
```suggestion
const currentModel = this.provider?.model ?? "fts-only";
if (this.fts.enabled && this.fts.available) {
try {
this.db
.prepare(`DELETE FROM ${FTS_TABLE} WHERE path = ? AND source = ?`)
.run(pathname, source);
} catch {}
}
```
The same one-liner change should be applied to the FTS `DELETE` in `syncMemoryFiles` (line ~741) and `syncSessionFiles` (line ~843) in `manager-sync-ops.ts`.
How can I resolve this? If you propose a fix, please make it concise.Last reviewed commit: 4eae898 |
| const currentModel = this.provider?.model ?? "fts-only"; | ||
| if (this.fts.enabled && this.fts.available) { | ||
| try { | ||
| this.db | ||
| .prepare(`DELETE FROM ${FTS_TABLE} WHERE path = ? AND source = ? AND model = ?`) | ||
| .run(pathname, source, this.provider.model); | ||
| .run(pathname, source, currentModel); | ||
| } catch {} | ||
| } |
There was a problem hiding this comment.
Orphaned FTS rows when model transitions across syncs
clearIndexedFileData deletes chunks rows without any model filter (DELETE FROM chunks WHERE path = ? AND source = ?), but the FTS cleanup only removes rows matching currentModel. If a user previously indexed with a real embedding provider (say model = "text-embedding-3-small") and then removes the provider, the next resync/re-index of a changed file will:
- Delete ALL chunks for that path (no model filter) ✓
- Try to delete FTS rows with
model = "fts-only"— nothing to delete yet - Insert new chunks and FTS rows with
model = "fts-only"✓ - Leave the old
model = "text-embedding-3-small"FTS rows untouched ✗
Since searchKeyword searches ALL models when providerModel is undefined (FTS-only mode — see manager-search.ts line 155–156: const modelClause = params.providerModel ? " AND model = ?" : ""), those orphaned rows will be returned in future searches, causing duplicate results for the same content.
The same issue affects the stale-row cleanup loops in syncMemoryFiles (around line 741 in manager-sync-ops.ts) and syncSessionFiles (around line 843 in manager-sync-ops.ts), both of which also only delete FTS rows for currentModel while deleting chunks without a model filter.
The fix for all three sites is to drop the model predicate when deleting from the FTS table so it matches the model-agnostic chunks cleanup:
| const currentModel = this.provider?.model ?? "fts-only"; | |
| if (this.fts.enabled && this.fts.available) { | |
| try { | |
| this.db | |
| .prepare(`DELETE FROM ${FTS_TABLE} WHERE path = ? AND source = ? AND model = ?`) | |
| .run(pathname, source, this.provider.model); | |
| .run(pathname, source, currentModel); | |
| } catch {} | |
| } | |
| const currentModel = this.provider?.model ?? "fts-only"; | |
| if (this.fts.enabled && this.fts.available) { | |
| try { | |
| this.db | |
| .prepare(`DELETE FROM ${FTS_TABLE} WHERE path = ? AND source = ?`) | |
| .run(pathname, source); | |
| } catch {} | |
| } |
The same one-liner change should be applied to the FTS DELETE in syncMemoryFiles (line ~741) and syncSessionFiles (line ~843) in manager-sync-ops.ts.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/memory/manager-embedding-ops.ts
Line: 771-778
Comment:
**Orphaned FTS rows when model transitions across syncs**
`clearIndexedFileData` deletes `chunks` rows without any model filter (`DELETE FROM chunks WHERE path = ? AND source = ?`), but the FTS cleanup only removes rows matching `currentModel`. If a user previously indexed with a real embedding provider (say `model = "text-embedding-3-small"`) and then removes the provider, the next resync/re-index of a changed file will:
1. Delete ALL chunks for that path (no model filter) ✓
2. Try to delete FTS rows with `model = "fts-only"` — nothing to delete yet
3. Insert new chunks and FTS rows with `model = "fts-only"` ✓
4. **Leave the old `model = "text-embedding-3-small"` FTS rows untouched** ✗
Since `searchKeyword` searches ALL models when `providerModel` is `undefined` (FTS-only mode — see `manager-search.ts` line 155–156: `const modelClause = params.providerModel ? " AND model = ?" : ""`), those orphaned rows will be returned in future searches, causing duplicate results for the same content.
The same issue affects the stale-row cleanup loops in `syncMemoryFiles` (around line 741 in `manager-sync-ops.ts`) and `syncSessionFiles` (around line 843 in `manager-sync-ops.ts`), both of which also only delete FTS rows for `currentModel` while deleting chunks without a model filter.
The fix for all three sites is to drop the `model` predicate when deleting from the FTS table so it matches the model-agnostic `chunks` cleanup:
```suggestion
const currentModel = this.provider?.model ?? "fts-only";
if (this.fts.enabled && this.fts.available) {
try {
this.db
.prepare(`DELETE FROM ${FTS_TABLE} WHERE path = ? AND source = ?`)
.run(pathname, source);
} catch {}
}
```
The same one-liner change should be applied to the FTS `DELETE` in `syncMemoryFiles` (line ~741) and `syncSessionFiles` (line ~843) in `manager-sync-ops.ts`.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4eae898336
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return; | ||
| } | ||
|
|
||
| const files = await listMemoryFiles( |
There was a problem hiding this comment.
Skip providerless indexing when FTS is disabled
Add a guard for !this.provider && !this.fts.enabled before starting file scans here (and in syncSessionFiles). With this change, providerless setups still read/chunk/write chunks rows even when query.hybrid.enabled=false, but those rows are never searchable because search() returns early when no provider and FTS is disabled (src/memory/manager.ts:285-287). This creates unnecessary sync latency and disk growth for configurations that intentionally turned hybrid/FTS off.
Useful? React with 👍 / 👎.
|
Good catch — I pushed a follow-up fix. Changes included:
Validated locally with:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5b046b2122
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (!this.provider && !this.fts.enabled) { | ||
| return; |
There was a problem hiding this comment.
Guard providerless sync on FTS availability
When no embedding provider is configured, this condition still runs indexing whenever query.hybrid.enabled is true, even if FTS failed to initialize (fts.available === false, e.g. a SQLite build without FTS5). In that state indexFile() writes chunks rows but never inserts chunks_fts rows, and providerless search() returns early with no results when FTS is unavailable, so sync work and disk usage grow for data that can never be queried; the same guard pattern is also present in syncSessionFiles.
Useful? React with 👍 / 👎.
|
Pushed a small follow-up update. I tightened the provider-less sync fast-exit in Validated locally with:
|
|
Superseded by #56473, which is now merged and covers the same FTS-only provider-less indexing fix area. The landed work also includes the follow-up default-threshold fix for provider-less keyword hits plus regression coverage. |
Summary
This PR fixes memory indexing when no embedding provider is configured.
Previously, the indexing pipeline bailed out early in provider-null mode, which prevented:
As a result, memory search could report FTS-only mode while still returning no indexed results.
This follow-up also fixes cleanup behavior when transitioning from provider-backed indexing to FTS-only mode, so old-model FTS rows do not remain behind and cause duplicate search results. It also skips provider-less sync work when FTS is disabled or unavailable.
What changed
syncMemoryFiles()syncSessionFiles()indexFile()so text chunking and persistence always happen"fts-only"clearIndexedFileData()syncMemoryFiles()syncSessionFiles()Scope
This patch intentionally preserves the existing configuration semantics around FTS/hybrid enablement. It fixes the broken provider-null indexing path when FTS-backed search is enabled, but does not change whether FTS should be auto-enabled when
query.hybrid.enabledis false.Validation
Passed locally:
pnpm exec vitest run src/memory/manager.fts-only.test.tspnpm exec vitest run src/memory/index.test.tspnpm buildpnpm checkCloses #43957