fix(memory): indexer continues on embedding errors instead of stopping#31961
fix(memory): indexer continues on embedding errors instead of stopping#31961teodorangel wants to merge 2 commits into
Conversation
Greptile SummaryThis PR fixes a real and well-documented pain point: a single embedding failure with Key concerns with the current implementation:
Confidence Score: 3/5
Last reviewed commit: 655d93f |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 655d93f14c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| tasks, | ||
| limit, | ||
| errorMode: "stop", | ||
| errorMode: "continue", |
There was a problem hiding this comment.
Restore fail-fast behavior for batch embedding group errors
Changing runWithConcurrency to errorMode: "continue" suppresses exceptions that batch embedding code depends on to trigger recovery. In src/memory/batch-runner.ts:46, a failing group from runOpenAiEmbeddingBatches/runGeminiEmbeddingBatches now gets swallowed, so runBatchWithFallback in src/memory/manager-embedding-ops.ts is never activated; the code then applies only partial byCustomId results and persists missing chunk embeddings as empty arrays while still updating the file hash as indexed. This can silently leave files permanently partially indexed until a forced reindex.
Useful? React with 👍 / 👎.
| // Kai fork: continue on error instead of stopping. | ||
| // With errorMode "stop", ONE embedding failure kills the entire index run — | ||
| // all concurrent workers stop immediately. With 22,350 files, only ~497 ever | ||
| // get indexed. Root cause behind GitHub issue #8561. | ||
| // Original: Patch 30 (dist, Mar 1, 2026) | ||
| const { results } = await runTasksWithConcurrency({ | ||
| tasks, | ||
| limit, | ||
| errorMode: "stop", | ||
| errorMode: "continue", | ||
| }); | ||
| if (hasError) { | ||
| throw firstError; | ||
| } | ||
| return results; |
There was a problem hiding this comment.
Embedding errors silently swallowed — no logging
With errorMode: "continue", any embedding failure is now completely silent. The runTasksWithConcurrency utility provides an onTaskError callback precisely for this case, but it's not wired up here. Without it, operators have no way to know which files failed to index or why.
The results array returned by runTasksWithConcurrency is pre-allocated with Array.from({ length: tasks.length }), so positions for failed tasks are left as uninitialised undefined slots and silently returned as part of T[]. While neither current caller uses the return value (both discard it), this is an invisible data integrity issue.
Consider passing onTaskError to at least log failures:
const { results } = await runTasksWithConcurrency({
tasks,
limit,
errorMode: "continue",
onTaskError: (error, index) => {
log.warn("runWithConcurrency: task failed, skipping", { index, error });
},
});
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/memory/internal.ts
Line: 322-332
Comment:
**Embedding errors silently swallowed — no logging**
With `errorMode: "continue"`, any embedding failure is now completely silent. The `runTasksWithConcurrency` utility provides an `onTaskError` callback precisely for this case, but it's not wired up here. Without it, operators have no way to know which files failed to index or why.
The `results` array returned by `runTasksWithConcurrency` is pre-allocated with `Array.from({ length: tasks.length })`, so positions for failed tasks are left as uninitialised `undefined` slots and silently returned as part of `T[]`. While neither current caller uses the return value (both discard it), this is an invisible data integrity issue.
Consider passing `onTaskError` to at least log failures:
```
const { results } = await runTasksWithConcurrency({
tasks,
limit,
errorMode: "continue",
onTaskError: (error, index) => {
log.warn("runWithConcurrency: task failed, skipping", { index, error });
},
});
```
How can I resolve this? If you propose a fix, please make it concise.| // With errorMode "stop", ONE embedding failure kills the entire index run — | ||
| // all concurrent workers stop immediately. With 22,350 files, only ~497 ever | ||
| // get indexed. Root cause behind GitHub issue #8561. | ||
| // Original: Patch 30 (dist, Mar 1, 2026) |
There was a problem hiding this comment.
Comment is missing a leading slash
/ Original: Patch 30 (dist, Mar 1, 2026) is a single-line comment with only one /, making it invalid syntax that TypeScript will reject as a stray division operator in an expression context.
| // Original: Patch 30 (dist, Mar 1, 2026) | |
| // Original: Patch 30 (dist, Mar 1, 2026) |
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/memory/internal.ts
Line: 326
Comment:
**Comment is missing a leading slash**
`/ Original: Patch 30 (dist, Mar 1, 2026)` is a single-line comment with only one `/`, making it invalid syntax that TypeScript will reject as a stray division operator in an expression context.
```suggestion
// Original: Patch 30 (dist, Mar 1, 2026)
```
How can I resolve this? If you propose a fix, please make it concise.| const { results } = await runTasksWithConcurrency({ | ||
| tasks, | ||
| limit, | ||
| errorMode: "stop", | ||
| errorMode: "continue", | ||
| }); |
There was a problem hiding this comment.
Progress counter will stall below 100% on any embedding failure
In both syncMemoryFiles (manager-sync-ops.ts:660-682) and syncSessionFiles (manager-sync-ops.ts:740-785), the params.progress.completed += 1 increment sits after the await this.indexFile(...) call. When indexFile throws, the task exits immediately and progress.completed is never incremented, even though progress.total was already bumped for that file. The result is a progress bar that never reaches 100% when any file fails.
With the previous errorMode: "stop" behaviour the whole operation failed, so this was moot. Now that indexing continues past failures, this becomes a user-visible glitch. A try/finally block (or moving the increment before the await) inside each task would ensure the counter always advances.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/memory/internal.ts
Line: 327-331
Comment:
**Progress counter will stall below 100% on any embedding failure**
In both `syncMemoryFiles` (`manager-sync-ops.ts:660-682`) and `syncSessionFiles` (`manager-sync-ops.ts:740-785`), the `params.progress.completed += 1` increment sits **after** the `await this.indexFile(...)` call. When `indexFile` throws, the task exits immediately and `progress.completed` is never incremented, even though `progress.total` was already bumped for that file. The result is a progress bar that never reaches 100% when any file fails.
With the previous `errorMode: "stop"` behaviour the whole operation failed, so this was moot. Now that indexing continues past failures, this becomes a user-visible glitch. A `try/finally` block (or moving the increment before the await) inside each task would ensure the counter always advances.
How can I resolve this? If you propose a fix, please make it concise.655d93f to
2bfc053
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2bfc05323c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| tasks, | ||
| limit, | ||
| errorMode: "stop", | ||
| errorMode: "continue", |
There was a problem hiding this comment.
Propagate batch group failures to trigger fallback
Changing this helper to errorMode: "continue" causes runEmbeddingBatchGroups in src/memory/batch-runner.ts to return partial results when one provider batch group throws, instead of failing the file index operation. embedChunksWithProviderBatch in src/memory/manager-embedding-ops.ts then applies only the successful byCustomId entries and leaves missing chunk embeddings as empty arrays while still finishing indexFile and upserting the file hash, so runBatchWithFallback and batch-failure tracking are bypassed. A transient OpenAI/Gemini/Voyage batch-group failure can therefore silently produce partially indexed files that are treated as complete.
Useful? React with 👍 / 👎.
| const { results } = await runTasksWithConcurrency({ | ||
| tasks, | ||
| limit, | ||
| errorMode: "stop", | ||
| errorMode: "continue", |
There was a problem hiding this comment.
Re-throw task errors so failed files are retried
With task errors now swallowed here, syncMemoryFiles/syncSessionFiles in src/memory/manager-sync-ops.ts return successfully even when some indexFile tasks fail, and runSync subsequently clears this.dirty, this.sessionsDirty, and this.sessionsDirtyFiles. In that state, unchanged files that failed due transient embedding/network errors are skipped on later sync passes, so indexing gaps persist until a manual force reindex or file modification occurs.
Useful? React with 👍 / 👎.
2bfc053 to
68f8106
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 68f8106b6c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| tasks, | ||
| limit, | ||
| errorMode: "stop", | ||
| errorMode: "continue", |
There was a problem hiding this comment.
Re-throw failed indexing tasks after concurrent sync
Changing this helper to errorMode: "continue" causes syncMemoryFiles and syncSessionFiles (in src/memory/manager-sync-ops.ts) to return successfully even when some indexFile tasks fail, and then runSync clears this.dirty, this.sessionsDirty, and this.sessionsDirtyFiles. In transient embedding/network failure scenarios, unchanged files that failed once are no longer retried on subsequent sync passes, so indexing gaps can persist until a manual force reindex or file modification.
Useful? React with 👍 / 👎.
| tasks, | ||
| limit, | ||
| errorMode: "stop", | ||
| errorMode: "continue", |
There was a problem hiding this comment.
Propagate batch group failures to preserve fallback path
With errorMode: "continue", failures inside runEmbeddingBatchGroups (used by OpenAI/Gemini/Voyage batch runners) are swallowed, so runBatchWithFallback in src/memory/manager-embedding-ops.ts never triggers fallback embeddings or batch-failure accounting for those group errors. The code then applies only successful custom_id results and indexFile stores remaining chunks with empty embeddings while still updating the file hash, which silently records partially indexed files as complete after a batch-group failure.
Useful? React with 👍 / 👎.
With errorMode 'stop', ONE file that fails to embed kills the entire index. Result: only ~497 out of 22,350 files ever get indexed. Root cause behind GitHub issue openclaw#8561. Original: Patch 30 (dist, Mar 1, 2026) Ported to source: Mar 2, 2026
68f8106 to
51524db
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 51524dba08
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| tasks, | ||
| limit, | ||
| errorMode: "stop", | ||
| errorMode: "continue", |
There was a problem hiding this comment.
Propagate task errors during atomic reindex
Switching this helper to errorMode: "continue" hides indexFile failures from callers, so runSafeReindex in src/memory/manager-sync-ops.ts can treat a force/full reindex as successful and still execute swapIndexFiles with a partially populated temp database. In transient embedding/provider failure scenarios, previously indexed files that fail on this pass are silently omitted from the new index, which breaks the atomic reindex guarantee and can drop searchable chunks until another successful full rebuild.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2dc59f5c7f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| tasks, | ||
| limit, | ||
| errorMode: "stop", | ||
| errorMode: "continue", |
There was a problem hiding this comment.
Fail batch embedding runs when any group task errors
With errorMode: "continue" here, runEmbeddingBatchGroups in src/memory/batch-runner.ts no longer throws when one OpenAI/Gemini/Voyage group fails, so runBatchWithFallback in src/memory/manager-embedding-ops.ts never activates the non-batch fallback path. The partial byCustomId map is then treated as success, and indexFile writes missing chunks with empty embeddings (embeddings[i] ?? []), silently committing partially indexed files after transient provider failures.
Useful? React with 👍 / 👎.
| tasks, | ||
| limit, | ||
| errorMode: "stop", | ||
| errorMode: "continue", |
There was a problem hiding this comment.
Propagate per-file indexing failures from concurrent sync
Swallowing task exceptions in this helper causes syncMemoryFiles and syncSessionFiles (in src/memory/manager-sync-ops.ts) to resolve even when indexFile fails, after which runSync clears dirty/sessionsDirty tracking as if indexing succeeded. In transient embedding or network error scenarios, unchanged files that failed once are skipped on later sync passes, so indexing gaps can persist until a force reindex or file modification occurs.
Useful? React with 👍 / 👎.
|
This pull request has been automatically marked as stale due to inactivity. |
|
Closing due to inactivity. |
Problem
runWithConcurrency()insrc/memory/internal.tsuseserrorMode: "stop", which causes the entire index run to abort when a single file fails to embed. With large workspaces (22K+ files), a single corrupt or unsupported file kills the entire indexing pipeline — only ~497 of 22,350 files ever get indexed.Related: #8561
Fix
Change
errorMode: "stop"toerrorMode: "continue". Failed files are skipped; the rest of the workspace gets indexed normally.Impact
src/memory/internal.tsTesting
Tested on a workspace with 22,350 files over 6 weeks. Before: indexer would stop at ~497 chunks on any embedding failure. After: full index completes (33,000+ chunks), skipping only the files that actually fail.