Skip to content

fix(memory): indexer continues on embedding errors instead of stopping#31961

Closed
teodorangel wants to merge 2 commits into
openclaw:mainfrom
teodorangel:fix/indexer-error-mode
Closed

fix(memory): indexer continues on embedding errors instead of stopping#31961
teodorangel wants to merge 2 commits into
openclaw:mainfrom
teodorangel:fix/indexer-error-mode

Conversation

@teodorangel

Copy link
Copy Markdown

Problem

runWithConcurrency() in src/memory/internal.ts uses errorMode: "stop", which causes the entire index run to abort when a single file fails to embed. With large workspaces (22K+ files), a single corrupt or unsupported file kills the entire indexing pipeline — only ~497 of 22,350 files ever get indexed.

Related: #8561

Fix

Change errorMode: "stop" to errorMode: "continue". Failed files are skipped; the rest of the workspace gets indexed normally.

Impact

  • One-line change in src/memory/internal.ts
  • No behavioral change for workspaces without embedding errors
  • Workspaces with problematic files now complete indexing instead of silently stopping

Testing

Tested on a workspace with 22,350 files over 6 weeks. Before: indexer would stop at ~497 chunks on any embedding failure. After: full index completes (33,000+ chunks), skipping only the files that actually fail.

@greptile-apps

greptile-apps Bot commented Mar 2, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a real and well-documented pain point: a single embedding failure with errorMode: "stop" was aborting the entire indexing pipeline for large workspaces. Switching to errorMode: "continue" is the right direction.

Key concerns with the current implementation:

  • Silent error swallowingrunTasksWithConcurrency exposes an onTaskError callback for exactly this use-case, but it is not used. Failed files are now dropped with zero logging, making it impossible to diagnose which files are problematic or why indexing produced fewer chunks than expected.
  • Syntax error in added comment — line 326 reads / Original: Patch 30 (dist, Mar 1, 2026) (single /), which TypeScript will parse as a division expression, not a comment.
  • Progress bar stalls below 100% on failure — in both syncMemoryFiles and syncSessionFiles the progress.completed increment is placed after await this.indexFile(...). When a task throws, the counter is never advanced. With errorMode: "stop" this was invisible (the whole operation failed); with errorMode: "continue" the UI progress bar will visibly stall short of 100% whenever any file fails.

Confidence Score: 3/5

  • The core fix is sound, but a syntax error in the added comment and the absence of any error logging make this not quite ready to merge as-is.
  • The behavioural change (continue vs stop) is correct and well-reasoned. However, a syntax error exists in the new comment block, all embedding errors are now silently swallowed with no observability, and a pre-existing progress-counter bug becomes user-visible. None of these are blockers individually, but together they reduce confidence enough to warrant a revision pass before merging.
  • src/memory/internal.ts — syntax error in comment and missing onTaskError logging

Last reviewed commit: 655d93f

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 655d93f14c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/memory/internal.ts
tasks,
limit,
errorMode: "stop",
errorMode: "continue",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Restore fail-fast behavior for batch embedding group errors

Changing runWithConcurrency to errorMode: "continue" suppresses exceptions that batch embedding code depends on to trigger recovery. In src/memory/batch-runner.ts:46, a failing group from runOpenAiEmbeddingBatches/runGeminiEmbeddingBatches now gets swallowed, so runBatchWithFallback in src/memory/manager-embedding-ops.ts is never activated; the code then applies only partial byCustomId results and persists missing chunk embeddings as empty arrays while still updating the file hash as indexed. This can silently leave files permanently partially indexed until a forced reindex.

Useful? React with 👍 / 👎.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 file reviewed, 3 comments

Edit Code Review Agent Settings | Greptile

Comment thread src/memory/internal.ts Outdated
Comment on lines 322 to 332
// Kai fork: continue on error instead of stopping.
// With errorMode "stop", ONE embedding failure kills the entire index run —
// all concurrent workers stop immediately. With 22,350 files, only ~497 ever
// get indexed. Root cause behind GitHub issue #8561.
// Original: Patch 30 (dist, Mar 1, 2026)
const { results } = await runTasksWithConcurrency({
tasks,
limit,
errorMode: "stop",
errorMode: "continue",
});
if (hasError) {
throw firstError;
}
return results;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Embedding errors silently swallowed — no logging

With errorMode: "continue", any embedding failure is now completely silent. The runTasksWithConcurrency utility provides an onTaskError callback precisely for this case, but it's not wired up here. Without it, operators have no way to know which files failed to index or why.

The results array returned by runTasksWithConcurrency is pre-allocated with Array.from({ length: tasks.length }), so positions for failed tasks are left as uninitialised undefined slots and silently returned as part of T[]. While neither current caller uses the return value (both discard it), this is an invisible data integrity issue.

Consider passing onTaskError to at least log failures:

const { results } = await runTasksWithConcurrency({
  tasks,
  limit,
  errorMode: "continue",
  onTaskError: (error, index) => {
    log.warn("runWithConcurrency: task failed, skipping", { index, error });
  },
});
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/memory/internal.ts
Line: 322-332

Comment:
**Embedding errors silently swallowed — no logging**

With `errorMode: "continue"`, any embedding failure is now completely silent. The `runTasksWithConcurrency` utility provides an `onTaskError` callback precisely for this case, but it's not wired up here. Without it, operators have no way to know which files failed to index or why.

The `results` array returned by `runTasksWithConcurrency` is pre-allocated with `Array.from({ length: tasks.length })`, so positions for failed tasks are left as uninitialised `undefined` slots and silently returned as part of `T[]`. While neither current caller uses the return value (both discard it), this is an invisible data integrity issue.

Consider passing `onTaskError` to at least log failures:
```
const { results } = await runTasksWithConcurrency({
  tasks,
  limit,
  errorMode: "continue",
  onTaskError: (error, index) => {
    log.warn("runWithConcurrency: task failed, skipping", { index, error });
  },
});
```

How can I resolve this? If you propose a fix, please make it concise.

Comment thread src/memory/internal.ts Outdated
// With errorMode "stop", ONE embedding failure kills the entire index run —
// all concurrent workers stop immediately. With 22,350 files, only ~497 ever
// get indexed. Root cause behind GitHub issue #8561.
// Original: Patch 30 (dist, Mar 1, 2026)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment is missing a leading slash

/ Original: Patch 30 (dist, Mar 1, 2026) is a single-line comment with only one /, making it invalid syntax that TypeScript will reject as a stray division operator in an expression context.

Suggested change
// Original: Patch 30 (dist, Mar 1, 2026)
// Original: Patch 30 (dist, Mar 1, 2026)
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/memory/internal.ts
Line: 326

Comment:
**Comment is missing a leading slash**

`/ Original: Patch 30 (dist, Mar 1, 2026)` is a single-line comment with only one `/`, making it invalid syntax that TypeScript will reject as a stray division operator in an expression context.

```suggestion
  // Original: Patch 30 (dist, Mar 1, 2026)
```

How can I resolve this? If you propose a fix, please make it concise.

Comment thread src/memory/internal.ts
Comment on lines +327 to 331
const { results } = await runTasksWithConcurrency({
tasks,
limit,
errorMode: "stop",
errorMode: "continue",
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Progress counter will stall below 100% on any embedding failure

In both syncMemoryFiles (manager-sync-ops.ts:660-682) and syncSessionFiles (manager-sync-ops.ts:740-785), the params.progress.completed += 1 increment sits after the await this.indexFile(...) call. When indexFile throws, the task exits immediately and progress.completed is never incremented, even though progress.total was already bumped for that file. The result is a progress bar that never reaches 100% when any file fails.

With the previous errorMode: "stop" behaviour the whole operation failed, so this was moot. Now that indexing continues past failures, this becomes a user-visible glitch. A try/finally block (or moving the increment before the await) inside each task would ensure the counter always advances.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/memory/internal.ts
Line: 327-331

Comment:
**Progress counter will stall below 100% on any embedding failure**

In both `syncMemoryFiles` (`manager-sync-ops.ts:660-682`) and `syncSessionFiles` (`manager-sync-ops.ts:740-785`), the `params.progress.completed += 1` increment sits **after** the `await this.indexFile(...)` call. When `indexFile` throws, the task exits immediately and `progress.completed` is never incremented, even though `progress.total` was already bumped for that file. The result is a progress bar that never reaches 100% when any file fails.

With the previous `errorMode: "stop"` behaviour the whole operation failed, so this was moot. Now that indexing continues past failures, this becomes a user-visible glitch. A `try/finally` block (or moving the increment before the await) inside each task would ensure the counter always advances.

How can I resolve this? If you propose a fix, please make it concise.

@teodorangel
teodorangel force-pushed the fix/indexer-error-mode branch from 655d93f to 2bfc053 Compare March 2, 2026 20:11

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2bfc05323c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/memory/internal.ts
tasks,
limit,
errorMode: "stop",
errorMode: "continue",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Propagate batch group failures to trigger fallback

Changing this helper to errorMode: "continue" causes runEmbeddingBatchGroups in src/memory/batch-runner.ts to return partial results when one provider batch group throws, instead of failing the file index operation. embedChunksWithProviderBatch in src/memory/manager-embedding-ops.ts then applies only the successful byCustomId entries and leaves missing chunk embeddings as empty arrays while still finishing indexFile and upserting the file hash, so runBatchWithFallback and batch-failure tracking are bypassed. A transient OpenAI/Gemini/Voyage batch-group failure can therefore silently produce partially indexed files that are treated as complete.

Useful? React with 👍 / 👎.

Comment thread src/memory/internal.ts
Comment on lines +326 to +329
const { results } = await runTasksWithConcurrency({
tasks,
limit,
errorMode: "stop",
errorMode: "continue",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Re-throw task errors so failed files are retried

With task errors now swallowed here, syncMemoryFiles/syncSessionFiles in src/memory/manager-sync-ops.ts return successfully even when some indexFile tasks fail, and runSync subsequently clears this.dirty, this.sessionsDirty, and this.sessionsDirtyFiles. In that state, unchanged files that failed due transient embedding/network errors are skipped on later sync passes, so indexing gaps persist until a manual force reindex or file modification occurs.

Useful? React with 👍 / 👎.

@teodorangel
teodorangel force-pushed the fix/indexer-error-mode branch from 2bfc053 to 68f8106 Compare March 2, 2026 20:38

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 68f8106b6c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/memory/internal.ts
tasks,
limit,
errorMode: "stop",
errorMode: "continue",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Re-throw failed indexing tasks after concurrent sync

Changing this helper to errorMode: "continue" causes syncMemoryFiles and syncSessionFiles (in src/memory/manager-sync-ops.ts) to return successfully even when some indexFile tasks fail, and then runSync clears this.dirty, this.sessionsDirty, and this.sessionsDirtyFiles. In transient embedding/network failure scenarios, unchanged files that failed once are no longer retried on subsequent sync passes, so indexing gaps can persist until a manual force reindex or file modification.

Useful? React with 👍 / 👎.

Comment thread src/memory/internal.ts
tasks,
limit,
errorMode: "stop",
errorMode: "continue",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Propagate batch group failures to preserve fallback path

With errorMode: "continue", failures inside runEmbeddingBatchGroups (used by OpenAI/Gemini/Voyage batch runners) are swallowed, so runBatchWithFallback in src/memory/manager-embedding-ops.ts never triggers fallback embeddings or batch-failure accounting for those group errors. The code then applies only successful custom_id results and indexFile stores remaining chunks with empty embeddings while still updating the file hash, which silently records partially indexed files as complete after a batch-group failure.

Useful? React with 👍 / 👎.

With errorMode 'stop', ONE file that fails to embed kills the entire index.
Result: only ~497 out of 22,350 files ever get indexed.
Root cause behind GitHub issue openclaw#8561.

Original: Patch 30 (dist, Mar 1, 2026)
Ported to source: Mar 2, 2026
@teodorangel
teodorangel force-pushed the fix/indexer-error-mode branch from 68f8106 to 51524db Compare March 3, 2026 03:53

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 51524dba08

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/memory/internal.ts
tasks,
limit,
errorMode: "stop",
errorMode: "continue",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Propagate task errors during atomic reindex

Switching this helper to errorMode: "continue" hides indexFile failures from callers, so runSafeReindex in src/memory/manager-sync-ops.ts can treat a force/full reindex as successful and still execute swapIndexFiles with a partially populated temp database. In transient embedding/provider failure scenarios, previously indexed files that fail on this pass are silently omitted from the new index, which breaks the atomic reindex guarantee and can drop searchable chunks until another successful full rebuild.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2dc59f5c7f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/memory/internal.ts
tasks,
limit,
errorMode: "stop",
errorMode: "continue",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Fail batch embedding runs when any group task errors

With errorMode: "continue" here, runEmbeddingBatchGroups in src/memory/batch-runner.ts no longer throws when one OpenAI/Gemini/Voyage group fails, so runBatchWithFallback in src/memory/manager-embedding-ops.ts never activates the non-batch fallback path. The partial byCustomId map is then treated as success, and indexFile writes missing chunks with empty embeddings (embeddings[i] ?? []), silently committing partially indexed files after transient provider failures.

Useful? React with 👍 / 👎.

Comment thread src/memory/internal.ts
tasks,
limit,
errorMode: "stop",
errorMode: "continue",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Propagate per-file indexing failures from concurrent sync

Swallowing task exceptions in this helper causes syncMemoryFiles and syncSessionFiles (in src/memory/manager-sync-ops.ts) to resolve even when indexFile fails, after which runSync clears dirty/sessionsDirty tracking as if indexing succeeded. In transient embedding or network error scenarios, unchanged files that failed once are skipped on later sync passes, so indexing gaps can persist until a force reindex or file modification occurs.

Useful? React with 👍 / 👎.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Apr 7, 2026
@openclaw-barnacle

Copy link
Copy Markdown

Closing due to inactivity.
If you believe this PR should be revived, post in #pr-thunderdome-dangerzone on Discord to talk to a maintainer.
That channel is the escape hatch for high-quality PRs that get auto-closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size: S stale Marked as stale due to inactivity

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant