fix: enhance error handling for document processing and add OCR requirement check for image files (backport of #1859) by ricofurtado · Pull Request #2017 · langflow-ai/openrag

ricofurtado · 2026-07-03T18:52:41Z

Backport of #1859 from release-cpd to main.

Merges two independent post-ingest verification improvements that had each evolved separately on release-cpd and main since they diverged:

main's on_error="assume_exists" handling (don't fail a file on transient OpenSearch errors after Langflow already reported success) is combined with fix: enhance error handling for document processing and add OCR requirement check for image files #1859's wait_for_visibility retry (ride out OpenSearch's near-real-time refresh window).
Keeps the real correctness fix from fix: enhance error handling for document processing and add OCR requirement check for image files #1859: the Langflow no-connector-id ingestion path now verifies success by filename via check_filename_exists, not by hash_id(item) — Langflow assigns its own document_id there, so the hash was never actually the stored id and the check was effectively a no-op.
Also includes the OCR-required-image-type detection in task_service.py and a flaky-test fix in the TS SDK integration suite.

Test plan

python3 -m py_compile on touched files
Full unit suite for touched areas passes

🤖 Generated with Claude Code

Summary by CodeRabbit

Bug Fixes
- Improved document deletion and verification reliability so newly ingested files are recognized even with indexing delays.
- Fixed cases where a document could appear missing right after ingestion, reducing false failures during upload and delete checks.
- Updated test coverage to use unique sample files, helping prevent leftover data from affecting results.

…rement check for image files (#1859) * fix: enhance error handling for document processing and add OCR requirement check for image files * style: ruff autofix (auto) * remove clickhouse connect from langflow image (#1801) * chore: fix ci (#1889) * Update processors.py * test fix * fix ci --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Lucas Oliveira <[email protected]> Co-authored-by: Edwin Jose <[email protected]> (cherry picked from commit a286581)

coderabbitai · 2026-07-03T18:53:07Z

Walkthrough

Adds a wait_for_visibility retry option to TaskProcessor.check_document_exists and check_filename_exists to handle OpenSearch near-real-time refresh delays, wires this into Langflow/connector ingestion verification, and updates the SDK delete-document integration test to use a unique file to avoid interference from earlier ingests.

Changes

Visibility Retry and Ingestion Verification

Layer / File(s)	Summary
check_document_exists retry logic `src/models/processors.py`	Adds kw-only `wait_for_visibility` parameter and retry-with-backoff when no hits are found, with docstring updates.
check_filename_exists retry logic `src/models/processors.py`	Adds kw-only `wait_for_visibility` parameter; resets alias candidates and retries after delay when no hits found across all aliases.
Ingestion verification wiring `src/models/processors.py`	`ConnectorFileProcessor.process_item` passes `wait_for_visibility=True`; `LangflowFileProcessor.process_item` switches from hash-based `check_document_exists` to `check_filename_exists(original_filename, wait_for_visibility=True)`.

SDK Integration Test

Layer / File(s)	Summary
Unique file per delete test `sdks/typescript/tests/integration.test.ts`	`should delete a document` now creates, ingests, and deletes a uniquely named temporary file instead of reusing the shared test file, keeping existing assertion logic.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant LangflowFileProcessor
  participant TaskProcessor
  participant OpenSearch

  LangflowFileProcessor->>TaskProcessor: check_filename_exists(original_filename, wait_for_visibility=True)
  TaskProcessor->>OpenSearch: query for filename
  OpenSearch-->>TaskProcessor: no hits
  TaskProcessor->>TaskProcessor: sleep with backoff, retry
  TaskProcessor->>OpenSearch: query for filename (retry)
  OpenSearch-->>TaskProcessor: hits found
  TaskProcessor-->>LangflowFileProcessor: exists = True

Possibly related PRs

langflow-ai/openrag#1695: Both PRs modify check_document_exists/OpenSearch existence check behavior in the same method.
langflow-ai/openrag#1851: Both PRs update post-ingestion verification logic in src/models/processors.py for Langflow/connector flows.
langflow-ai/openrag#1857: Both PRs address OpenSearch near-real-time visibility issues via retry-based existence checks in the same ingestion verification paths.

Suggested reviewers: edwinjosechittilappilly, lucaseduoli, mfortman11

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title matches the main document-processing fixes and correctly frames the change as a backport.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch cpd-backport-1859-ocr-error-handling

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/models/processors.py (1)
159-237: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Add on_error handling to check_filename_exists
check_filename_exists still treats final OpenSearch errors as “missing”, so the post-ingest verification path can mark an already ingested file as FAILED on a transient outage. Mirror check_document_exists here, and pass on_error="assume_exists" at the verification call site; dedupe callers can keep the default assume_missing.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/models/processors.py` around lines 159 - 237, `check_filename_exists`
currently treats final OpenSearch failures as missing, which can incorrectly
fail post-ingest verification; update this helper to support an `on_error`
option like `check_document_exists`, and in the final exception path honor
`on_error="assume_exists"` instead of always returning false. Then update the
verification call site that uses `check_filename_exists` to pass
`on_error="assume_exists"`, while leaving dedupe/default callers on the existing
`assume_missing` behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/models/processors.py`:
- Around line 159-237: `check_filename_exists` currently treats final OpenSearch
failures as missing, which can incorrectly fail post-ingest verification; update
this helper to support an `on_error` option like `check_document_exists`, and in
the final exception path honor `on_error="assume_exists"` instead of always
returning false. Then update the verification call site that uses
`check_filename_exists` to pass `on_error="assume_exists"`, while leaving
dedupe/default callers on the existing `assume_missing` behavior.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 7995aa04-520a-4e34-9a3a-64819cd6c5fd

📥 Commits

Reviewing files that changed from the base of the PR and between bdd873f and 027a6d0.

📒 Files selected for processing (2)

sdks/typescript/tests/integration.test.ts
src/models/processors.py

…to release-saas-ga-0.6.2 (#2021) * fix: error in skip duplicates functionality for folder ingest for connectors (backport of #1941) (#2018) * fix: error in skip duplicates functionality for folder ingest for connectors 75616(#1842) (#1941) * fix: enhance duplicate handling and indexing for connector file uploads * style: ruff autofix (auto) * fix: improve duplicate handling in connector sync functionality * chore: trigger CodeRabbit --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> (cherry picked from commit effaf02) * style: ruff autofix (auto) * fix: remove duplicate test definitions from main merge The automated merge-from-main on this branch concatenated two tests (test_connector_processor_indexes_cleaned_filename, test_langflow_connector_processor_uses_cleaned_filename) that already existed under both names, tripping ruff F811. Keep one copy of each, preferring the version wired with connector_id_exists=True to match current check_document_exists behavior. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> (cherry picked from commit 286e9db) * fix: Ingestion of buckets not working right after connection is tested - CPD #75506 (#1942) (#2019) * fix: update bucket ingestion messages for clarity and consistency * fix: guard bucket pre-selection against connection ID mismatch The defaults query returns the first S3/COS connection regardless of active status. Only pre-select saved bucket_names when the defaults connection_id matches the current connector.connectionId to avoid seeding the wrong buckets if a stale connection exists alongside the active one. * fix: prevent defaults from overwriting user bucket selections Two issues in the initial-selection effect: 1. hasAppliedInitial was only set when valid buckets were found, leaving the effect live across bucket refreshes and allowing stale defaults to overwrite selections made after a refetch. 2. No guard against the async race where buckets loads before initialSelectedBuckets: user clicks buckets, defaults arrive later and silently overwrite them. Fix: mark hasAppliedInitial unconditionally on first evaluation, and only apply defaults when selectedBuckets is still empty (user hasn't acted yet). --------- (cherry picked from commit f3823b8) Co-authored-by: Claude Sonnet 4.6 <[email protected]> (cherry picked from commit 36f115f) * fix: handle orphan files and return appropriate responses when syncing connectors (#1785) (#2016) * feat: handle orphan files and return appropriate responses when syncing connectors * feat: refactor GoogleDriveOAuth to separate required scopes and add handling for missing optional group scopes in tests * style: ruff autofix (auto) * feat: add handling for skipped files and warnings in task processing and UI components --------- (cherry picked from commit aca865a) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> (cherry picked from commit 79b33f0) * fix: enhance error handling for document processing and add OCR requirement check for image files (#1859) (#2017) * fix: enhance error handling for document processing and add OCR requirement check for image files * style: ruff autofix (auto) * remove clickhouse connect from langflow image (#1801) * chore: fix ci (#1889) * Update processors.py * test fix * fix ci --------- (cherry picked from commit a286581) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Lucas Oliveira <[email protected]> Co-authored-by: Edwin Jose <[email protected]> (cherry picked from commit 4c6acd8) --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Lucas Oliveira <[email protected]> Co-authored-by: Edwin Jose <[email protected]>

…model option (#2031) * fix: remediate image_scan CVEs in backend, frontend, and langflow images (#2000) * fix light color text catergory chips in light mode (#2013) Co-authored-by: Olfa Maslah <[email protected]> * fix: upgraded version of docling (#1995) * changed version of docling * fix logs not being the correct ones * go back to sequential building to save space * add cache to builds * remove single use results from docling manager * revert single use results * change the ray tenant id header * added env * changed cache to hit if no changes were made * fix hash for cache * style: ruff autofix (auto) * fixed coderabbit * style: ruff autofix (auto) * added comment on docling manager * style: ruff autofix (auto) --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat: add chunk level page source (#2015) * Changed export docling document to append page number * update result on frontend to append page number * document how to get page number in readmes * Update flows/components/export_docling_document.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update flows/components/export_docling_document.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * fix: apply CodeRabbit auto-fixes Fixed 1 file(s) based on 1 unresolved review comment. Co-authored-by: CodeRabbit <[email protected]> * update component code --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: CodeRabbit <[email protected]> * fix: error in skip duplicates functionality for folder ingest for connectors (backport of #1941) (#2018) * fix: error in skip duplicates functionality for folder ingest for connectors 75616(#1842) (#1941) * fix: enhance duplicate handling and indexing for connector file uploads * style: ruff autofix (auto) * fix: improve duplicate handling in connector sync functionality * chore: trigger CodeRabbit --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> (cherry picked from commit effaf02) * style: ruff autofix (auto) * fix: remove duplicate test definitions from main merge The automated merge-from-main on this branch concatenated two tests (test_connector_processor_indexes_cleaned_filename, test_langflow_connector_processor_uses_cleaned_filename) that already existed under both names, tripping ruff F811. Keep one copy of each, preferring the version wired with connector_id_exists=True to match current check_document_exists behavior. --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: Ingestion of buckets not working right after connection is tested - CPD #75506 (#1942) (#2019) * fix: update bucket ingestion messages for clarity and consistency * fix: guard bucket pre-selection against connection ID mismatch The defaults query returns the first S3/COS connection regardless of active status. Only pre-select saved bucket_names when the defaults connection_id matches the current connector.connectionId to avoid seeding the wrong buckets if a stale connection exists alongside the active one. * fix: prevent defaults from overwriting user bucket selections Two issues in the initial-selection effect: 1. hasAppliedInitial was only set when valid buckets were found, leaving the effect live across bucket refreshes and allowing stale defaults to overwrite selections made after a refetch. 2. No guard against the async race where buckets loads before initialSelectedBuckets: user clicks buckets, defaults arrive later and silently overwrite them. Fix: mark hasAppliedInitial unconditionally on first evaluation, and only apply defaults when selectedBuckets is still empty (user hasn't acted yet). --------- (cherry picked from commit f3823b8) Co-authored-by: Claude Sonnet 4.6 <[email protected]> * fix: handle orphan files and return appropriate responses when syncing connectors (#1785) (#2016) * feat: handle orphan files and return appropriate responses when syncing connectors * feat: refactor GoogleDriveOAuth to separate required scopes and add handling for missing optional group scopes in tests * style: ruff autofix (auto) * feat: add handling for skipped files and warnings in task processing and UI components --------- (cherry picked from commit aca865a) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: enhance error handling for document processing and add OCR requirement check for image files (#1859) (#2017) * fix: enhance error handling for document processing and add OCR requirement check for image files * style: ruff autofix (auto) * remove clickhouse connect from langflow image (#1801) * chore: fix ci (#1889) * Update processors.py * test fix * fix ci --------- (cherry picked from commit a286581) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Lucas Oliveira <[email protected]> Co-authored-by: Edwin Jose <[email protected]> * fix: bump golang.org/x/net to v0.55.0 to remediate CVE-2026-25680 and CVE-2026-39821 (#2014) * feat: add new design for messages and add inline references (#2020) * update opensearch multimodal and flows to support parser, chunk size and chunk overlap * update backend to support citations and chunk metadata * update frontend to support citations and metadata, implement new design for assistant message * changed the prompt for the agent * changed the prompt for the agent * remove hallucinated chunk references * adjust design of citations and where popover appears * adjusted design for message, popover and cards * fix React Doctor picks * fix: apply CodeRabbit auto-fixes Fixed 5 file(s) based on 4 unresolved review comments. Co-authored-by: CodeRabbit <[email protected]> * style: apply biome auto-fixes [skip ci] * updated colors in ibm mode * fixed react doctor picks * added correct type and validation for function calls * remove style for message when on onboarding * fix message not being w full * remove search query in end of the message * change popup to use shadcn popover * fix coderabbit's suggestion * increase prompt limit --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: CodeRabbit <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * apply new settings on retry (#2010) * fix: preserve HTTPException status in update_docling_preset (#1586) (#2004) * fix: report actual synced connection count in connector_sync (#1547) (#2006) * fix: restore LLM model values in reapply_all_settings fallback (#1587) (#2008) * fix: prevent partial matches for exact token searches (#2003) * fix: prevent partial matches for exact token searches (#1937) * style: ruff autofix (auto) * chore: remove unused json import --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: target agent component when updating chat flow system prompt (#2001) * fix: target langflow agent component for system prompt update instead of provider LLM component Signed-off-by: vchen7629 <[email protected]> * fix: removed unused llm_provider arg from _update_langflow_system_prompt Signed-off-by: vchen7629 <[email protected]> * test: add unit tests for update_chat_flow_system_prompt Signed-off-by: vchen7629 <[email protected]> * style: ruff autofix (auto) * fix(test): resolve flow path relative to repo root in test_langflow_agent_system_prompt Signed-off-by: vchen7629 <[email protected]> * style: ruff autofix (auto) --------- Signed-off-by: vchen7629 <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: replaced hardcoded docker.io prefix with a env variable prefix for private repo compatibility (#2002) Signed-off-by: vchen7629 <[email protected]> * fix: resolve Pydantic delta serialization warnings in agent streaming (#2005) * fix: updated agent model dump to exclude delta field Signed-off-by: vchen7629 <[email protected]> * test: added regression unit test to verify response field + no warnings * style: ruff autofix (auto) * test: added additional unit test to verify that model_dump without exclusion raises the warning Signed-off-by: vchen7629 <[email protected]> * style: ruff autofix (auto) * test: tighten delta warning assertion to match specific Pydantic error Signed-off-by: vchen7629 <[email protected]> * style: ruff autofix (auto) --------- Signed-off-by: vchen7629 <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * fix: prevent exception details from leaking in API error responses (#2007) * fix: removed str(e) in exceptions to prevent internals from leaking Signed-off-by: vchen7629 <[email protected]> * style: ruff autofix (auto) * fix: added missing status_code Signed-off-by: vchen7629 <[email protected]> * style: ruff autofix (auto) --------- Signed-off-by: vchen7629 <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Changed docling options to include local support and be shown in the Langflow tab * style: ruff autofix (auto) * enable ollama and delete unused file * fix next lint --------- Signed-off-by: vchen7629 <[email protected]> Co-authored-by: Gautham N Pai <[email protected]> Co-authored-by: Wallgau <[email protected]> Co-authored-by: Olfa Maslah <[email protected]> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: CodeRabbit <[email protected]> Co-authored-by: Rico Furtado <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Edwin Jose <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Mike Fortman <[email protected]> Co-authored-by: hunterxtang <[email protected]> Co-authored-by: NishadA05 <[email protected]> Co-authored-by: Zephyrus <[email protected]>

github-actions Bot added backend 🔷 Issues related to backend services (OpenSearch, Langflow, APIs) bug 🔴 Something isn't working. labels Jul 3, 2026

ricofurtado requested review from lucaseduoli and mpawlow July 3, 2026 18:53

github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Jul 3, 2026

coderabbitai Bot reviewed Jul 3, 2026

View reviewed changes

Merge branch 'main' into cpd-backport-1859-ocr-error-handling

0c8248c

github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Jul 3, 2026

lucaseduoli approved these changes Jul 3, 2026

View reviewed changes

github-actions Bot added the lgtm label Jul 3, 2026

Merge branch 'main' into cpd-backport-1859-ocr-error-handling

b3c816c

github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Jul 3, 2026

Merge branch 'main' into cpd-backport-1859-ocr-error-handling

763f8f0

github-actions Bot added bug 🔴 Something isn't working. and removed bug 🔴 Something isn't working. labels Jul 3, 2026

ricofurtado merged commit 4c6acd8 into main Jul 4, 2026
20 checks passed

github-actions Bot deleted the cpd-backport-1859-ocr-error-handling branch July 4, 2026 00:44

ricofurtado mentioned this pull request Jul 4, 2026

fix: backport connector-ingestion fixes (#2016, #2017, #2018, #2019) to release-saas-ga-0.6.2 #2021

Merged

4 tasks

mpawlow removed their request for review July 4, 2026 19:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: enhance error handling for document processing and add OCR requirement check for image files (backport of #1859)#2017

fix: enhance error handling for document processing and add OCR requirement check for image files (backport of #1859)#2017
ricofurtado merged 4 commits into
mainfrom
cpd-backport-1859-ocr-error-handling

ricofurtado commented Jul 3, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 3, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ricofurtado commented Jul 3, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test plan

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ricofurtado commented Jul 3, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 3, 2026 •

edited

Loading