Skip to content

fix(rag): stop indexing on a broken vision model and name it - #7154

Merged
kovtcharov-amd merged 4 commits into
amd:mainfrom
ityashag:fix/vlm-failure-names-model
Nov 11, 2026
Merged

kovtcharov-amd merged 4 commits into
amd:mainfrom
ityashag:fix/vlm-failure-names-model

Conversation

@ityashag

Copy link
Copy Markdown
Contributor

Summary

A vision model that is installed but crashing no longer turns a scanned PDF into blank pages. Indexing stops after the model has failed twice in a row without reading any image, and the error names the model and its last error. Before, every page waited out the failure in turn (minutes for a 10-page scan), and the run ended with a hint to download the model that was already there.

Why

The issue's Strix Halo box reset the Qwen3-VL backend on every image. Each page's failure became a page warning, so the document indexed as text-only, or failed as "no extractable text, enable VLM image extraction by downloading …", which sends the user after the wrong fix. The bot's suggested fix on the issue (stop once the model has failed with nothing read, and name it) is what this does.

Linked issue

Closes #4741

Changes

  • PDFs and PowerPoints stop on a broken vision model. Two failures in a row with no image read raise an error that names the model and its last error, and says to check Lemonade's log and index again. A model that reads some images and fails on others still indexes, with those pages in degraded_pages, as before.
  • The PowerPoint-to-PDF path doesn't fall back and ask the same broken model again.
  • index_document returns error_type: "vision_model_failed", and the agent tool's hint says the model failed, not the file, instead of "suggest they check the file".
  • The PDF page loop catches only vision errors. PDFium errors already have their own handler, so the broad catch was only hiding bugs.
  • Docs: the RAG SDK guide and the RAG SDK and tools specs describe the new error.

Test plan

Run in a python:3.12 container:

  • pytest tests/unit/rag/test_failing_vision_model.py: 7 passed. It indexes real reportlab PDFs and python-pptx decks with a vision model that fails, works, or fails after working.
  • pytest tests/unit/test_rag_file_resolution.py::TestIndexDocumentVisionModelFailed: passes; the tool keeps the error and doesn't blame the file.
  • pytest tests/unit/rag tests/unit/test_vlm_extraction_failures.py tests/unit/test_rag_file_resolution.py tests/unit/test_vision_uses_downloaded_model.py tests/test_rag.py: 909 passed, 35 skipped.
  • Full pytest tests/unit: 38,761 passed, 590 skipped, 15 failed, of 39,270 collected, with no crashed workers. The 15 fail the same way on main in this container: 14 shell-guardrail and skill-grant tests refuse gh and npm because neither is installed there, and one expects a folder it can't create, but the container runs as root. 44 tests have :: inside their parameters, so they were rerun by file to account for them.
  • python util/lint.py --all: passes.

The agent tool's reply for this failure gains a hint, so rag_quality is the eval category to watch on the next RC.

Evidence

The same 10-page scan, indexed with a vision model that fails every image with backend_watchdog_reset:

# main
vision model asked: 10 times
error: No extractable text in PDF: scan.pdf
  ...
  2. Or enable VLM image extraction by downloading Qwen3-VL-4B-Instruct-GGUF in Lemonade

# this PR
vision model asked: 2 times
error_type: vision_model_failed
error: Could not read the images in scan.pdf: the vision model Qwen3-VL-4B-Instruct-GGUF failed 2 times in a row without reading any of them.
Last error: Vision extraction failed for page 2, image 1: Backend error: backend_watchdog_reset
Check Lemonade's log for errors from that model, fix it or restart Lemonade, then index this file again.
  • Agent exposed in the Agent UI: N/A
  • MCP tools / servers: N/A
  • CLI: N/A
  • HTTP API / REST: N/A

Checklist

  • I have linked a GitHub issue above (Closes #N / Fixes #N / Refs #N).
  • I have described why this change is being made, not just what changed.
  • I have run linting and tests locally (python util/lint.py --all, pytest tests/unit/).
  • I have attached real-world evidence matched to the surface I changed (see Evidence above), or marked each surface N/A.
  • I have updated documentation if user-visible behavior changed (see CONTRIBUTING.md).

A vision model that was installed but crashing turned a scanned PDF into
blank pages: every page waited out the failure in turn, and the run ended
with a hint to download the model that was already there. Indexing a PDF or
PowerPoint now stops once the model has failed twice in a row without
reading any image, with an error naming the model and its last error.
index_document reports it as error_type "vision_model_failed", and the
agent tool says the model failed, not the file.

Closes amd#4741
@github-actions github-actions Bot added documentation Documentation changes rag RAG system changes tests Test changes performance Performance-critical changes agents labels Nov 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Verdict: Approve with suggestions

If the vision model is installed but crashes, indexing a scanned PDF or PowerPoint now stops after two images fail with nothing read. Before, it waited out the failure on every page and ended with advice to download a model that was already installed. The error now names the model and its last error, and the agent tells the user the model failed, not the file. Nothing blocks the merge. The two suggestions below are small.

  • When a folder is indexed, each scanned file still asks the broken model twice before it fails. This could stop at the first file, the way an embedder outage already does. That can be a follow-up.
  • One line in the tools spec puts the new error code under the wrong field. A one-line fix is below.

Real-world evidence

N/A: this runner produced no evidence bundle. The PR description has a before/after run of the SDK on a 10-page scan. On main the vision model was asked 10 times and the run ended with the download hint. With this PR it was asked twice and returned error_type: vision_model_failed, naming the model. No reply from the Agent UI chat was captured, so how the agent words this error to the user rests on reading the code and the unit tests. I couldn't run the new tests here (pytest isn't installed on this runner).

🔍 Technical details

Issues

🟢 Indexing a folder asks the broken model twice per scanned file (src/gaia/agents/tools/rag_tools.py:~2857)
The index_directory loop already breaks on embedding_unavailable because "every remaining file would fail the same way". A broken vision model fails every remaining scan the same way, so a folder of 20 scans still makes 40 failed calls and lists 20 identical reasons. Breaking on vision_model_failed would finish the job. The auto-index branch near line 1459 also wraps this error as a generic "auto-indexing failed: …" and drops the new hint. The message survives, so this is optional.

🟢 Error code documented under the wrong field (docs/spec/rag-tools-mixin.mdx:357)
The continuation comment hangs "vision_model_failed" off next_step, but it is an error_type value. For this error the tool also returns hint, not next_step.

                "error_type": str,   # "embedding_unavailable" on an embedder
                                     # outage; "vision_model_failed" when the
                                     # vision model read none of the images
                "next_step": str     # with how to fix an embedder outage

Notes, not findings

  • The PDF loop now catches only VLMExtractionError (it used to catch Exception). I checked what can still raise there. extract_images_from_page_pdfium turns decode errors into PdfImageExtractionError, which is caught inside. VLMClient.extract_from_image wraps everything except LeaseWaitCancelled into VLMExtractionError. So no ordinary page failure escapes the loop, and a cancelled lease still propagates.
  • A page with several images, where the first is read and a later one fails, counts as a failure with nothing read, because extract_from_page_images raises before it returns. This is an edge case: such a page now counts toward the stop limit, even though one of its images was read.
  • index_document catches VisionModelFailedError ahead of the generic handler. The PPTX→PDF fallback re-raises it instead of asking the model again, and a test covers that.
  • The tool hint changes what the LLM sees, so rag_quality on the next RC is the right category to watch, as the PR says.

Strengths

  • The tests index real reportlab PDFs and python-pptx decks instead of mocking the loop. They cover four cases: every image fails, one failure followed by reads, a read followed by failures, and the converted-PDF path.
  • The limit only fires when nothing was read, so a model that works but is flaky still produces a degraded index, as before.
  • Moving the VLMExtractionError import to module level removes a local import, and the error keeps its cause through raise … from.

A page whose second image failed counted as the vision model reading
nothing, though it read the images before it, so a model that works but
fails on some images could stop indexing as broken. The tools spec now
lists vision_model_failed under error_type.
@ityashag

Copy link
Copy Markdown
Contributor Author

Applied two of the suggestions in ee13333:

  • A page whose later image fails no longer counts as "nothing read": the model read the images before it, so a flaky model on multi-image pages keeps indexing with those pages degraded.
  • The tools spec now lists vision_model_failed under error_type.

I left folder indexing as it is. Stopping the folder on this error would also skip its text files and text-heavy PDFs, which index fine without the vision model, so the user would lose documents that work to save a few seconds per scan.

🔍 Technical details
  • _VisionAttempts.record_failure counts a read when the failing VLMExtractionError has image_num > 1. New test: every page's second image fails, so all four pages are degraded and the run doesn't stop. It fails without the change.
  • index_directory keeps going after vision_model_failed, unlike embedding_unavailable, which fails every file the same way.
  • python util/lint.py --all passes. Full pytest tests/unit: 38,762 passed, 590 skipped, 15 failed of 39,271 collected; the same 15 fail on main in this container (missing gh/npm, and running as root).

@ityashag

Copy link
Copy Markdown
Contributor Author

The one failing check, GAIA CLI on Linux, isn't from this change: add-apt-repository can't reach the Launchpad API (launchpadlib fails in login_anonymously with a timeout or RemoteDisconnected). The same step has failed on main and on another PR since 12:05 UTC, so a re-run should pass once Launchpad answers again.

🔍 Runs and one gap in the outage check
Time (UTC) Branch Result
11:52 main passed
12:05 feat/rag-lemonade-rerank-375 failed
12:32 this PR failed
12:40 main failed
13:27 this PR failed

The step reports this as "add-apt-repository failed … for a reason other than a Launchpad outage". diagnose_launchpad_outage only recognizes GPGKeyTemporarilyNotFoundError and Failed to fetch …ppa.launchpadcontent.net/lemonade-team, so a connection failure while add-apt-repository talks to the Launchpad API isn't named as an outage. I can open a small fix for that, with a test, if it's useful.

@ityashag

Copy link
Copy Markdown
Contributor Author

@kovtcharov-amd this is ready for your review. The bot's suggestions are applied in ee13333. The one red check, GAIA CLI on Linux, is the Launchpad outage that is also failing main: api.launchpad.net still wasn't answering at 14:05 UTC. #7163 makes that check name the outage.

@kovtcharov-amd
kovtcharov-amd added this pull request to the merge queue Nov 11, 2026
Merged via the queue into amd:main with commit a03b255 Nov 11, 2026
45 of 46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents documentation Documentation changes performance Performance-critical changes rag RAG system changes tests Test changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(rag): a failing VLM silently indexes PDF images as blank

2 participants