Skip to content

fix: scanned PDF pages reach chat vision models#97354

Merged
joshavant merged 2 commits into
mainfrom
fix/scanned-pdf-chat-images
Jun 28, 2026
Merged

fix: scanned PDF pages reach chat vision models#97354
joshavant merged 2 commits into
mainfrom
fix/scanned-pdf-chat-images

Conversation

@joshavant

@joshavant joshavant commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Closes #96541

AI-assisted by Codex.

What Problem This Solves

Fixes an issue where users sending scanned or image-only PDFs through chat channels would have those rendered PDF page images extracted by media understanding but silently dropped before the reply model saw them.

The affected workflow is chat-channel replies with file attachment extraction enabled, especially scanned PDFs that have no extractable text and need vision input.

Why This Change Was Made

This threads rendered PDF page images from media-understanding file extraction into the normal reply image pipeline instead of treating them as text-only extraction side effects. It preserves attachment ordering across described media, rendered file pages, and native current-turn images, and applies the same behavior to the ACP dispatch path.

The stale placeholder that said rendered PDF images were not forwarded is removed from runtime output and docs. Local PDF extraction also now honors the existing input-file timeout limit so a stalled extraction path does not hang indefinitely.

User Impact

Users can attach scanned or image-only PDFs in chat channels and expect vision-capable reply models to receive the rendered pages. The model now has access to the actual PDF page images instead of only seeing a placeholder that the PDF was rendered.

Evidence

  • Focused tests passed: pnpm test src/auto-reply/reply/get-reply.message-hooks.test.ts src/media/input-files.fetch-guard.test.ts src/media/pdf-extract.test.ts extensions/document-extract/document-extractor.test.ts src/media-understanding/apply.test.ts src/media-understanding/media-understanding-misc.test.ts src/auto-reply/reply/current-turn-images.test.ts src/auto-reply/reply/agent-runner.media-paths.test.ts src/auto-reply/reply/dispatch-acp.test.ts -- --reporter=dot passed 5 Vitest shards / 281 tests.
  • git diff --check passed.
  • pnpm changed:lanes --json reported core, coreTests, and docs.
  • pnpm check:changed passed on Blacksmith Testbox tbx_01kw3ve1hw53mx08yn0t05egst (swift-crayfish), exit 0. This covered the code/test diff before the final docs-only edit.
  • Docs validation after the docs edit passed: pnpm docs:list and pnpm docs:check-mdx.
  • Autoreview passed with no accepted/actionable findings.
  • Linux source extraction Crabbox proof passed on Azure Crabbox cbx_38ab3f65a3e0, run run_ee855ee7e830: rendered PDF fallback produced imageCount=1, placeholder [PDF content rendered to images], and no stale images not forwarded placeholder.
  • Live Telegram post-fix proof passed on Azure Crabbox cbx_19bc9a4ae812, run run_fb1b4f306692: real Telegram telegram credential, sent scanned PDF message 25170, and observed LIVE_PROOF_PASS issue=96541 transport=telegram credentialKind=telegram imageInputCount=1.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation size: L maintainer Maintainer-authored PR labels Jun 28, 2026
@clawsweeper

clawsweeper Bot commented Jun 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 27, 2026, 11:53 PM ET / 03:53 UTC.

Summary
The PR threads rendered scanned-PDF page images from media-understanding file extraction into normal chat and ACP reply image attachments, updates placeholder/docs wording, resolves managed inbound media URIs in the media cache, and adds a PDF extraction timeout.

PR surface: Source +227, Tests +200, Docs 0. Total +427 across 15 files.

Reproducibility: yes. Source inspection reproduces the bug path with high confidence: current main and v2026.6.10 receive PDF page images from extraction but the chat media-understanding result only forwards a placeholder text block.

Review metrics: 1 noteworthy metric.

  • Parallel Fix Candidates: 2 open sibling PRs, 1 closed superseded PR. Multiple branches target the same scanned-PDF bug, so maintainers should select one canonical implementation before merge.

Stored data model
Persistent data-model change detected: persistent cache schema: src/media-understanding/media-understanding-misc.test.ts, serialized state: src/auto-reply/reply/current-turn-images.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #96541
Summary: This PR is one candidate fix for the canonical scanned-PDF chat delivery bug, with multiple sibling PRs attempting the same root-cause repair.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Choose the canonical scanned-PDF forwarding branch and rerun focused media/reply/ACP validation on that exact branch.

Mantis proof suggestion
A real Telegram lane is useful because the changed behavior is model-visible scanned-PDF attachment delivery from a chat message. A maintainer can ask Mantis to capture proof by posting this exact PR comment:

@openclaw-mantis telegram live: verify a scanned PDF sent in Telegram reaches the reply model as one image input and does not drop native image attachments.

Risk before merge

  • [P1] Multiple open PRs target the same scanned-PDF delivery bug, so maintainers need to choose one canonical branch before merge.
  • [P1] The PR changes model-visible image attachment delivery and ordering for normal chat and ACP turns; mistakes here would alter message context delivered to the model.
  • [P2] Local PDF extraction now uses the existing input-file timeout budget instead of being able to run indefinitely, which maintainers should accept deliberately.

Maintainer options:

  1. Land One Canonical Typed Path (recommended)
    Choose this PR or another proof-positive typed-image branch, verify normal and ACP ordering on that branch, and close the duplicate candidates after merge.
  2. Accept The Timeout Behavior
    Maintainers can explicitly accept that local PDF extraction now obeys the existing input-file timeout to avoid stalled turns.
  3. Pause If A Sibling Lands First
    If fix(media): forward scanned PDF page images #96690 lands first with equivalent normal and ACP forwarding, close this PR as superseded rather than carrying another branch.

Next step before merge

  • [P2] Protected maintainer handling and overlapping proof-positive PRs make this a branch-selection decision rather than an automated repair task.

Security
Cleared: No concrete security or supply-chain concern found; the diff adds no dependencies or workflow changes, and managed media URI resolution remains constrained by inbound-reference parsing plus allowed-root checks.

Review details

Best possible solution:

Land one canonical typed image-forwarding fix that preserves current image, described-media, and recent-history semantics, then close the duplicate candidate PRs and the linked bug from the merged implementation.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection reproduces the bug path with high confidence: current main and v2026.6.10 receive PDF page images from extraction but the chat media-understanding result only forwards a placeholder text block.

Is this the best way to solve the issue?

Yes. The typed extractedFileImages handoff through existing normal and ACP image payload paths is the right fix shape; the remaining question is which overlapping branch maintainers want to land.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 830467bc9306.

Label changes

Label justifications:

  • P2: The PR fixes a bounded chat media-understanding bug for scanned PDF attachments without a core outage or security emergency signal.
  • merge-risk: 🚨 message-delivery: The patch changes which PDF-derived image attachments are delivered into normal chat and ACP model turns, so mistakes would affect model-visible message context.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes focused tests, source-extraction Crabbox output, and live Telegram output showing a scanned PDF produced one model image input after the fix.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes focused tests, source-extraction Crabbox output, and live Telegram output showing a scanned PDF produced one model image input after the fix.
  • mantis: telegram-visible-proof: Mantis should capture Telegram visible proof. The PR changes user-visible Telegram scanned-PDF attachment delivery and is suitable for short live Telegram proof.
Evidence reviewed

PR surface:

Source +227, Tests +200, Docs 0. Total +427 across 15 files.

View PR surface stats
Area Files Added Removed Net
Source 9 305 78 +227
Tests 5 202 2 +200
Docs 1 1 1 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 15 508 81 +427

Acceptance criteria:

  • [P1] pnpm test src/auto-reply/reply/get-reply.message-hooks.test.ts src/media/input-files.fetch-guard.test.ts src/media/pdf-extract.test.ts extensions/document-extract/document-extractor.test.ts src/media-understanding/apply.test.ts src/media-understanding/media-understanding-misc.test.ts src/auto-reply/reply/current-turn-images.test.ts src/auto-reply/reply/agent-runner.media-paths.test.ts src/auto-reply/reply/dispatch-acp.test.ts -- --reporter=dot.
  • [P1] pnpm check:changed.

What I checked:

  • Root policy applied: Read the full root policy; it requires whole-surface review, sibling path checks, source/current behavior proof, and message-delivery merge-risk treatment for this PR. (AGENTS.md:1, 830467bc9306)
  • Current-main chat bug: Current main turns extracted PDF page images into the stale placeholder and returns only file text blocks, so chat reply code has no image payload to forward. (src/media-understanding/apply.ts:496, 830467bc9306)
  • Sibling response behavior: The OpenResponses path already adds a rendered-PDF placeholder and also concatenates file.images into model image inputs. (src/gateway/openresponses-http.ts:612, 830467bc9306)
  • Latest release still affected: v2026.6.10 has the same placeholder-only chat behavior while OpenResponses forwards file.images, so the bug is not already fixed in the shipped release. (src/media-understanding/apply.ts:43, aa69b12d0086)
  • PR head typed handoff: The PR adds extractedFileImages to ApplyMediaUnderstandingResult and collects extracted PDF images with attachment indexes before returning them to callers. (src/media-understanding/apply.ts:382, 0b9c184b7d13)
  • PR head normal reply handoff: resolveCurrentTurnImages accepts extractedFileImages, strips metadata, and merges them with native current-turn images by attachment index where known. (src/auto-reply/reply/current-turn-images.ts:143, 0b9c184b7d13)

Likely related people:

  • joshavant: CONTRIBUTING lists Josh Avant for Core/CLI/Gateway/Security/Agents, and git blame ties the current placeholder block to commit 898ca97. (role: listed core area owner and current line author; confidence: high; commits: 898ca9741cba; files: src/media-understanding/apply.ts, src/auto-reply/reply/current-turn-images.ts, src/auto-reply/reply/dispatch-acp.ts)
  • Frank Yang: Commit cb18ce7 added the media-understanding file attachment extraction path and tests that this PR extends. (role: introduced chat file-extraction behavior; confidence: high; commits: cb18ce7a8566; files: src/media-understanding/apply.ts, src/media-understanding/apply.test.ts)
  • Peter Steinberger: Commit bbc67f3 expanded the OpenResponses input path that already forwards extracted PDF file images. (role: adjacent sibling-surface contributor; confidence: high; commits: bbc67f375472; files: src/gateway/openresponses-http.ts, src/media/input-files.ts)
  • NianJiu: Commit 55a0c9b restored WebChat image understanding routing and touched current-turn image handling adjacent to this PR. (role: adjacent chat image contributor; confidence: medium; commits: 55a0c9b1f474; files: src/media-understanding/apply.ts, src/auto-reply/reply/current-turn-images.ts)
  • Alex Knight: Commit e850750 recently changed managed inbound PDF extraction on the same media-understanding apply path. (role: recent media-understanding contributor; confidence: medium; commits: e850750754f2; files: src/media-understanding/apply.ts, src/media/input-files.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@joshavant
joshavant force-pushed the fix/scanned-pdf-chat-images branch from 13fcd41 to 7082091 Compare June 28, 2026 03:32
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. mantis: telegram-visible-proof Mantis should capture Telegram visible proof. labels Jun 28, 2026
@clawsweeper clawsweeper Bot added P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. labels Jun 28, 2026
@joshavant
joshavant merged commit 07b9349 into main Jun 28, 2026
165 of 168 checks passed
@joshavant
joshavant deleted the fix/scanned-pdf-chat-images branch June 28, 2026 04:01
This was referenced Jun 28, 2026
Closed
Closed
Closed
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 28, 2026
@joshavant
fix: scanned PDF pages reach chat vision models (openclaw#97354)
faaeb4b 
* fix: forward scanned PDF page images to chat models

* fix: remove stale reply option cast
@github-actions github-actions Bot mentioned this pull request Jun 28, 2026
Open
QiuYuang pushed a commit to QiuYuang/openclaw that referenced this pull request Jul 1, 2026
@joshavant @QiuYuang
fix: scanned PDF pages reach chat vision models (openclaw#97354)
3e6806e 
* fix: forward scanned PDF page images to chat models

* fix: remove stale reply option cast
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 1, 2026
@joshavant chenyangjun-xy
fix: scanned PDF pages reach chat vision models (openclaw#97354)
d434dc2 
* fix: forward scanned PDF page images to chat models

* fix: remove stale reply option cast
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 3, 2026
@joshavant @chenyangjun-xy
fix: scanned PDF pages reach chat vision models (openclaw#97354)
4c03f44 
* fix: forward scanned PDF page images to chat models

* fix: remove stale reply option cast
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

docs Improvements or additions to documentation maintainer Maintainer-authored PR mantis: telegram-visible-proof Mantis should capture Telegram visible proof. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: L status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: scanned / image-only PDFs are silently dropped on chat channels (rendered page images extracted then discarded)

1 participant

@joshavant