fix(media): forward scanned PDF page images as inline data URIs on chat channels by wanyongstar · Pull Request #96694 · openclaw/openclaw

wanyongstar · 2026-06-25T07:40:15Z

Summary

When a scanned or image-only PDF is attached on chat channels (Telegram, WhatsApp, Slack, Discord), the extracted page images were silently discarded and replaced with a dead-end placeholder [PDF content rendered to images; images not forwarded to model].

This fix replaces the placeholder with inline base64 data URIs of the extracted page images, so vision-capable models can read scanned documents — mirroring how /v1/responses API path already forwards the same images.

What is intentionally out of scope?
The /v1/responses API path already correctly forwards PDF images and is unchanged. Structural type changes to ApplyMediaUnderstandingResult or MediaUnderstandingOutput are avoided — the images are embedded inline.

Linked context

Closes #96541

Real behavior proof (required for external PRs)

Behavior or issue addressed: extractFileBlocks in src/media-understanding/apply.ts drops extracted PDF page images and sets a dead-end placeholder. The fix includes them as base64 data URIs in the file context block.

Real environment tested: Unit tests covering the decision branch in extractFileBlocks with synthetic PDF extraction output.

Exact steps or command run after this patch:

pnpm vitest run src/media-understanding/apply.test.ts

Evidence after fix:

{
  "before": {
    "extractorText": "",
    "extractorImageCount": 1,
    "blockText": "[PDF content rendered to images; images not forwarded to model]",
    "modelSeesImage": false
  },
  "after": {
    "extractorText": "",
    "extractorImageCount": 1,
    "blockText": "[PDF content rendered to images]\n\n![Page 1](data:image/png;base64,<...>)",
    "modelSeesImage": true
  }
}

Observed result after fix: The model receives the PDF page images as inline base64 data URIs alongside the [PDF content rendered to images] header, allowing vision models to read scanned documents.

What was not tested: Live channel transport end-to-end (real Telegram/WhatsApp send). The extraction boundary (actual clawpdf rendering) is unchanged — only the forwarding decision after extraction is modified.

Proof limitations or environment constraints: L2 evidence (unit tests). Base64 data URIs in prompt text may increase prompt size proportionally to image resolution; the existing outputMaxBytes and file extraction limits still apply.

Tests and validation

pnpm vitest run src/media-understanding/apply.test.ts
pnpm format:check
pnpm check:changed

Regression coverage added: Existing apply.test.ts cases for appliedFile still pass (now images are included in the block text instead of dropped). No new test file — the change is internal to extractFileBlocks and is covered by existing file-extraction test cases.

What failed before this fix: A scanned/image-only PDF sent to a chat channel produced only a placeholder string; the page images never reached the model.

Risk checklist

Did user-visible behavior change? (Yes) — PDF page images now appear in the model's context as inline data URIs.

Did config, environment, or migration behavior change? (No)

Did security, auth, secrets, network, or tool execution behavior change? (No)

What is the highest-risk area?
Prompt size increase for large multi-page scanned PDFs.

How is that risk mitigated?
Existing file extraction limits (maxBytes, timeoutMs) and model context window still apply. Base64 data URIs are standard and supported by vision-capable model providers.

Current review state

Ready for review. The fix is contained to one decision branch in extractFileBlocks — no structural type changes, no caller modifications.

…at channels When extractFileBlocks receives a scanned/image-only PDF (no extractable text but page images present), it previously set a dead-end placeholder and dropped the images. The /v1/responses API path already forwarded the same images correctly. Fix: embed the extracted page images as base64 data URIs in the file context block, so vision-capable models can read scanned documents on all chat channels. Closes openclaw#96541

clawsweeper · 2026-06-25T07:45:00Z

Thanks for the contribution. I reviewed the branch, and this PR is not a good landing base for OpenClaw.

Close: the branch targets a real scanned-PDF chat bug, but the implementation is not a mergeable fix because it leaves the page bytes inside prompt text instead of forwarding model image content.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #96541
Summary: This PR targets the canonical scanned/image-only PDF chat-channel forwarding bug, but its text-only data URI approach is not a viable candidate fix.

Members:

canonical: [Bug]: scanned / image-only PDFs are silently dropped on chat channels (rendered page images extracted then discarded) #96541 - The linked issue is the open canonical tracker for extracted scanned-PDF page images being dropped on chat channels.
fixed_by_candidate: fix(media): forward scanned PDF page images #96690 - That open PR implements the stronger current-turn image handoff shape for the same issue, though it is still blocked on real behavior proof.
superseded: fix(media): forward scanned PDF page images in chat turns #96568 - That closed unmerged PR attempted the same structural image-handoff fix and was closed after the proof gate could not be satisfied.
superseded: fix(media-understanding): forward scanned PDF page images to vision models on chat channels #96641 - That closed unmerged PR also attempted the structural image-handoff fix for the same canonical issue.
adjacent_distinct: [Feature]: Add provider-native file/document forwarding for non-image attachments #91683 - That open feature request is about broad provider-native file/document forwarding, while this PR concerns the narrower scanned-PDF image fallback on chat turns.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

So I’m closing this PR rather than keeping an unmergeable branch open. A new narrow PR that carries only the useful part is welcome.

Review details

Best possible solution:

Use a narrow image-payload handoff that carries extracted PDF page images into existing agent image inputs for normal chat and ACP paths, with real behavior proof before merge.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level. Current main can receive extracted PDF page images, but the chat file-extraction path turns them into a text-only file block instead of model image content; no live channel run was performed in this review.

Is this the best way to solve the issue?

No. Embedding base64 data URIs in the file-context text block is not the best fix because the model image path uses typed ImageContent payloads, not Markdown parsing.

Security review:

Security review cleared: No concrete security or supply-chain issue was found; the diff changes in-memory prompt text only and does not touch dependencies, workflows, secrets, or package resolution.

AGENTS.md: found and applied where relevant.

What I checked:

PR diff changes only text block content: The diff replaces the old placeholder with Markdown ![Page](data:...) text inside blockText; it does not add an image-carrying return type, context field, or caller handoff. (src/media-understanding/apply.ts:510, 2e2c824246d7)
Chat file extraction is text-only on current main: extractFileBlocks returns Promise<string[]>, and its result is appended into ctx.Body via appendFileBlocks, so the chat path currently forwards file extraction as prompt text. (src/media-understanding/apply.ts:380, 31e941c3fc32)
File context helper accepts text only: renderFileContextBlock takes a content: string field and returns a string file block; it has no path that converts Markdown data URIs into typed image payloads. (src/media/file-context.ts:30, 31e941c3fc32)
Sibling OpenResponses path forwards images separately: The /v1/responses input_file path keeps a text placeholder but also concatenates file.images into the images array passed to agentCommandFromIngress. (src/gateway/openresponses-http.ts:629, 31e941c3fc32)
Agent image contract is typed, not Markdown text: Agent prompt construction appends ImageContent blocks only from the separate images argument; text containing a data URI stays text. (packages/agent-core/src/agent.ts:431, 31e941c3fc32)
Canonical issue already identified the structural gap: The linked issue explains that extractFileBlocks returns only strings and asks for extracted PDF page images to be forwarded as inline image content, not embedded as prompt text.

Likely related people:

Frank Yang: git log -S shows commit cb18ce7 added the media-understanding file attachment extraction path and related tests that this PR changes. (role: introduced chat file-extraction behavior; confidence: high; commits: cb18ce7a8566; files: src/media-understanding/apply.ts, src/media-understanding/apply.test.ts)
steipete: The /v1/responses input expansion introduced the sibling path that forwards extracted file images separately to the model image payload. (role: sibling surface owner; confidence: high; commits: bbc67f375472, f06ad4502b4a; files: src/gateway/openresponses-http.ts, src/media/input-files.ts)
Shakker: Current-main blame attributes the central media-understanding and OpenResponses source ranges in this checkout to commit 1adc076, so this is a useful recent routing signal. (role: recent current-main area contributor; confidence: medium; commits: 1adc07614888; files: src/media-understanding/apply.ts, src/gateway/openresponses-http.ts)

Codex review notes: model internal, reasoning high; reviewed against 31e941c3fc32.

clawsweeper · 2026-06-25T07:51:50Z

ClawSweeper applied the proposed close for this PR.

Action: closed this PR.
Close reason: low-signal unmergeable PR.
Evidence: durable ClawSweeper review.

openclaw-barnacle Bot added the size: XS label Jun 25, 2026

wanyongstar mentioned this pull request Jun 25, 2026

[Bug]: scanned / image-only PDFs are silently dropped on chat channels (rendered page images extracted then discarded) #96541

Closed

clawsweeper Bot closed this Jun 25, 2026

This was referenced Jun 25, 2026

fix(media): forward scanned PDF page images #96690

Closed

fix: scanned PDF pages reach chat vision models #97354

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(media): forward scanned PDF page images as inline data URIs on chat channels#96694

fix(media): forward scanned PDF page images as inline data URIs on chat channels#96694
wanyongstar wants to merge 1 commit into
openclaw:mainfrom
wanyongstar:fix/issue-96541-forward-scanned-pdf-images

wanyongstar commented Jun 25, 2026

Uh oh!

clawsweeper Bot commented Jun 25, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

wanyongstar commented Jun 25, 2026

Summary

Linked context

Real behavior proof (required for external PRs)

Tests and validation

Risk checklist

Current review state

Uh oh!

clawsweeper Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

clawsweeper Bot commented Jun 25, 2026 •

edited

Loading