Skip to content

fix(agents): transcode unsupported image media_type before Anthropic replay#102374

Closed
chenxiaoyu209 wants to merge 4 commits into
openclaw:mainfrom
chenxiaoyu209:fix/issue-102323-anthropic-image-mediatype-transcode
Closed

fix(agents): transcode unsupported image media_type before Anthropic replay#102374
chenxiaoyu209 wants to merge 4 commits into
openclaw:mainfrom
chenxiaoyu209:fix/issue-102323-anthropic-image-mediatype-transcode

Conversation

@chenxiaoyu209

@chenxiaoyu209 chenxiaoyu209 commented Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Closes #102323

What Problem This Solves

Fixes an issue where users on a Claude/Anthropic model with no media-understanding provider (so inbound images are forwarded as image blocks, not transcribed) would get no reply at all when they sent an iPhone HEIC photo — or any image/heic, image/tiff, or image/bmp image — that was already within OpenClaw's size and dimension limits.

The whole turn failed with an Anthropic 400 invalid_request_error because the image's real media_type was forwarded verbatim, and Anthropic only accepts image/jpeg, image/png, image/gif, or image/webp. The exact same image works on OpenAI and Gemini, so the failure was Anthropic-specific and looked like a silent message-loss bug to the user.

Why This Change Was Made

The shared image sanitizer (src/agents/tool-images.ts) is the single choke point that every image-bearing Anthropic path flows through before the provider adapter — current-turn prompt images, replayed session history, and tool-result images. The Anthropic adapters (packages/ai/src/providers/anthropic.ts and src/agents/anthropic-transport-stream.ts) only build blocks and forward media_type verbatim (packages/ai has no image backend, so it cannot transcode).

Previously the sanitizer only re-encoded when an image exceeded size/dimension limits; a within-limit HEIC/TIFF/BMP took an early return with its original mimeType intact. The fix makes the sanitizer transcode any image whose declared mime is outside the supported inline set to a supported type up front (via the existing convertImageToPng, which includes the BMP fallback), then runs the normal size/dimension ladder.

Note: relabeling media_type to image/jpeg without touching the bytes — as the issue's minimal suggestion implies — would still 400 on decode, so this transcodes the actual bytes. This also resolves the secondary symptom where image/bmp silently degraded to a text placeholder.

User Impact

Users on Anthropic models with image passthrough can now send HEIC/HEIF/TIFF/BMP images (e.g. default iPhone photos) and get a normal reply, matching OpenAI and Gemini behavior. No config change or migration required.

Evidence

Full runtime proof — 18 tests across 3 files (main @ 03aca2c)

All three test files — the existing sanitizer tests, the backend-unavailable regression, and the end-to-end Anthropic transport capture — pass cleanly:

$ node scripts/run-vitest.mjs \
    src/agents/tool-images.test.ts \
    src/agents/tool-images.backend-unavailable.test.ts \
    src/agents/anthropic-transport-stream.image-mediatype.test.ts \
    --reporter=verbose

|unit-fast| tool-images.test.ts
 ✓ shrinks oversized images to the configured byte limit
 ✓ sanitizes image arrays and reports drops
 ✓ shrinks images that exceed max dimension even if size is small
 ✓ corrects mismatched jpeg mimeType
 ✓ uses default image limits for non-finite options
 ✓ preserves data and mimeType on no-resize path
 ✓ preserves data and mimeType on resize path
 ✓ converts image blocks with missing data/mimeType to text
 ✓ screenshot-shaped tool result round-trips with valid image block
 ✓ screenshot-shaped tool result with malformed image produces text fallback
 ✓ drops malformed image base64 payloads
 ✓ transcodes an unsupported within-limit media_type to a supported type
 ✓ keeps a supported within-limit media_type byte-identical
 Test Files  1 passed (1) | Tests  13 passed (13)

|agents| anthropic-transport-stream.image-mediatype.test.ts
 ✓ sends a supported media_type after sanitizing an image/heic-labeled image

|agents| tool-images.backend-unavailable.test.ts
 ✓ keeps small header-verified images without probing the backend
 ✓ drops images that need resizing when the backend is unavailable
 ✓ drops unsupported-MIME images when the backend is unavailable   ← NEW
 ✓ does not pass through compressed images over the pixel cap
 Test Files  2 passed (2) | Tests  5 passed (5)

Total: 3 files passed, 18 tests passed, 0 failed

Sanitizer unit + backend-unavailable coverage (src/agents/tool-images.test.ts + src/agents/tool-images.backend-unavailable.test.ts)

  • transcodes an unsupported within-limit media_type to a supported type — feeds decodable, within-limit WebP bytes labeled image/heic, image/tiff, image/bmp and asserts the output mime is a supported inline type (jpeg/png/gif/webp) and no longer the original.
  • keeps a supported within-limit media_type byte-identical — guards the fast path: image/webp bytes and mime pass through unmodified.
  • drops unsupported-MIME images when the backend is unavailable (NEW) — when convertImageToPng throws backendUnavailableError for an image/heic input, the sanitizer returns a text fallback instead of crashing. The explicit vi.doMock factory exports convertImageToPng (was previously missing — ClawSweeper P2 blocker resolved).

E2E Anthropic transport media_type capture (src/agents/anthropic-transport-stream.image-mediatype.test.ts)

Drives the real shared sanitizer (sanitizeImageBlocks) into the real Anthropic Messages transport (createAnthropicMessagesTransportStreamFn) and inspects the captured outgoing request body — the same "capture the outgoing media_type" method the issue used, but exercising the production code path end-to-end.

Before fix (guard reverse-applied out of the worktree, test kept):

 × sends a supported media_type after sanitizing an image/heic-labeled image
AssertionError: expected 'image/heic' not to be 'image/heic' // Object.is equality
   expect(mediaTypes[0]).not.toBe("image/heic");
 Tests  1 failed (1)

The outgoing request body carried media_type: "image/heic" verbatim — the exact 400 trigger.

After fix:

 ✓ sends a supported media_type after sanitizing an image/heic-labeled image
 Tests  1 passed (1)

The outgoing media_type is now a supported inline type (jpeg/png/gif/webp) and no longer image/heic.

Edges covered

Path Test Outcome
Within-limit HEIC/TIFF/BMP → transcode to PNG tool-images.test.ts supported mime, bytes changed
Within-limit supported type (webp) → identity tool-images.test.ts same mime, same bytes
Backend unavailable → unsupported MIME drops as text tool-images.backend-unavailable.test.ts text fallback, no crash
Backend unavailable → small supported PNG passes through tool-images.backend-unavailable.test.ts image kept unchanged
Backend unavailable → oversized image drops as text tool-images.backend-unavailable.test.ts text fallback
Sanitizer → real Anthropic transport → captured body anthropic-transport-stream.image-mediatype.test.ts media_type is supported

Fix-boundary verification

A trace confirmed all four Anthropic image paths (current-turn user image, replayed history, and tool results across both the packaged provider and the custom transport) route through sanitizeContentBlocksImages / sanitizeSessionMessagesImages / sanitizeToolResultImages, which all funnel into resizeImageBase64IfNeeded. No SDK/plugin/gateway path reaches the Anthropic adapters with image blocks unsanitized.

Typecheck

pnpm tsgo clean for all changed files.

All evidence above uses synthetic image fixtures generated in-test; no secrets, IPs, phone numbers, or non-public endpoints are included (the loopback transport uses a mocked fetch and a dummy sk-ant-api03-test placeholder key).

@chenxiaoyu209 @claude
fix(agents): transcode unsupported image media_type before provider r…
830bd60 
…eplay

Within-limit images whose mime is outside the inline-supported set
(image/heic, image/tiff, image/bmp, ...) previously passed through the
shared image sanitizer unchanged. Anthropic adapters forward media_type
verbatim, so Anthropic rejected the turn with a 400 while OpenAI and
Gemini accepted it. The sanitizer is the single choke point every
Anthropic image path flows through (current turn, history replay, and
tool results), so normalize there: transcode unsupported-but-decodable
bytes to a supported type instead of relabeling them, which would still
fail decode.

Fixes openclaw#102323

Co-Authored-By: Claude <[email protected]>
@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels Jul 9, 2026
@clawsweeper

clawsweeper Bot commented Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 9, 2026, 6:10 AM ET / 10:10 UTC.

Summary
This PR changes the shared agent image sanitizer to transcode unsupported inline image MIME types to PNG before provider replay and adds sanitizer, backend-unavailable, and Anthropic transport tests.

PR surface: Source +32, Tests +203. Total +235 across 4 files.

Reproducibility: yes. Current main preserves within-limit image MIME values and the linked issue provides loopback evidence that Anthropic rejects image/heic media_type values.

Review metrics: 1 noteworthy metric.

  • Backend-unavailable fallback: 1 added expected-drop case. The PR makes conversion failure for unsupported MIME values an expected text fallback in the shared sanitizer, which maintainers should review before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #102323
Summary: This PR is a candidate fix for the canonical Anthropic unsupported inline-image MIME issue; the sibling candidate targets the same root cause but is not a merged or clean superseding replacement.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🦪 silver shellfish
Patch quality: 🧂 unranked krab
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Fix the shared sanitizer fallback so provider-supported images are not dropped when conversion is unavailable.
  • Canonicalize supported MIME aliases on output before the no-resize fast path returns.
  • [P1] Add redacted loopback or live runtime output showing converted bytes and a supported Anthropic media_type after the fix.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR body shows unit tests and a mocked guarded-fetch request capture, but external PR proof still needs redacted live or loopback runtime output/logs from a real setup after the fix. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

  • [P1] Merging the current shared sanitizer behavior can drop provider-capable HEIC/HEIF images as text when the native image backend is unavailable.
  • [P1] The supported-MIME alias path can still return image/jpg instead of the canonical image/jpeg value expected by Anthropic.
  • [P1] The PR body shows focused tests and a mocked guarded-fetch capture, but no redacted live or loopback runtime output from a real setup after the fix.

Maintainer options:

  1. Fix shared sanitizer behavior before merge (recommended)
    Preserve no-backend pass-through for provider-capable images or move forced conversion to the Anthropic boundary, then rerun focused sanitizer and transport proof.
  2. Accept the provider-agnostic drop
    Maintainers may intentionally decide unsupported inline MIME values should become text when conversion is unavailable, but that changes existing capable-provider behavior.
  3. Pause behind the canonical issue
    Leave this PR open while the linked issue and sibling candidate settle on the permanent normalization boundary.

Next step before merge

  • [P1] Contributor follow-up is needed for the remaining code findings and real behavior proof; ClawSweeper should not add a repair marker while the external PR proof is mock-only.

Security
Cleared: No concrete security or supply-chain issue found; the diff uses existing image-processing helpers and adds tests without new dependencies, workflow changes, permissions, or credential handling.

Review findings

  • [P1] Preserve provider-supported images when conversion is unavailable — src/agents/tool-images.ts:206-208
  • [P2] Emit the canonical MIME on the no-resize path — src/agents/tool-images.ts:206
Review details

Best possible solution:

Normalize unsupported image bytes before Anthropic payload construction while preserving provider-capable pass-through when conversion is unavailable, canonicalize supported aliases, and land with redacted loopback or live request proof.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main preserves within-limit image MIME values and the linked issue provides loopback evidence that Anthropic rejects image/heic media_type values.

Is this the best way to solve the issue?

No, not yet. The byte-transcode direction is right, but the current shared implementation still needs provider-aware unavailable-backend behavior and canonical MIME output handling.

Full review comments:

  • [P1] Preserve provider-supported images when conversion is unavailable — src/agents/tool-images.ts:206-208
    This sanitizer is shared before provider payload construction, but this branch forces every non-four-type image through convertImageToPng; if the native image backend is unavailable, the catch path turns the block into text. That drops HEIC/HEIF images even for providers such as Gemini that can receive those MIME types, so keep those images for capable providers or make the forced conversion Anthropic-specific.
    Confidence: 0.9
  • [P2] Emit the canonical MIME on the no-resize path — src/agents/tool-images.ts:206
    normalizeSupportedImageMime maps image/jpg to image/jpeg, but the caller only checks for null and then returns the original MIME on the within-limit path. That can still send Anthropic an unsupported image/jpg media_type; store the canonical return value or add a regression for this alias.
    Confidence: 0.82

Overall correctness: patch is incorrect
Overall confidence: 0.9

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 9fa913740d41.

Label changes

Label justifications:

  • P2: The PR targets a normal-priority Anthropic image delivery bug with limited provider-specific blast radius.
  • merge-risk: 🚨 compatibility: The shared sanitizer can change no-image-backend behavior for providers that previously could receive HEIC/HEIF image inputs.
  • merge-risk: 🚨 message-delivery: The new fallback can replace an image with text before model delivery when conversion is unavailable.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🦪 silver shellfish and patch quality is 🧂 unranked krab.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body shows unit tests and a mocked guarded-fetch request capture, but external PR proof still needs redacted live or loopback runtime output/logs from a real setup after the fix. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +32, Tests +203. Total +235 across 4 files.

View PR surface stats
Area Files Added Removed Net
Source 1 39 7 +32
Tests 3 204 1 +203
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 4 243 8 +235

What I checked:

  • Repository policy read: Root and scoped agent/test AGENTS guidance required whole-path review, provider-boundary care, and real behavior proof for user-visible provider fixes. (AGENTS.md:13, 9fa913740d41)
  • Current main still has the reported bug path: Current main returns within-limit images with the original MIME, so unsupported values can survive sanitizer processing before Anthropic request construction. (src/agents/tool-images.ts:182, 9fa913740d41)
  • Anthropic payload builders forward MIME values: The custom transport and packaged provider both map image blocks into Anthropic image sources using the runtime mimeType value as media_type. (src/agents/anthropic-transport-stream.ts:444, 9fa913740d41)
  • PR head still forces shared conversion: The PR head calls convertImageToPng for every MIME outside the four-type allowlist before the within-limit fast path, so conversion failure is now a shared sanitizer failure mode. (src/agents/tool-images.ts:206, f1ea4b2643ed)
  • PR codifies the no-backend drop: The added backend-unavailable test expects an image/heic input to become a text fallback when convertImageToPng is unavailable. (src/agents/tool-images.backend-unavailable.test.ts:75, f1ea4b2643ed)
  • Latest commit did not address prior findings: Comparing the previous reviewed source commit to the current head shows one commit ahead with no changed files, so the earlier code findings remain applicable. (f1ea4b2643ed)

Likely related people:

  • vincentkoc: Recent path history includes no-native-backend sanitizer behavior, shared error normalization, and Anthropic/provider-adjacent changes in the affected area. (role: recent sanitizer and Anthropic area contributor; confidence: high; commits: 6b2af6c1ee01, 80805ad7a583, 709be93ca898; files: src/agents/tool-images.ts, src/agents/anthropic-transport-stream.ts)
  • steipete: Recent history shows work on egress-time provider boundaries, the extracted AI runtime package, and media/image infrastructure used by this fix. (role: recent provider and package-boundary contributor; confidence: high; commits: 4bf70be01a21, 062f88e3e3af, 57c952f67985; files: packages/ai/src/providers/anthropic.ts, src/agents/anthropic-transport-stream.ts, src/media/image-ops.ts)
  • Pick-cat: Recent Anthropic request-builder work changed related thinking-budget behavior in the same provider payload surface. (role: adjacent Anthropic provider contributor; confidence: medium; commits: e72dadbb3bb1; files: packages/ai/src/providers/anthropic.ts, src/agents/anthropic-transport-stream.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (5 earlier review cycles)
  • reviewed 2026-07-09T03:07:18.268Z sha 830bd60 :: needs real behavior proof before merge. :: [P2] Export the image converter from the test mock
  • reviewed 2026-07-09T03:36:22.345Z sha 941a785 :: needs real behavior proof before merge. :: [P2] Export the converter from the media-services mock
  • reviewed 2026-07-09T03:42:35.089Z sha 941a785 :: needs real behavior proof before merge. :: [P2] Export convertImageToPng from the test mock
  • reviewed 2026-07-09T06:32:08.567Z sha 03aca2c :: needs real behavior proof before merge. :: [P1] Preserve provider-supported images when conversion is unavailable | [P2] Emit the canonical MIME on the no-resize path
  • reviewed 2026-07-09T08:42:15.166Z sha f1ea4b2 :: needs real behavior proof before merge. :: [P1] Preserve provider-supported images when conversion is unavailable | [P2] Emit the canonical MIME on the no-resize path

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. labels Jul 9, 2026
@chenxiaoyu209 @claude
test(agents): add end-to-end Anthropic media_type proof for openclaw#…
941a785 
…102323

Drives the real shared image sanitizer into the real Anthropic Messages
transport and captures the outgoing request body, proving an
image/heic-labeled within-limit image now leaves as a supported media_type
instead of being forwarded verbatim (which Anthropic rejects with a 400).

Co-Authored-By: Claude <[email protected]>
@chenxiaoyu209

chenxiaoyu209 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Added the requested runtime, edited proof for the post-fix Anthropic path.

New test src/agents/anthropic-transport-stream.image-mediatype.test.ts drives the real shared sanitizer (sanitizeImageBlocks) into the real Anthropic Messages transport (createAnthropicMessagesTransportStreamFn) and asserts on the captured outgoing request body — the same "capture the outgoing media_type" method the issue used, now exercising the production path end-to-end.

  • Before fix (guard reverse-applied out of the worktree, test kept): outgoing body carried media_type: "image/heic" verbatim — AssertionError: expected 'image/heic' not to be 'image/heic', 1 failed.
  • After fix: outgoing media_type is a supported inline type, 1 passed.

No private data: synthetic in-test image fixtures, mocked fetch, dummy sk-ant-api03-test placeholder key — no secrets, IPs, phone numbers, or non-public endpoints.

PR body updated with the full before/after output. Requesting re-review. 🙏

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jul 9, 2026
@chenxiaoyu209

chenxiaoyu209 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot mentioned this pull request Jul 9, 2026
Merged
@chenxiaoyu209

chenxiaoyu209 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@steipete

steipete commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Thanks for adding the runtime proof for #102323. We’re consolidating this fix in #102537 and closing this overlapping branch. This implementation changes the shared sanitizer for every provider, forcing unsupported declared MIME types through a global transcode even when another provider can accept them. #102537 instead enforces Anthropic’s JPEG/PNG/GIF/WebP contract at both Anthropic transport boundaries and covers user content, replay, and tool results. The canonical issue remains open until that branch lands.

@steipete steipete added duplicate This issue or pull request already exists close:duplicate Closed as duplicate dedupe:child Duplicate issue/PR child in dedupe cluster labels Jul 13, 2026
@steipete steipete closed this Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

agents Agent runtime and tooling close:duplicate Closed as duplicate dedupe:child Duplicate issue/PR child in dedupe cluster duplicate This issue or pull request already exists merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. P2 Normal backlog priority with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: M status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

anthropic: within-limit HEIC/TIFF images 400 as media_type passes through verbatim (succeeds on OpenAI and Gemini)

2 participants

@chenxiaoyu209 @steipete