fix: normalize unsupported Anthropic inline images by TUARAN · Pull Request #102537 · openclaw/openclaw

TUARAN · 2026-07-09T08:10:20Z

What Problem This Solves

Closes #102323. Anthropic accepts only JPEG, PNG, GIF, and WebP image blocks, but OpenClaw could forward within-limit HEIC, TIFF, or BMP MIME values and bytes unchanged. Anthropic rejected the whole turn, so the user received no reply.

Why This Change Was Made

Normalize user and tool-result images at both current package-owned Anthropic payload builders: the packaged provider and Anthropic transport.
Keep @openclaw/ai independent of Rastermill through one optional, source-compatible host port with an inert identity default; OpenClaw core injects the byte-aware implementation.
Detect actual bytes before trusting declared MIME. Preserve supported formats, convert TIFF/HEIC-like inputs to JPEG, and use the existing Photon-capable PNG fallback for BMP.
Preserve non-vision fast paths and explicit omission placeholders without decoding dropped images.
Bound work before and after conversion at OpenClaw's 10 MB decoded-image safety ceiling. Account normalized images incrementally against one 64 MB request-scope safety budget before retaining each converted result.
Keep existing HEIC conversion error identity while adding a generic JPEG encoder for this provider boundary.

The host capability is accepted as the permanent package/core seam: it is optional for published package consumers, while the runtime getter always supplies the inert fallback.

User Impact

Supported-size HEIC/TIFF/BMP images now reach Anthropic in a supported encoded format instead of failing the turn. Existing JPEG/PNG/GIF/WebP and text-only behavior remains unchanged. Oversized or aggregate-abusive inline payloads fail locally before unbounded decode/conversion work.

Evidence

Official contract: current @anthropic-ai/sdk restricts base64 image media types to JPEG, PNG, GIF, and WebP.
node scripts/run-vitest.mjs <package host/provider/transport and core normalizer tests> — 204 tests passed on exact rebased head 50acd5c74db.
Real official Anthropic API proof: a repository image converted locally to TIFF entered OpenClaw as image/tiff, normalized through the core host, and completed on claude-haiku-4-5 with stopReason: stop, 4 output tokens, and no error. Temporary image/script and credential window were removed afterward.
node scripts/check-changed.mjs -- <eleven touched files> — core/core-test typecheck, full core lint, format, import cycles, dependency/API/database guards passed on Node 24.18 using the documented local fallback after repeated Blacksmith status API failures.
node scripts/build-all.mjs — full AI/packages/unified build, runtime postbuild, 140 Plugin SDK export checks, and Control UI build passed.
AutoReview: all in-scope security/compatibility findings were accepted and fixed (pre-decode bounds, non-vision image-only turns, BMP fallback, post-conversion bounds, route-policy separation, request-scope accounting, incremental conversion). Final review clean.
git diff --check passed.

Original fix by @TUARAN; replacement commit preserves Co-authored-by: TUARAN <[email protected]>.

clawsweeper · 2026-07-09T08:31:48Z

Codex review: needs maintainer review before merge. Reviewed July 17, 2026, 12:06 AM ET / 04:06 UTC.

Summary
The PR adds byte-aware Anthropic inline-image normalization for packaged-provider and custom-transport user and tool-result payloads, converting unsupported formats to JPEG before request construction.

PR surface: Source +141, Tests +278. Total +419 across 9 files.

Reproducibility: yes. Current-main behavior is reproducible by sending a within-limit unsupported image through the real Anthropic payload builder and observing the unsupported media_type; the branch's exact-head loopback run proves the corrected outbound payload.

Review metrics: 1 noteworthy metric.

Transport host API surface: 1 optional capability added. The added Anthropic-specific host operation is source-compatible but creates a permanent cross-package contract that needs owner approval.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #102323
Summary: The open issue is the canonical bug report; this PR is the only remaining viable candidate fix, while the other implementation attempts were closed unmerged.

Members:

canonical: anthropic: within-limit HEIC/TIFF images 400 as media_type passes through verbatim (succeeds on OpenAI and Gemini) #102323 - The issue contains the current-main reproduction, expected behavior, affected provider paths, and linked fix context.
superseded: fix: normalize Anthropic unsupported inline images #102339 - The same author closed the earlier implementation and replaced it with this refreshed, broader branch.
duplicate: fix(agents): transcode unsupported image media_type before Anthropic replay #102374 - This closed unmerged PR attempted the same Anthropic unsupported-image fix at the shared sanitizer boundary.
duplicate: fix(anthropic): normalize unsupported image MIME types in sanitizer #102562 - This closed unmerged PR attempted the same fix through the existing image sanitizer.
duplicate: fix(anthropic): normalize unsupported inline image media types before payload #102683 - This closed unmerged PR closely matches the current host-owned normalizer and dual-payload-boundary approach.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦀 challenger crab
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

[P1] Merging establishes an Anthropic-specific optional operation on the shared AiTransportHost contract, so this package-to-core boundary may become a lasting API seam.
[P1] Packaged @openclaw/ai consumers that do not install OpenClaw's host implementation retain the identity normalizer and will now fail locally when the post-normalization MIME is unsupported rather than forwarding the invalid payload to Anthropic.

Maintainer options:

Approve and document the host seam (recommended)
An owner accepts the optional Anthropic normalizer as the permanent package-to-core contract before the PR is merged.
Refactor before merge
Move normalization to an existing generic media boundary and prove packaged-provider, custom-transport, non-vision, and package-only behavior without the new host capability.
Pause the PR
Keep the linked bug and this PR open until the provider-host ownership direction is settled rather than landing an unconfirmed permanent seam.

Next step before merge

[P2] An owner should approve or reject the permanent AiTransportHost capability; no narrow mechanical repair remains.

Maintainer decision needed

Question: Should normalizeAnthropicInlineContentBlocks become the permanent optional AiTransportHost boundary for packaged Anthropic payload normalization?
Rationale: The implementation is correct, source-compatible, and well proven, but repository evidence does not establish whether maintainers want this provider-specific operation on the shared transport host or want normalization moved to an earlier generic media boundary.
Likely owner: steipete — The unresolved choice concerns the permanent package/core architecture and provider host contract rather than a mechanical defect.
Options:
- Approve the host port (recommended): Accept the optional capability as the narrow package-independence seam and retain the current payload and compatibility coverage.
- Move normalization earlier: Require a shared pre-provider media boundary that covers all four Anthropic paths without adding a provider-specific host capability, then repeat fresh-install and packaged-consumer proof.

Security
Cleared: The diff adds no dependency, permission, secret, workflow, publishing, or downloaded-code changes, and the image conversion remains inside the existing media processor boundary.

Review details

Best possible solution:

Keep the byte-aware conversion at the Anthropic payload boundary with one shared implementation, and explicitly approve and lock the optional host-port contract before merging.

Do we have a high-confidence way to reproduce the issue?

Yes. Current-main behavior is reproducible by sending a within-limit unsupported image through the real Anthropic payload builder and observing the unsupported media_type; the branch's exact-head loopback run proves the corrected outbound payload.

Is this the best way to solve the issue?

Yes, technically: provider-boundary byte normalization is narrower and safer than MIME relabeling or a global Anthropic-specific sanitizer rule. The remaining uncertainty is ownership of the new host capability, not the conversion algorithm.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 135aa7a0025d.

Label changes

Label justifications:

P2: This fixes a reproducible Anthropic image-turn failure with limited provider-specific blast radius and no emergency runtime impact.
merge-risk: 🚨 compatibility: The PR adds a provider-specific host capability and changes unsupported-image failure behavior for package consumers without an injected normalizer.
rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦀 challenger crab and patch quality is 🦞 diamond lobster.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): Exact-head live output exercises the real packaged Anthropic provider and custom transport against a loopback capture server, showing unsupported user and tool-result images leave OpenClaw as JPEG and complete successfully.
proof: sufficient: Contributor real behavior proof is sufficient. Exact-head live output exercises the real packaged Anthropic provider and custom transport against a loopback capture server, showing unsupported user and tool-result images leave OpenClaw as JPEG and complete successfully.

Evidence reviewed

PR surface:

Source +141, Tests +278. Total +419 across 9 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	6	179	38	+141
Tests	3	278	0	+278
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	9	457	38	+419

What I checked:

Current-main bug path: The linked issue demonstrates that within-limit HEIC/TIFF blocks reach both Anthropic request builders with unsupported media types and cause the entire turn to fail without a reply. (github.com) (packages/ai/src/providers/anthropic.ts:1478, 135aa7a0025d)
Shared byte-aware normalization: The branch adds one normalizer that canonicalizes base64, detects the actual byte format, preserves supported formats, and converts unsupported or mismatched image bytes to JPEG. (src/media/anthropic-inline-images.ts:1, fca297e7bf00)
Packaged-provider integration: The packaged Anthropic provider normalizes image-bearing content before converting user and tool-result blocks, while retaining the text-only and non-vision fast paths. (packages/ai/src/providers/anthropic.ts:156, fca297e7bf00)
Custom-transport integration: The custom Anthropic transport applies the same normalizer to its image payload construction rather than retaining a separate MIME-only policy. (src/agents/anthropic-transport-stream.ts:341, fca297e7bf00)
Compatibility-sensitive host surface: The branch adds one optional Anthropic-specific capability to AiTransportHost, with an identity implementation for callers that do not inject the OpenClaw media implementation. (packages/ai/src/host.ts:35, fca297e7bf00)
Exact-head real behavior proof: At exact head fca297e7bf0007d60c6fa81c4129a06a31711edf, the real packaged provider and custom transport sent HEIC/TIFF cases as image/jpeg through a loopback Anthropic-compatible capture server and completed with stopReason: stop. (fca297e7bf00)

Likely related people:

steipete: The remaining question is a package-to-core provider boundary decision; Peter owns the repository policy surface and has recent broad agent-runtime architecture history. (role: repository architecture owner; confidence: medium; commits: 860dad268d35; files: AGENTS.md, .github/CODEOWNERS, packages/ai/src/host.ts)
EVA: Eva co-authored recent broad agent-runtime contract work and is a useful secondary routing candidate for the transport-host boundary. (role: adjacent agent-runtime contributor; confidence: medium; commits: 860dad268d35; files: src/agents/, packages/ai/src/)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (11 earlier review cycles; latest 8 shown)

reviewed 2026-07-09T10:06:42.322Z sha a92d003 :: needs changes before merge. :: [P1] Keep the host interface source-compatible
reviewed 2026-07-09T10:16:51.532Z sha a92d003 :: needs changes before merge. :: [P1] Keep the host interface source-compatible
reviewed 2026-07-09T14:49:36.921Z sha a7aa21d :: needs maintainer review before merge. :: none
reviewed 2026-07-09T15:19:11.029Z sha 78fc292 :: needs maintainer review before merge. :: none
reviewed 2026-07-09T15:43:43.609Z sha 78fc292 :: needs maintainer review before merge. :: none
reviewed 2026-07-10T08:37:22.519Z sha fdbf9fc :: needs maintainer review before merge. :: none
reviewed 2026-07-10T11:45:36.147Z sha c4652e1 :: needs maintainer review before merge. :: none
reviewed 2026-07-17T03:26:19.502Z sha fca297e :: needs maintainer review before merge. :: none

TUARAN · 2026-07-09T08:55:10Z

@clawsweeper re-review

clawsweeper · 2026-07-09T08:55:13Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

TUARAN · 2026-07-09T09:59:01Z

@clawsweeper re-review

clawsweeper · 2026-07-09T09:59:03Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

TUARAN · 2026-07-09T15:02:43Z

@clawsweeper re-review

clawsweeper · 2026-07-09T15:02:46Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

TUARAN · 2026-07-10T15:55:27Z

CI is green on c4652e1. The branch compare shows behind_by: 0 and the PR diff is unchanged; GitHub still shows the exact-merge required status as expected/waiting, likely waiting on the external status reporter.

TUARAN · 2026-07-17T01:59:12Z

Refreshed this PR onto current upstream/main and resolved the two overlap conflicts while preserving main's current media-payload guards.

Exact tested head: fca297e7bf0007d60c6fa81c4129a06a31711edf

Verification:

node scripts/run-vitest.mjs packages/ai/src/providers/anthropic.test.ts src/agents/anthropic-transport-stream.test.ts src/media/anthropic-inline-images.test.ts — 184 tests passed across 3 shards
node scripts/run-tsgo.mjs -p test/tsconfig/tsconfig.core.test.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/core-test.tsbuildinfo — passed
node scripts/build-all.mjs — passed, including tsdown, plugin assets, SDK export guard, runtime postbuild, and Control UI
exact-head loopback payload proof — packaged user HEIC, packaged tool-result TIFF, and custom-transport user HEIC all left OpenClaw as image/jpeg; all three returned stopReason: stop
fresh autoreview — clean, no accepted/actionable findings (correctness 0.88)

The loopback proof used the real packaged streamAnthropic() path and real custom Anthropic transport against a temporary local Anthropic-compatible capture server. The PR body contains the full six-field real behavior proof.

Final-head CI: Real behavior proof and check-lint pass. check-test-types currently fails only in test/scripts/release-candidate-checklist.test.ts because current main imports isDirectReleaseCandidateExecution while scripts/release-candidate-checklist.d.mts does not declare that export; this PR does not touch that release surface. The touched core test-types lane passes locally with the command above.

Please re-review this exact head.

clawsweeper · 2026-07-17T01:59:15Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The durable review comment was published and its verdict was queued for target-serialized routing.
Run: https://github.com/openclaw/clawsweeper/actions/runs/29553115352
Updated: 2026-07-17T03:48:20.429Z

Co-authored-by: TUARAN <[email protected]>

clawsweeper · 2026-07-21T18:17:08Z

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix: normalize unsupported Anthropic inline images This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

steipete · 2026-07-21T18:31:29Z

Merged via squash.

Prepared head SHA: 50acd5c74db0bde5fed6acc5b1aaa6e464e84525
Landed commit: 3c32f327a445c9bff90ca812d281330a3a64472c

Co-authored-by: Peter Steinberger <[email protected]>

openclaw-barnacle Bot added agents Agent runtime and tooling size: M labels Jul 9, 2026

TUARAN force-pushed the fix/anthropic-heic-tiff-media-type branch from a82e4c5 to 4317cdd Compare July 9, 2026 08:52

TUARAN mentioned this pull request Jul 9, 2026

fix: normalize Anthropic unsupported inline images #102339

Closed

clawsweeper Bot mentioned this pull request Jul 9, 2026

fix(anthropic): normalize unsupported image MIME types in sanitizer #102562

Closed

clawsweeper Bot added the merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. label Jul 9, 2026

TUARAN force-pushed the fix/anthropic-heic-tiff-media-type branch 2 times, most recently from 0e3c3df to a92d003 Compare July 9, 2026 09:50

clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jul 9, 2026

clawsweeper Bot mentioned this pull request Jul 9, 2026

fix(agents): transcode unsupported image media_type before Anthropic replay #102374

Closed

vincentkoc mentioned this pull request Jul 9, 2026

fix(anthropic): normalize unsupported inline image media types before payload #102683

Closed

TUARAN force-pushed the fix/anthropic-heic-tiff-media-type branch from a92d003 to 2021524 Compare July 9, 2026 14:15

openclaw-barnacle Bot added app: web-ui App: web-ui size: XL and removed size: M labels Jul 9, 2026

TUARAN force-pushed the fix/anthropic-heic-tiff-media-type branch from 2021524 to fe8aa29 Compare July 9, 2026 14:19

TUARAN requested a review from a team as a code owner July 9, 2026 14:19

github-actions Bot added the dependencies-changed PR changes dependency-related files label Jul 9, 2026

openclaw-barnacle Bot added gateway Gateway runtime cli CLI command changes labels Jul 9, 2026

openclaw-barnacle Bot added the size: M label Jul 9, 2026

github-actions Bot removed the dependencies-changed PR changes dependency-related files label Jul 9, 2026

TUARAN force-pushed the fix/anthropic-heic-tiff-media-type branch 3 times, most recently from e5f4c46 to 78fc292 Compare July 9, 2026 15:02

clawsweeper Bot added the merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. label Jul 10, 2026

TUARAN force-pushed the fix/anthropic-heic-tiff-media-type branch from c4652e1 to a172460 Compare July 17, 2026 01:58

clawsweeper Bot added rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jul 17, 2026

steipete self-assigned this Jul 21, 2026

fix(anthropic): normalize unsupported inline images

50acd5c

Co-authored-by: TUARAN <[email protected]>

steipete force-pushed the fix/anthropic-heic-tiff-media-type branch from fca297e to 50acd5c Compare July 21, 2026 18:14

openclaw-barnacle Bot added size: L and removed agents Agent runtime and tooling size: M labels Jul 21, 2026

steipete merged commit 3c32f32 into openclaw:main Jul 21, 2026
199 of 203 checks passed

github-actions Bot mentioned this pull request Jul 21, 2026

📡 Upstream Digest — 2026-07-21 19:05 UTC curtismercier/openclaw-mods#1366

Open

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 22, 2026

fix(anthropic): normalize unsupported inline images (openclaw#102537)

352c1e0

Co-authored-by: Peter Steinberger <[email protected]>

Uh oh!

Uh oh!

Conversation

TUARAN commented Jul 9, 2026 • edited by steipete Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TUARAN commented Jul 9, 2026

Uh oh!

clawsweeper Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TUARAN commented Jul 9, 2026

Uh oh!

clawsweeper Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TUARAN commented Jul 9, 2026

Uh oh!

clawsweeper Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TUARAN commented Jul 10, 2026

Uh oh!

TUARAN commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented Jul 21, 2026

Uh oh!

Uh oh!

steipete commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

TUARAN commented Jul 9, 2026 •

edited by steipete

Loading

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading

TUARAN commented Jul 17, 2026 •

edited

Loading

clawsweeper Bot commented Jul 17, 2026 •

edited

Loading