You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 504b470
Browse filesBrowse the repository at this point in the historyBrowse files
gateway: images a tool returned reach a Chat model after the tool messages (charan-rathore in #167: Claude Code's tool_result images were dropped on the way to Chat Completions)
A tool result holding images, as Claude Code's screenshot tools and its Read of a .png send it (an Anthropic tool_result with image blocks) or as a Responses function_call_output with input_image parts, lost its images whenever the request was translated: the result was read as its text only, so a Chat model was told "Read shot.png" and never shown it, and a result that was only an image came through empty. #167 turned such a request away with a 400; now the images go through. The shared request keeps a tool result's images beside its text. Chat's tool messages hold text only, so a tool message says "[The tool returned an image; it follows in the next message.]" after its text, and the images go right after the run of tool messages, each result's labelled with the tool and call it came from ("[From the result of Read (tool call toolu_1):]"), as image_url parts (a data: URL for inline ones). When the user's own message comes next, as Claude Code's system reminder beside a result does, the images open that message rather than being a second user message in a row, which some models' chat templates turn away; otherwise they are a user message of their own, before the assistant's next turn. Tool messages still directly follow the assistant's tool_calls. An Anthropic upstream gets the images inside its tool_result, a Responses one in the function_call_output's output list, Code Assist as inlineData after the functionResponse, as Gemini CLI sends a file it read; Kiro, Cursor and Devin take the text as before. A model that can't see still has each image described in its place before translating, so nothing follows its tool message.
Tested end to end against fake upstreams: Claude Code's tool_use then a tool_result of text and a base64 PNG to a Chat model that sees (with and without the user's text after it), two parallel results with three images (base64 and a URL) giving tool, tool, then one user message with all three labelled, a screenshot from an older turn kept before the assistant's answer, Codex's function_call_output with input_image to Chat, Claude Code to a Responses provider, and a text-only model still given the description in the tool message. The four Chat tests fail without the change. go build, vet and the gateway tests pass; windows and linux (CGO off) build.
0 commit comments