gateway: preserve Gemini PDF and audio file data on text-only routes - #95

Thanks! The idea is right: a PDF or audio fileData isn't an image. But I pulled the branch and traced where the part goes after textOnlyBody, and the fix only covers half of the path.

parseGemini (internal/gateway/gemini.go:150) still turns every fileData into Part{Kind: Image}. So a Gemini request with a PDF, routed to a text-only model on another protocol (the usual case for a text-only route), is now translated into an image_url and sent to that text-only model. On your branch:

blocked=false
chat={"messages":[{"content":[{"text":"read this","type":"text"},{"image_url":{"url":"gs://bucket/file.pdf"},"type":"image_url"}],"role":"user"}],...}

Main rejects that turn cleanly as an image the route can't take, or falls back to a model that can. With the patch it becomes an upstream 400 from the text-only model instead.

To merge this, parseGemini also needs to stop treating a non-image fileData/inlineData as an image (so no image_url is built for it). A test should cover the translated body as well as textOnlyBody's result. Happy to take a v2.

Thanks for v2. The Chat side is fixed now: no more image_url for a PDF. But moving the check into parseGemini breaks the one upstream that can read the file: Gemini CLI / Antigravity accounts (Code Assist), which buildCodeAssist rebuilds from the IR.

Same request, {"fileData":{"mimeType":"application/pdf","fileUri":"gs://b/f.pdf"}}, through parseGemini → buildCodeAssist:

main:    "parts":[{"text":"read"},{"fileData":{"fileUri":"gs://b/f.pdf","mimeType":"application/pdf"}}]
this PR: "parts":[{"text":"read"},{"text":"[attachment application/pdf: gs://b/f.pdf]"}]

So a Gemini model that used to receive the PDF now gets a line of text instead.

Suggested fix: keep the file in the IR (as a Part with its MediaType and URL, or a new file kind). Decide at build time: buildCodeAssist keeps emitting fileData/inlineData for it, while the Chat / Responses / Anthropic builders turn a non-image/* media type into the [attachment …] text. Please add a buildCodeAssist case to your translation test alongside the Chat one.

Thanks for v3. The Code Assist path is fixed now. I pulled it and checked:

• PDF and audio fileData / inlineData reach Code Assist with their mimeType and URI or data intact.
• Chat, Anthropic, Responses and Kiro get [attachment …] text instead of an image.
• The text-only gate still blocks images and unknown file types.
• go vet and go test -race pass.

One regression is left. renderParts in internal/gateway/claude_subscription.go has no case File. renderClaudePrompt uses it for the Claude, Cursor, Grok and Devin subscriptions, so the attachment now drops out of the prompt entirely: with a PDF, the prompt is just Human: read. On main, non-image inlineData at least became [attachment audio/wav] text.

Please:

1. Add case File: text.WriteString(attachmentText(p)) there, with a test that renderClaudePrompt keeps the placeholder.
2. Rebase on current main. buildResponses now takes a host argument (buildResponses(r, model, host, rejectTemp)), so the new test at image_input_test.go:447 needs buildResponses(r, "text", "", false).

After that I'm happy to merge.

Thanks — v4 addresses both points. I pulled it, merged it onto current main, and checked:

• renderClaudePrompt keeps the [attachment …] placeholder. I reverted the fix locally and the new test fails, so it guards the regression.
• go vet and go test -race ./internal/gateway pass.
• No other case Image consumer drops File.

Merged; it will ship in the next release.

Released in v0.1.152: https://github.com/yetone/magpie-releases/releases/tag/v0.1.152