core: audio and video prompt attachments are silently dropped #54600

Activity

1. opencode-agent commented on Oct 11, 2026

   

   opencode-agentBot

   on Oct 11, 2026 – with OpenCode Agent

   Contributor

   More actions

   Thanks for the clear report. I can confirm this.

   The script below calls toLLMMessages with a user message that has one inline attachment:

   • image/png: the model gets the prompt text and a media part.
   • audio/wav and video/mp4: the model gets only the prompt text. The attachment disappears and nothing reports an error.

   As you found, attachmentContent in packages/core/src/session/runner/to-llm-message.ts returns [] for any MIME type other than text, directories, the image types and PDF.

   This still happens on the latest dev (38454c6). That file hasn't changed since v2.0.26.

   Related:

   • fix(core): send audio and video prompt attachments to the model #54601: your open PR that fixes this
   • Attached AVIF images are silently dropped from model context in V2 #41722: AVIF image attachments are dropped by the same code
   • read: audio files are rejected as binary, so multimodal models never receive them #54491 / feat(read): attach audio and video files for multimodal models #54496: the same audio/video problem in the read tool

   Reproduction script (audio-video-attachments.test.ts)

   // Copy into packages/core/test/ of an opencode checkout and run:
   //   cd packages/core && bun test test/audio-video-attachments.test.ts
   import { expect, test } from "bun:test"
   import { Model } from "@opencode/core/model"
   import { Provider } from "@opencode/core/provider"
   import { SessionMessage } from "@opencode/core/session/message"
   import { Base64, FileAttachment } from "@opencode/schema/prompt"
   import { toLLMMessages } from "@opencode/core/session/runner/to-llm-message"
   import { DateTime } from "effect"
   
   const created = DateTime.makeUnsafe(0)
   const model = Model.Ref.make({ id: Model.ID.make("gemini"), providerID: Provider.ID.make("google") })
   
   for (const [mime, name] of [
     ["audio/wav", "clip.wav"],
     ["video/mp4", "clip.mp4"],
     ["image/png", "control.png"],
   ] as const) {
     test(`${mime} prompt attachment reaches the model`, () => {
       const messages = toLLMMessages(
         [
           SessionMessage.User.make({
             id: SessionMessage.ID.make("msg_user"),
             type: "user",
             text: "Describe this file",
             files: [FileAttachment.make({ data: Base64.make("AAECAw=="), mime, source: { type: "inline" }, name })],
             time: { created },
           }),
         ],
         model,
       )
       console.log(mime, "->", JSON.stringify(messages[0]?.content))
       expect(messages[0]?.content.some((part) => part.type === "media")).toBe(true)
     })
   }

   Run it: bun audio-video-attachments.test.ts
2. opencode-agent

   added

   reproduced

   on Oct 11, 2026
3. dixiatielu commented on Oct 16, 2026

   

   dixiatielu

   on Oct 16, 2026

   More actions

   Also hitting this on v2.0.30 (Linux), end to end against a real Gemini endpoint.

   Setup: custom provider using @ai-sdk/google with a custom baseURL (Gemini-native /v1beta), model declared with modalities.input: ["text","image","audio","video","pdf"]. GET /api/model shows the capabilities correctly.

   I put a logging proxy in front of the endpoint:

   • opencode run -f clip.mp4 / -f sound.mp3 → request contains only the text part (the local file is also sniffed as application/octet-stream, which feat(read): attach audio and video files for multimodal models #54496 addresses)
   • same via POST /api/session/:id/prompt with an explicit data:video/mp4;base64,... / data:audio/mpeg;base64,... → still only the text part
   • -f pic.png → arrives as inlineData as expected

   Sending the same mp4/mp3 directly to the endpoint as inlineData works fine, so it's purely the client-side drop described here.

   #54601 is green and mergeable. Would be great to get it reviewed, since right now the failure is silent and the model just replies that no file was attached.