Repository navigation
feat(ai): support Chat and Responses audio input - #58884
rekram1-node wants to merge 2 commits into
Conversation
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Audio lowering looks right. I compiled requests for Meta Responses, Alibaba Responses and Alibaba Chat with user audio, tool-result audio and URL audio. Each produced the request format described, and tool audio moved into a following user message where needed. The packages/ai tests and typecheck:tsc pass. I couldn't make live provider calls.
Public API: no change to the SDK, server, plugin or config. Inside packages/ai, OpenResponses.ProviderAdapter.lowerMedia now returns an Effect, and all adapters in this repo are updated.
One behaviour change to be aware of: every route that builds requests through OpenAIChat.fromRequest (OpenRouter, Groq, DigitalOcean, Venice, OpenAI-compatible) now sends input_audio to the provider instead of rejecting audio locally. Providers without audio support will now return their own error instead of our clear local one.
The new Meta and Alibaba Responses request formats, and Alibaba's tool-audio move, have no tests; only the Chat test was updated. Could you add one small Responses test that checks user and tool audio for Alibaba (and Meta's nested format)? That would protect the code that is new in this PR. See also the inline comment about the copied loop.
| from: Effect.fn("AlibabaResponses.fromRequest")(function* (req) { | ||
| const opts = yield* ProviderShared.validateWith(Schema.decodeUnknownEffect(Options))(req.providerOptions ?? {}) | ||
| const body = yield* OpenResponses.fromRequestWithAdapter(req, adapter) | ||
| // Model Studio accepts audio only in user messages, not function outputs. |
There was a problem hiding this comment.
This loop is the same as the one in meta-responses.ts; only the moved part types and the placeholder text differ. Could it become one helper in OpenResponses, for example hoistToolMedia(input, predicate, placeholder), that both adapters call? Otherwise the two copies will drift apart, for example if one gets a fix for how placeholders handle mixed text and media.
There was a problem hiding this comment.
The new test covers user and tool-result audio for Meta and Alibaba Responses, as I asked in my earlier review. It is small and easy to follow. openai-responses.test.ts (157 tests) and typecheck:tsc pass. The public API is unchanged. Nothing outside the tests changed, so the inline thread about the copied loop is still open.
Issue for this PR
None.
Type of change
What does this PR do?
Chat rejects audio, while Meta and Alibaba Responses send it as ordinary files. I added audio input for user messages and tool results using each API's native representation.
MIME aliases normalize to format names, such as
audio/mpegtomp3. Provider-specific handling stays in the adapters, including moving tool audio to user input where required. Existing xAI file handling stays unchanged.Core forwarding and audio output are out of scope.
How did you verify your code works?
bun run checkand AI-packagebun run typecheck:tscpass.Screenshots / recordings
Not a UI change.
Checklist
Added by the opencode bot (AI-generated).
Related: #54600, #54491