fix(channels/whatsapp): send ElevenLabs voice notes as Ogg/Opus and sniff audio mime (#6116)#6118
Merged
Merged
Conversation
…niff audio mime ElevenLabs TTS returns MP3 by default, but the WhatsApp gateway hard-coded the outgoing mime as `audio/ogg; codecs=opus`, so the payload (MP3) did not match the declared container. Mobile WhatsApp tolerated it, but the mautrix-whatsapp/Beeper bridge rejected the message as "Unsupported content type in Web mode" — a silent drop with no voice bubble. Request Opus from ElevenLabs and stop mislabelling the buffer: - New `[tts.elevenlabs].output_format` config field (default `opus_48000_32`); the synth path now sends `?output_format=` with a 3-tier resolution (tool-arg override > config > built-in default) and labels `TtsResult.format` from the real format prefix. - The ElevenLabs media driver defaults to `opus_48000_32` too; the TTS tool injects the configured `output_format` on the media path when the call omits it. - The WhatsApp gateway gains a `detectAudioMime` magic-byte sniffer; `sendAudio` sends the detected mime and auto-downgrades PTT to a plain audio attachment (with a warn) for any non-Opus buffer instead of corrupting the message. Tests: tts 3-tier resolution + opus-label unit tests; 9 `detectAudioMime` gateway cases. Config baseline regenerated for the new example block. Recovers the fix from the stale-closed PR #5283 by @f-liva.
houko
enabled auto-merge (squash)
June 15, 2026 09:29
github-actions
Bot
added
area/runtime
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #6116. Recovers the fix from the stale-closed PR #5283 (credit: @f-liva), re-derived cleanly on current
main.ElevenLabs TTS returns MP3 by default, but the WhatsApp gateway hard-coded the outgoing mime as
audio/ogg; codecs=opus, so the payload (MP3) did not match the declared container.Mobile WhatsApp tolerated it, but the mautrix-whatsapp / Beeper bridge rejected the message as
Unsupported content type in Web mode— a silent drop, no voice bubble.Change
[tts.elevenlabs].output_formatconfig field (defaultopus_48000_32). The synth path (tts.rs) sends?output_format=with a 3-tier resolution (tool-arg override > config > built-in default) and labelsTtsResult.formatfrom the real format prefix instead of a hardcoded"mp3".mp3_44100_128toopus_48000_32; the TTS tool (tool_runner/media.rs) injects the configuredoutput_formaton the media path when the call omitsformat.detectAudioMime(magic-byte sniff: OggS / ID3 / MPEG-sync / RIFF / fLaC / ftyp);sendAudiosends the detected mime and auto-downgrades PTT to a plain audio attachment (with a warn) for any non-Opus buffer instead of mislabelling it.librefang.toml.exampledocuments the new block;xtask/baselines/config.sha256regenerated.Verification
cargo check -p librefang-types -p librefang-runtime -p librefang-runtime-media --lib— clean.cargo clippy -p librefang-types -p librefang-runtime -p librefang-runtime-media --all-targets -- -D warnings— 0 warnings.cargo test -p librefang-runtime --lib tts::— 14 passed, incl. newtest_three_tier_resolution_elevenlabs_formatandtest_default_elevenlabs_format_label_is_opus.cargo test -p librefang-runtime-media— 83 passed.node --test packages/whatsapp-gateway/index.test.js— the newdetectAudioMimesuite is 9/9 green. (Two pre-existing failures in unrelatedforward_dispatch/onOwnerNoticesuites reproduce on cleanmainand are untouched by this change.)Behaviour note
Operators whose custom TTS pipelines relied on the ElevenLabs MP3 default must now pass
format: "mp3_44100_128"explicitly (or set[tts.elevenlabs].output_format). This is intentional — Opus is required for WhatsApp PTT.