Same root cause as #5284:

• synthesize_elevenlabs() (crates/librefang-runtime/src/tts.rs ~178) doesn't accept a format_override parameter and forces self.config.elevenlabs.output_format unconditionally. OpenAI and Google TTS in the same module DO accept the override. As written this:
  • breaks API-contract parity — format_override works for openai/google but is silently ignored for elevenlabs;
  • nullifies #5284's tool-call-arg path for the ElevenLabs provider specifically.
    Fix: add format_override: Option<&str> to the elevenlabs signature, use it in the request-build (after the config-default resolution).

Two nits:

• detectAudioMime (packages/whatsapp-gateway/index.js ~3368) matches the OggS container magic and then asserts audio/ogg; codecs=opus — but the container marker alone doesn't prove Opus codec; Ogg/Vorbis would false-positive. Probably acceptable for WhatsApp PTT (Vorbis would fail downstream anyway), but a one-line comment explaining "container-only detection assumes Opus per config context" would prevent confusion.
• 8 unit tests for detectAudioMime in isolation, but no integration test asserting synthesize_elevenlabs with default config produces a buffer that detectAudioMime reads as Opus. Worth a #[tokio::test] that mocks the ElevenLabs response with Opus bytes and asserts TtsResult.format == "opus".

The fix strategy (request Opus at source rather than transcode) is the right call.

Thanks for the thorough fix — the root-cause analysis (MP3 bytes labelled audio/ogg; codecs=opus rejected by mautrix-whatsapp) is convincing, and the magic-byte sniffer + auto-downgrade in sendAudio is the right shape. The code itself reads correctly. Three CI blockers need resolving before this can merge, though:

1. Schema baseline drift (OpenAPI Drift job, fail). TtsElevenLabsConfig derives schemars::JsonSchema (crates/librefang-types/src/config/types.rs:1283), so adding the new output_format field (line 1298) changes the config JSON schema and therefore xtask/baselines/config.sha256. CI's auto-commit path can't run on a fork PR (IS_INTERNAL_PR: false), so this needs to be regenerated locally and committed:

   cargo xtask schema-check gen
   

   then commit the updated xtask/baselines/config.sha256 (and any openapi.json / sdk/ deltas it produces).

2. Formatting (Quality job, fail on cargo xtask fmt). The reported diffs are all in files this PR doesn't touch (crates/librefang-cli/src/main.rs, crates/librefang-cli/src/tui/event.rs, crates/librefang-migrate/src/openclaw.rs, crates/librefang-types/src/config/types.rs:6542). That points at a stale base — the branch is on 4bc691f while main is now at 9db55b6, and main has since reformatted those files. A rebase on current origin/main should clear it.

3. Test / Windows (shard 1/2) fail. It ran ~45 min (vs ~12s for shard 2/2), so it looks like a timeout/flake rather than a logic failure, but please re-check after the rebase.

One non-blocking note on correctness, just to confirm intent: TtsResult.format is now derived from the output_format prefix (tts.rs:251-256) instead of the hard-coded "mp3". That value flows into tool_runner/media.rs:470 as the saved-file extension (music_{timestamp}.{format}), so the default now yields …​.opus. That's the intended/correct behavior given the container, just flagging it as a downstream effect of the change.

Once the baseline is regenerated and the branch is rebased, this should be good to go.

Review

The core fix is sound — this is a real source-side fix (request Opus from ElevenLabs), not a mislabel. Requesting changes for one genuine CI item plus two minor nits. The other red checks are stale-base.

Required: regenerate config.sha256 (real drift, not stale-base)

The OpenAPI Drift check is a genuine failure caused by this PR. The PR edits librefang.toml.example (adds the [tts.elevenlabs] block) but does not regenerate the schema baseline that hashes that file:

• xtask/baselines/config.sha256 still pins 4f12fa2a07651bb76ce35378a3cb4126b964e4b953c74c52871d20e7ba49c37e librefang.toml.example
• actual hash of the file in this branch is a0df9a053023011bf36dacd7f5a179c59f39570fc873df7d8f2f27e384d81f9e

The baseline surface list is in crates/librefang-api/tests/config_schema_overlay.rs:47-51 (config → librefang.toml.example → xtask/baselines/config.sha256). Fix:

cargo xtask schema-check gen

then commit the updated xtask/baselines/config.sha256 (and any openapi.json / sdk/* the regen touches). This is in-scope for the PR since the PR is what changed the hashed file.

Stale-base (rebase clears these — not code defects)

• Test / Ubuntu (shard 1/4): fails on every_kernel_config_struct_field_is_exposed_via_overlay (config_schema_overlay.rs:287) and comms_send_rejects_oversize_message (network_routes_integration.rs:430). Both files are byte-identical to origin/main on this branch (git diff origin/main HEAD is empty for them), and they're unrelated to the audio diff. The branch merged main at 5875b13d, which is one commit behind origin/main (9f38b30a). These are the known stale-base signatures.
• Quality (rustfmt): every Diff in … line is in a file NOT in this PR (triggers.rs, routes/agents.rs, routes/skills.rs, server.rs, agent_execution.rs, …). Whole-file fmt drift from the stale base, not the audio change.

Please rebase onto current origin/main so the test/fmt lanes clear, and regenerate the config baseline as above.

Logic — correct approach

• tts.rs:194-197 now appends ?output_format={output_format} (default opus_48000_32), so ElevenLabs returns real OGG/Opus bytes. This is the right fix — it does not relabel MP3 as Ogg; it changes what the API actually emits.
• tts.rs:251-257 derives the format label from the requested prefix (opus_48000_32 → "opus"), so the label tracks the real container.
• index.js detectAudioMime sniffs magic bytes and only keeps ptt: true when the buffer is genuinely Ogg/Opus, downgrading to a plain attachment otherwise (sendAudio). Good defense-in-depth; no silent corruption.
• Nice interaction with the existing ffmpeg path: tool_runner/media.rs:711 skips convert_to_ogg_opus when format is already opus, so requesting Opus at the source avoids a redundant transcode. Coherent.

Minor nits (in-scope, please address)

1. File extension .opus vs .ogg — tool_runner/media.rs:750 builds the filename as tts_{ts}.{format}, so an ElevenLabs opus_48000_32 result is saved as *.opus. ElevenLabs returns an OGG container (the buffer starts with OggS, which is exactly what detectAudioMime keys on), so the conventional extension is .ogg. .opus is a tolerated Ogg/Opus extension and the gateway sniffs bytes (not extension), so this isn't a functional break — but the saved filename misrepresents the container. Consider mapping the opus prefix to an ogg extension, or document the choice.
2. Ogg/Vorbis false-positive is acknowledged in the detectAudioMime comment (index.js). Asserting codecs=opus on any OggS buffer is acceptable under the current TTS envelope, but if a non-Opus Ogg ever reaches this path it would be mislabeled. Fine to leave given the comment; flagging for visibility.

Once rebased + config.sha256 regenerated, this looks good to merge.

The audio fix itself is correct — config field plumbing is complete (config/types.rs:1283-1320: struct-level #[serde(default)], all derives, output_format in the Default impl), synthesize_elevenlabs threads format_override and derives the label from the prefix (tts.rs:179-256), the gateway sniffer downgrades non-Opus PTT, and config.sha256 is regenerated.

Blocking on PR hygiene, not code: #5283 and #5284 contain duplicate copies of this entire change. git log origin/main.. on each branch shows both carry a fix(audio): … commit (38879423 here vs 0b0bb622 on #5284) and a plumb format_override commit, and #5284's diff is a strict superset of this one — the same tts.rs / config/types.rs / librefang.toml.example / index.js / config.sha256 hunks. Whichever merges first will conflict the other.

Please pick one path: either (a) close this and let #5284 (which subsumes it) carry the whole change, or (b) keep #5283 as the base and properly rebase #5284 onto this branch so #5284's diff narrows to just the media.rs tool_text_to_speech delta. Happy to re-review once the duplication is resolved.

Non-blocking nits (acknowledged, fine to leave): .opus extension on an OGG container (media.rs:743/776); the Ogg/Vorbis false-positive in detectAudioMime.

Rebase required + one type-contract bug in the format_override path.

1. Rebase needed (blocking)

The branch's merge base is 34c44b62 and origin/main is now 14 commits ahead of that point. The Quality CI run (job/77488129989) fails with six clippy errors, none of which are in any file this PR touches:

• crates/librefang-api/src/routes/registry.rs:287 — needless borrow on .strip_prefix(&home_dir)
• crates/librefang-api/src/validation.rs:468-471 — doc list items without indentation
• crates/librefang-api/src/webchat.rs:277 — manual char comparison

These were all fixed in main by 72c1f60d (fix(api): clear clippy Quality lane) which landed after the PR's Merge branch 'main' into pr/audio-mime-fix commit (3e7faa7f). The merge only pulled up to 34c44b62. The fix: rebase (or merge again) onto current origin/main — the clippy errors are not yours to resolve, they are already resolved upstream, you just need to pick them up.

2. format_override / ElevenLabs output_format type-contract mismatch (correctness bug)

tts.rs:182-193: synthesize_elevenlabs receives format_override: Option<&str> and forwards it verbatim as the ElevenLabs ?output_format= query parameter, which expects compound values like opus_48000_32, mp3_44100_128, etc.

The call site at tool_runner/media.rs:581 passes the tool input's format field:

// tool_runner/media.rs:514, 581
let format = input["format"].as_str();   // user-supplied; e.g. "mp3", "opus"
let result = engine.synthesize(text, voice, format).await?;

If a caller passes format: "mp3" (the valid OpenAI/Google label), the ElevenLabs path constructs ?output_format=mp3, which is not a recognised ElevenLabs format string — the API will reject it.

The happy path (no tool-level format override → config default opus_48000_32) is correct. But the override channel is broken for ElevenLabs: a bare label that is valid for OpenAI becomes an invalid compound key for ElevenLabs.

Suggested fix: In synthesize_elevenlabs, reject or remap a format_override that lacks the ElevenLabs compound structure (e.g. no _<rate> suffix), or document that format overrides are not forwarded to the ElevenLabs path and pass None unconditionally, letting only the new config field ([tts.elevenlabs] output_format) drive the choice.

Code review (what is correct)

• The root-cause diagnosis is accurate: ElevenLabs returns MP3 by default, and sending MP3 bytes labelled audio/ogg; codecs=opus breaks the Beeper/mautrix-whatsapp bridge.
• detectAudioMime magic-byte table in index.js is correct: OggS prefix, ID3 header, 0xFF 0xEx MPEG sync, RIFF, fLaC, ftyp-at-offset-4. The comment acknowledging OggS ≠ Ogg/Opus but explaining why it is safe under this gateway's operating envelope is appropriate.
• The auto-downgrade (effectivePtt = ptt && isOggOpus) with a console.warn is a safe fallback rather than a silent corrupt.
• TtsElevenLabsConfig.output_format field is properly added to the struct, Default impl, and has #[serde(default)] on the parent struct — consistent with the CLAUDE.md convention for config fields.
• The split('_').next() prefix derivation correctly maps opus_48000_32 → "opus", mp3_44100_128 → "mp3", etc. for TtsResult.format.
• Tests pin the 3-tier resolution chain and the format-label derivation; the JS test suite covers all 8 detectAudioMime cases.

Summary: The logic is sound on the no-override path that fixes the reported bug. Two things need addressing before merge: (1) rebase onto current main to pick up the upstream clippy fix that unblocks Quality CI, and (2) guard or document the format_override parameter in synthesize_elevenlabs so a bare label like "mp3" cannot silently produce an invalid ElevenLabs API call.

Thanks for the careful root-cause writeup and the layered defense (TTS encoder fix plus gateway-side magic-byte sniffer) — the diff shape is exactly right. Two blockers and one substantive should-fix below; once cleared this is ready to merge.

Blocking

• New tests violate -D warnings — clippy fails the workspace lint with 6 fresh unnecessary_literal_unwrap hits on the test scaffolding you added:

  • crates/librefang-runtime/src/tts.rs:420 Some("pcm_16000") + unwrap_or(...)
  • crates/librefang-runtime/src/tts.rs:425,434 None: Option<&str> + unwrap_or(...) (x2)
  • crates/librefang-runtime/src/tts.rs:448 Some("arg_voice") + unwrap_or(...)
  • crates/librefang-runtime/src/tts.rs:453,459 None: Option<&str> + unwrap_or(...) (x2)

  Reproduces locally with cargo clippy -p librefang-runtime --all-targets -- -D warnings. The intent is good — pin the 3-tier resolution — but clippy folds the literal-Some/literal-None unwrap_or chain at compile time so the assertion no longer exercises the real call-site logic. Easiest fixes:

  1. Wrap the constructed Option through a #[inline(never)] fn black_box(x: Option<&str>) -> Option<&str> { x } (or std::hint::black_box) to defeat the lint and the const-fold simultaneously.
  2. Extract the 3-tier resolution from synthesize_elevenlabs into a private free fn (resolve_format(override_: Option<&str>, cfg: &TtsElevenLabsConfig) -> &str) and have both the synth call and the tests use it. This is the better answer — the current tests "mirror the resolution logic directly" which is exactly the regression mode the comment warns against (someone changes the call-site, tests still pass).

• CONFLICTING against main — packages/whatsapp-gateway/index.test.js collides with the trailing describe block from main (visible via git merge-tree). Please rebase onto current origin/main; the only real conflict is at the bottom of the test file and will resolve by keeping both describe blocks. The xtask/baselines/config.sha256 line will also need regeneration after rebase.

Should fix

• Parallel ElevenLabs driver still defaults to MP3 — crates/librefang-runtime-media/src/elevenlabs.rs:115 reads request.format.as_deref().unwrap_or("mp3_44100_128"). This driver is preferred over TtsEngine in the actual hot path: crates/librefang-runtime/src/tool_runner/media.rs:518-573 tries MediaDriverCache first and only falls back to TtsEngine when no media driver is configured. With media drivers configured (the common case for any deploy that's wired up crates/librefang-runtime-media), the runtime-media driver wins, hands MP3 bytes to the gateway, and the new sniffer correctly downgrades ptt=true to a regular audio attachment — the user-visible voice-note bubble silently doesn't appear. The fix is half-applied. Either (a) plumb the same opus_48000_32 default through ElevenLabsMediaDriver::synthesize_speech, or (b) honor tts_engine.tts_config().elevenlabs.output_format when building MediaTtsRequest in tool_text_to_speech around crates/librefang-runtime/src/tool_runner/media.rs:547. Worth doing here rather than punting to a follow-up — the PR title promises a fix for "WhatsApp PTT format mismatch", and the primary call path still triggers it.

Nits

• detectAudioMime Ogg false-positive surface — your comment at packages/whatsapp-gateway/index.js:3452-3456 correctly flags that OggS magic proves the container but not the codec. Since the gateway now treats the codec assertion as load-bearing (PTT bubble vs file), it's worth ~5 lines to actually verify: an OpusHead literal appears at offset 28 of the first Ogg page for Opus streams, and at the same offset Vorbis streams carry \x01vorbis. Pinning that removes the future-regression risk the comment acknowledges.
• librefang.toml.example [tts.elevenlabs] block lives under a commented [tts] parent — the example file uses # to comment out the whole TTS section. Worth checking the line [tts.elevenlabs] is also # [tts.elevenlabs] (currently # [tts.elevenlabs] per the diff, so this is fine — flagging for your own re-read after rebase regenerates the sha256 baseline).

Verification I ran:

• cargo check --workspace --lib — green
• cargo clippy -p librefang-runtime --all-targets -- -D warnings — 6 errors (above)
• cargo test -p librefang-runtime --lib tts — 11/11 pass
• cargo test -p librefang-types --lib — pass
• node --test packages/whatsapp-gateway/index.test.js — detectAudioMime 8/8 pass (two unrelated pre-existing failures in forwardToLibreFang / dispatch-log suites)

Please rebase onto latest main, fix the clippy errors (preferably via the extracted-resolver route), and address the runtime-media default. Happy to re-review once pushed.

Re-reviewed — the only commits since the prior review are merge-from-main syncs. The format_override type-contract bug at the heart of the prior review is still unfixed:

1. crates/librefang-runtime/src/tts.rs:17-22 (resolve_elevenlabs_format) and the call into ElevenLabs at tts.rs:215-218 — format_override.unwrap_or(cfg.output_format.as_str()) is forwarded verbatim as ?output_format={…}. A caller passing the bare OpenAI/Google-shaped format: "mp3" produces an invalid ElevenLabs query (which expects the compound mp3_44100_128 etc.). Either reject non-compound format values at the boundary (InvalidParameter), or drop the override channel for the ElevenLabs synthesizer and use only the config field. The runtime-media driver default + clippy + 3-tier resolver extraction are all good; only the type-contract mismatch remains.

Note: the PR is otherwise complete and ready to land once the override-vs-compound contract is resolved.