Skip to content

fix(tts): tool_text_to_speech respects [tts.<provider>] voice + format from config#5284

Closed
f-liva wants to merge 5 commits into
librefang:mainfrom
f-liva:pr/tts-config-voice-format-fallback
Closed

fix(tts): tool_text_to_speech respects [tts.<provider>] voice + format from config#5284
f-liva wants to merge 5 commits into
librefang:mainfrom
f-liva:pr/tts-config-voice-format-fallback

Conversation

@f-liva

@f-liva f-liva commented May 19, 2026

Copy link
Copy Markdown
Contributor

Summary

tool_text_to_speech only fell back to [tts.<provider>] config for google_tts. ElevenLabs and OpenAI silently used each API's default voice/format when the LLM omitted those args, ignoring config.toml settings.

Stacked on #5283

This PR depends on the new TtsElevenLabsConfig.output_format field introduced in #5283. GitHub will automatically rebase against main once #5283 merges.

Behavior change

Input → resolved Before After
text_to_speech(text=…) with ElevenLabs configured voice=None → API default (Rachel), format=None → API default (mp3_44100_128) voice=cfg.voice_id, format=cfg.output_format
text_to_speech(text=…, voice='Sarah') LLM voice forwarded LLM voice forwarded (unchanged)
text_to_speech(text=…) with provider='google_tts' Absolute config override Absolute config override (unchanged)
OpenAI same as ElevenLabs API defaults (alloy, mp3) cfg.voice, cfg.format

Test plan

  • E2E test: text_to_speech(text='prova') produces audio with the voice configured in [tts.elevenlabs].voice_id (verified on production via Beeper voice-note delivery)
  • Unit test for tool_text_to_speech resolution chain (none existing on main; would be a separate PR for test scaffolding)

Files

File Change
crates/librefang-runtime/src/tool_runner/media.rs Add (config_default_voice, config_default_format) tuple + chain into MediaTtsRequest.voice / .format

@github-actions github-actions Bot added size/M 50-249 lines changed area/runtime Agent loop, LLM drivers, WASM sandbox area/sdk JavaScript and Python SDKs and removed size/M 50-249 lines changed labels May 19, 2026

@houko houko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 3-tier resolution (tool_arg > provider_config > built-in default) lands correctly in media.rs, but there's a buried bug in the fallback path that nullifies it for ElevenLabs specifically:

  • synthesize_elevenlabs() (crates/librefang-runtime/src/tts.rs ~175-244) takes only voice_override, not format_override. The call at tts.rs:85 is self.synthesize_elevenlabs(text, voice_override).await — format never gets plumbed in. So the operator's [tts.elevenlabs].output_format is respected via the new media-driver path, but ignored whenever the older engine.synthesize(...) fallback runs at media.rs:581.
    Fix: add format_override: Option<&str> to synthesize_elevenlabs and pass it through at the call site. (#5283 has the exact same gap from the same root cause — fixing them together is cleaner.)

Two more before merge:

  • No unit test for the 3-tier resolution chain. The PR description acknowledges this; please land at least one test pinning the precedence so a future refactor can't silently re-introduce the "config ignored" regression.
  • For google_tts, effective_voice already carries a hardcoded resolution and config_default_voice is dead in the .or() chain (media.rs:551-552). Functionally correct but confusing — a one-line comment explaining the asymmetry would prevent a future contributor from "fixing" it incorrectly.

Intersection note: this PR stacks cleanly with #5283 (no double-mutation risk), as long as the format_override parameter is added in both.

@github-actions github-actions Bot added needs-changes Changes requested by reviewer size/M 50-249 lines changed labels May 20, 2026
@f-liva

f-liva commented May 20, 2026

Copy link
Copy Markdown
Contributor Author

Addressed in 124b7e40:

Item Status Evidence
Blocker 1 — plumb format_override through synthesize_elevenlabs Fixed synthesize_elevenlabs now takes format_override: Option<&str>; resolved as format_override.unwrap_or(self.config.elevenlabs.output_format.as_str()). Call site at tts.rs:85 updated. Grep confirms one call site only — no other plumbing needed. The legacy engine.synthesize(...) fallback path at media.rs:602 now honors [tts.elevenlabs].output_format like the media-driver path.
Blocker 2 — 3-tier resolution unit test Added 3 new tests in tts.rs::tests: test_three_tier_resolution_elevenlabs_format, test_three_tier_resolution_elevenlabs_voice, test_three_tier_resolution_openai_format_and_voice. Each pins the override > config > built-in default precedence. 11/11 tts tests pass.
Nit — google_tts asymmetry comment Added Comment block above config_default_voice in media.rs explains that google_tts is intentionally absent from the .or() chain because its voice is already resolved into effective_voice earlier; "fixing" the asymmetry without first removing the upper branch would duplicate the resolution.

Verification

  • cargo check -p librefang-runtime --lib clean.
  • cargo test -p librefang-runtime tts → 11 passed.
  • cargo clippy -p librefang-runtime --lib -- -D warnings clean.

Re: PR #5283 — couldn't find that number on librefang/librefang (not a PR here). If it lives on a fork or another repo and shares the same root cause, the fix shape from this commit (format_override: Option<&str> parameter + unwrap_or(config.<provider>.<field>.as_str())) carries over directly. Happy to coordinate if you can point me at it.

f-liva pushed a commit to f-liva/librefang that referenced this pull request May 20, 2026
…abs + container-only comment + tests

- `synthesize_elevenlabs` (tts.rs) now accepts `format_override: Option<&str>`
  with the same 3-tier resolution as `synthesize_openai` / `synthesize_google`:
  tool_arg override > [tts.elevenlabs].output_format > built-in default.
  Restores API-contract parity (previously the override was silently dropped
  on the ElevenLabs path, nullifying librefang#5284's tool-call-arg pathway specifically
  for this provider) and updates the sole call site in `synthesize`.

- `detectAudioMime` (whatsapp-gateway/index.js): add inline comment above the
  OggS branch explaining that the magic is container-only and would in theory
  false-positive on Ogg/Vorbis — Vorbis is not a realistic input under
  LibreFang's TTS configuration envelope (default `opus_48000_32`) and would
  fail downstream anyway, so the `codecs=opus` assertion is correct here.

- Add 3 unit tests in `tts::tests`:
  * `test_three_tier_resolution_elevenlabs_format` — pins the precedence chain
    for the ElevenLabs format resolution so a future refactor cannot silently
    re-introduce the "config ignored" regression.
  * `test_three_tier_resolution_elevenlabs_voice` — same shape for the voice id.
  * `test_default_elevenlabs_format_label_is_opus` — pins the contract the
    wa-gateway sniffer depends on: default ElevenLabs config must derive
    `TtsResult.format == "opus"` so the gateway's OggS sniff lines up with
    the actual codec. Also exercises non-default formats (mp3, pcm, ulaw)
    against the prefix-derivation chain.

Verification: `cargo check -p librefang-runtime --lib` ok, `cargo test -p librefang-runtime --lib tts` 11/11 pass, `cargo clippy -p librefang-runtime --lib -- -D warnings` clean, `cargo fmt -p librefang-runtime` no-op, wa-gateway `node --test --test-name-pattern detectAudioMime` 8/8 pass.

@houko houko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core change here is sound. The (config_default_voice, config_default_format) fallback applied via .or() after the LLM args (media.rs:570-577) correctly makes a bare text_to_speech(text=…) respect [tts.elevenlabs] / [tts.openai] config, and the comment explaining why google_tts is deliberately omitted (already resolved into effective_voice and winning the .or() chain first) is accurate — re-adding it would be dead code, agreed.

That said, the same CI blockers that affect #5283 (which this is stacked on) apply here, since the diff carries those commits:

  1. Schema baseline drift (OpenAPI Drift job, fail). The inherited TtsElevenLabsConfig.output_format field changes the config JSON schema, so xtask/baselines/config.sha256 is stale. As a fork PR (IS_INTERNAL_PR: false), the auto-commit path can't run — regenerate locally with cargo xtask schema-check gen and commit the result.

  2. Formatting (Quality job, fail). All reported diffs are in untouched files (librefang-cli/src/main.rs, tui/event.rs, librefang-migrate/src/openclaw.rs, librefang-types/src/config/types.rs:6542), i.e. stale base — main has reformatted them since this branch's base. A rebase on current origin/main should clear it.

Since both of those originate in #5283, resolving them there and rebasing this branch on top is the cleanest path. I've left the same notes on #5283.

  1. Missing unit test for the resolution chain. This is a behavior change to tool_text_to_speech, and the repo prefers a test at the injection site (crates/librefang-runtime/CLAUDE.md: "Always write integration tests at the injection site, not just the implementation site"). The PR description defers it to a separate PR — please add a #[tokio::test] (or a focused unit test) here that asserts: (a) bare text_to_speech(text=…) with ElevenLabs configured resolves to cfg.voice_id + cfg.output_format; (b) an explicit voice/format arg still wins; (c) google_tts still takes the absolute-override path. That locks in exactly the matrix in your behavior-change table and guards against future .or()-chain regressions.

Items 1 and 2 are hard CI gates; item 3 is required by repo test policy for a route/behavior change. Once #5283's baseline + rebase land and a test is added, this should be ready.

@houko houko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — config fallback logic is sound; one real CI failure must be fixed before merge

The resolution logic itself is correct and well-tested. Holding the merge only on the OpenAPI Drift check, which is a genuine, in-scope failure (the other two red checks are stale-base and clear on rebase).

CI failure triage

Check Class Reason
OpenAPI Drift REAL (in-scope) Must regenerate codegen artifacts — see below
Quality (cargo fmt) stale-base ~30 whole-file rustfmt diffs across the workspace (crates/librefang-kernel/src/triggers.rs, crates/librefang-api/src/routes/agents.rs, crates/librefang-memory/src/semantic.rs, …) — none in this PR's diff. Your own tts.rs / media.rs are fmt-clean. Clears on rebase onto origin/main.
Test / Ubuntu shard 1/4 stale-base config_schema_overlay.rs:287 fails on field external_auth_proxy (added on main after your base, unrelated to TTS); network_routes_integration.rs:430 comms_send_rejects_oversize_message. Both are known main-drift signatures. Clears on rebase.

Required fix (OpenAPI Drift) — in-scope, mechanical

TtsElevenLabsConfig derives schemars::JsonSchema (crates/librefang-types/src/config/types.rs:1284), so adding output_format (line 1294-ish in your diff) changes the generated config schema. You also edited librefang.toml.example, which is the input to xtask/baselines/config.sha256 (that baseline currently hashes only librefang.toml.example). Neither openapi.json, the generated SDKs, nor xtask/baselines/* were regenerated, so the drift guard fails:

cargo xtask codegen --openapi
python3 scripts/codegen-sdks.py
cargo xtask schema-check gen

Then commit the regenerated openapi.json, sdk/*, and xtask/baselines/*. This is step 4 of the "Configuration field ritual" in crates/librefang-types/CLAUDE.md — any KernelConfig field change requires regenerating these. Doing this in the same rebase that picks up main will also clear the two stale-base reds.

Logic review — OK

Verified the 3-tier resolution (per-call arg > [tts.<provider>] config > built-in default) holds on both code paths:

  • Driver path (crates/librefang-runtime/src/tool_runner/media.rs:551-555): voice: effective_voice.or_else(|| voice).or(config_default_voice) and format: format.or(config_default_format). For elevenlabs/openai effective_voice is None (only set for google_tts), so the effective order is per-call arg → config → driver default. Correct.
  • The elevenlabs config default feeds MediaTtsRequest.format, which the driver puts into the ?output_format= query param (crates/librefang-runtime-media/src/elevenlabs.rs:138-140) — so opus_48000_32 is the right shape there. The openai default (mp3) maps to response_format (crates/librefang-runtime-media/src/openai.rs:181-187) — also the right shape. The two providers use different format vocabularies and your tuple routes each correctly.
  • TtsEngine fallback path (media.rs:581tts.rs synthesize_*): config defaults are resolved inside synthesize_openai/synthesize_elevenlabs via unwrap_or(&self.config…), so this path also respects config.
  • Serde forward-compat: TtsElevenLabsConfig has struct-level #[serde(default)] (config/types.rs:1285) and the Default impl now lists output_format, so an absent [tts.elevenlabs] section or missing key never panics. Good.
  • The format-label derivation in synthesize_elevenlabs (opus_48000_32"opus" via prefix split) is reasonable and the new TtsResult.format is no longer hardcoded to "mp3".
  • New unit tests pin the precedence chain — appreciated.

Minor

  • The PR description's "Files" table lists only media.rs, but the diff also touches tts.rs, config/types.rs, librefang.toml.example, and packages/whatsapp-gateway/index.js{,.test.js}. Please update the description so reviewers and git log reflect the full scope (the whatsapp-gateway detectAudioMime change is a sensible companion fix, but it should be enumerated, with its own test coverage noted).

No logic blockers. Once the codegen artifacts are regenerated and the branch is rebased onto origin/main, this should go green.

f-liva pushed a commit to f-liva/librefang that referenced this pull request May 22, 2026
…abs + container-only comment + tests

- `synthesize_elevenlabs` (tts.rs) now accepts `format_override: Option<&str>`
  with the same 3-tier resolution as `synthesize_openai` / `synthesize_google`:
  tool_arg override > [tts.elevenlabs].output_format > built-in default.
  Restores API-contract parity (previously the override was silently dropped
  on the ElevenLabs path, nullifying librefang#5284's tool-call-arg pathway specifically
  for this provider) and updates the sole call site in `synthesize`.

- `detectAudioMime` (whatsapp-gateway/index.js): add inline comment above the
  OggS branch explaining that the magic is container-only and would in theory
  false-positive on Ogg/Vorbis — Vorbis is not a realistic input under
  LibreFang's TTS configuration envelope (default `opus_48000_32`) and would
  fail downstream anyway, so the `codecs=opus` assertion is correct here.

- Add 3 unit tests in `tts::tests`:
  * `test_three_tier_resolution_elevenlabs_format` — pins the precedence chain
    for the ElevenLabs format resolution so a future refactor cannot silently
    re-introduce the "config ignored" regression.
  * `test_three_tier_resolution_elevenlabs_voice` — same shape for the voice id.
  * `test_default_elevenlabs_format_label_is_opus` — pins the contract the
    wa-gateway sniffer depends on: default ElevenLabs config must derive
    `TtsResult.format == "opus"` so the gateway's OggS sniff lines up with
    the actual codec. Also exercises non-default formats (mp3, pcm, ulaw)
    against the prefix-derivation chain.

Verification: `cargo check -p librefang-runtime --lib` ok, `cargo test -p librefang-runtime --lib tts` 11/11 pass, `cargo clippy -p librefang-runtime --lib -- -D warnings` clean, `cargo fmt -p librefang-runtime` no-op, wa-gateway `node --test --test-name-pattern detectAudioMime` 8/8 pass.
Federico Liva added 4 commits May 22, 2026 20:22
…/Opus

ElevenLabs TTS returned MP3 (default), but the wa-gateway sendAudio path
hardcoded `mimetype: audio/ogg; codecs=opus`. Mobile WhatsApp tolerated the
mismatch via MP3 magic-byte fallback, but the Beeper / mautrix-whatsapp
bridge rejected the message as "Unsupported content type in Web mode" —
no voice bubble, no audio file, silent drop.

Two changes:

1. ElevenLabs synth now requests `?output_format={cfg}` so the API returns
   the requested container directly. New `TtsElevenLabsConfig.output_format`
   field (default `opus_48000_32`), configurable via [tts.elevenlabs] in
   librefang.toml. The `format` label on TtsResult is derived from the
   prefix of the configured value (`opus_48000_32` → "opus").

2. wa-gateway sendAudio sniffs the buffer's actual mime via magic bytes
   (OggS, ID3/MPEG-sync, RIFF, fLaC, ftyp) and sends with the real mime.
   If PTT was requested but the buffer isn't Ogg/Opus, downgrade to a
   regular audio attachment with a warning. Defensive against future
   producer drift.

Unit tests: detectAudioMime describe block (8 cases) + test_config_defaults
asserts `output_format = opus_48000_32` default.
…t from config

The media-driver branch of `tool_text_to_speech` only fell back to the
provider config for `google_tts`. For ElevenLabs and OpenAI, when the
LLM called `text_to_speech(text=...)` without explicit `voice` / `format`
arguments:

* `voice = None`  → driver used each API's default voice (Rachel for
  ElevenLabs, Alloy for OpenAI) instead of the user-configured one.
* `format = None` → driver used each API's default container
  (mp3_44100_128 for ElevenLabs) ignoring `[tts.elevenlabs].output_format`
  (required for WhatsApp PTT compatibility — see prior commit).

Net effect: agents had to remember voice + format on every call,
contradicting the implicit contract that `[tts.<provider>]` config
sections drive defaults.

Fix: new `(config_default_voice, config_default_format)` tuple keyed on
`resolved_provider`, appended to the resolution chain so input args take
precedence but config values fill the gap when omitted. Google keeps its
existing absolute override (voice names not portable). ElevenLabs / OpenAI
now use input voice/format if provided, falling back to `[tts.<provider>]`
from config.toml.

The old TtsEngine fallback path (tts.rs synthesize_elevenlabs / openai)
was already correct via `voice_override.unwrap_or(&config.<x>)` — only
the new media-driver path needed this fix.

Stacked on prior commit: depends on `TtsElevenLabsConfig.output_format`
field introduced together with the WhatsApp PTT audio fix.
…abs + 3-tier test

- synthesize_elevenlabs now accepts format_override and applies the same
  3-tier resolution as synthesize_openai/synthesize_google: tool_arg
  override > [tts.elevenlabs].output_format > built-in default. Previously
  the older TtsEngine fallback path (media.rs ~602) ignored the configured
  output_format, so a bare text_to_speech() call hitting the fallback got
  the ElevenLabs API default (mp3) instead of opus_48000_32, breaking
  WhatsApp PTT compatibility for the fallback codepath.
- Add 3 unit tests pinning the 3-tier precedence chain for elevenlabs
  voice + format and openai voice + format, so future refactors can't
  silently re-introduce the "config ignored" regression.
- Add comment on media.rs config_default_voice block explaining why
  google_tts is intentionally absent from the .or() chain (it's resolved
  earlier via effective_voice) — prevents a contributor from "fixing"
  the asymmetry incorrectly.
…block

Per houko librefang#5284 review (OpenAPI Drift check). Ran:
- cargo xtask codegen --openapi    -> no openapi.json changes
- python3 scripts/codegen-sdks.py  -> no sdk/* changes
- cargo xtask schema-check gen     -> config.sha256 updated

The schema-check baselines for openapi and agent already matched
this branch's generated artefacts; only `config.sha256` needed to
catch up to the `librefang.toml.example` text added by this PR.
@f-liva
f-liva force-pushed the pr/tts-config-voice-format-fallback branch from 2ea1f5b to e58ad98 Compare May 22, 2026 18:27
@f-liva

f-liva commented May 22, 2026

Copy link
Copy Markdown
Contributor Author

Regenerated codegen artefacts via the documented ritual:

cargo xtask codegen --openapi      # no openapi.json changes
python3 scripts/codegen-sdks.py    # no sdk/* changes
cargo xtask schema-check gen       # updated config.sha256

Turned out only the config.sha256 baseline was stale — openapi.json and sdk/* already reflected the new output_format field (TtsElevenLabsConfig's JsonSchema derive emits it on this branch). Pushed the single baseline change.

Also rebased onto current origin/main to clear the stale-base fmt drift you flagged on the Quality lane.

@houko houko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The unique media.rs change is correct — the per-provider (config_default_voice, config_default_format) fallback applied via .or() after the LLM args (tool_runner/media.rs:546-577) makes a bare text_to_speech(text=…) respect [tts.<provider>], and the google-asymmetry comment is accurate. Two blockers.

1. Duplicates #5283. git log origin/main.. shows this branch re-contains #5283's entire change as separate commits (0b0bb622 audio fix + a plumb format_override commit), so this diff is a superset of #5283 rather than stacked on it. Please either base this branch on #5283 (narrowing this diff to the media.rs delta) or have one PR carry the whole cluster and close the other.

2. Injection-site test still missing. The test_three_tier_resolution_* tests in tts.rs::tests re-implement the unwrap_or chain inline — they assert on a local copy of the logic, not on tool_text_to_speech itself (git grep tool_text_to_speech finds zero test call sites). A regression in the media.rs .or() chain wouldn't be caught. Per librefang-runtime/CLAUDE.md ("test at the injection site"), please add a test that actually calls tool_text_to_speech with ElevenLabs configured and asserts the resolved MediaTtsRequest.voice/.format (bare call → config default; explicit arg → arg wins; google → absolute-override path).

Nit: update the Files table — the diff also touches tts.rs, config/types.rs, librefang.toml.example, the whatsapp-gateway files, and config.sha256, not just media.rs.

@github-actions github-actions Bot added the has-conflicts PR has merge conflicts that need resolution label May 23, 2026
@houko

houko commented May 23, 2026

Copy link
Copy Markdown
Contributor

The standing request-changes gate from my 2026-05-22 review is still in place — both blockers there are unaddressed (the only head change since is the Merge branch 'main' commit 8ab431b8, which pulled in #4997 and left tts.rs/media.rs byte-identical to what I reviewed). Recapping so it's not lost, plus one new item:

  • Injection-site test still missing. test_three_tier_resolution_* in crates/librefang-runtime/src/tts.rs::tests re-implement the unwrap_or chain inline (e.g. format_override.unwrap_or(config.elevenlabs.output_format.as_str())) — they assert on a local copy of the logic, not on tool_text_to_speech. git grep tool_text_to_speech finds zero test call sites, so a regression in the real media.rs .or() chain or the synthesize_* plumbing wouldn't be caught. Per crates/librefang-runtime/CLAUDE.md ("test at the injection site"), please add a test that calls tool_text_to_speech with ElevenLabs configured and asserts the resolved MediaTtsRequest.voice / .format (bare call → config default; explicit arg → arg wins; google → absolute-override path).

  • Duplicates fix(audio): WhatsApp PTT format mismatch — ElevenLabs MP3 sent as Ogg/Opus #5283. This branch still re-contains fix(audio): WhatsApp PTT format mismatch — ElevenLabs MP3 sent as Ogg/Opus #5283's commits (0b0bb622 audio fix + the format_override plumb) as a superset rather than stacking on it. Please base on fix(audio): WhatsApp PTT format mismatch — ElevenLabs MP3 sent as Ogg/Opus #5283 to narrow this to the media.rs delta, or have one PR carry the whole cluster and close the other.

New: the branch is conflicting again (mergeStateStatus: DIRTY). You merged main at 8ab431b8, but main has since advanced to ef0f6d37, and #5230's compileGroupTriggerRegex tests now collide with your detectAudioMime block in packages/whatsapp-gateway/index.test.js (one conflict hunk, lines ~1547–1684 — both sides are additive describe(...) blocks, so keep both). A fresh rebase/merge onto current origin/main clears it.

CI note for triage: the red Quality check is not from this PR — it's clippy (Rust 1.95.0) firing doc_lazy_continuation at crates/librefang-api/src/validation.rs:469-471 and manual_pattern_char_comparison at crates/librefang-api/src/webchat.rs:277, both of which exist verbatim on current origin/main and are outside this diff. OpenAPI Drift and the test shards are now green. So the only PR-side blockers remain the injection-site test and the #5283 relationship; the rebase will also pick up whatever fix lands for the inherited clippy lints.

@houko houko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase required — the branch has a merge conflict in packages/whatsapp-gateway/index.test.js against the feat(wa-gateway): compute wasMentioned commit (#5230) that landed on main after the last merge. Please rebase onto current main and resolve.

The Quality CI failure (doc_lazy_continuation / manual_pattern_char_comparison in validation.rs and webchat.rs) is a pre-existing clippy issue unrelated to this PR — it disappears after a clean rebase against the current main where it was fixed (#5654).

Code review — correctness assessment:

The core fix is sound. Two code paths exist in tool_text_to_speech:

  1. Media-driver path (media_drivers cache hit): The new (config_default_voice, config_default_format) tuple correctly populates MediaTtsRequest.voice / .format via .or(config_default_voice) / .or(config_default_format) when the LLM omits those args. The ElevenLabs media driver (crates/librefang-runtime-media/src/elevenlabs.rs:115) and OpenAI media driver (same file, :180–181) both have hardcoded fallbacks ("mp3_44100_128" / "alloy" + "mp3") that are correctly bypassed once request.format / request.voice is Some(...) from the config.

  2. TtsEngine fallback path (media.rs:607): engine.synthesize(text, voice, format) passes the raw tool args; config defaults are applied internally in synthesize_elevenlabs (:192, format_override.unwrap_or(self.config.elevenlabs.output_format.as_str())) and synthesize_openai (:116). Both paths are correct and consistent.

The output_formatTtsResult.format conversion in synthesize_elevenlabs (split('_').next().unwrap_or("opus")) correctly extracts "opus" from "opus_48000_32" and "mp3" from "mp3_44100_128" — matching the approach already used in the media driver's parse_output_format.

One issue to address before merge:

The config_default_voice/config_default_format block only activates when resolved_provider is Some("elevenlabs") or Some("openai"). When resolved_provider is None (no provider arg from LLM, no tts.provider in config — auto-detection via detect_for_capability), the block returns (None, None) and the media driver's own hardcoded fallbacks apply, ignoring [tts.elevenlabs] config. This is a narrower version of the original bug: a bare text_to_speech(text=...) call where the provider is auto-detected (not explicitly configured) still ignores config. Given that the PR's description scopes the fix to the configured-provider case and the PR summary table is accurate, this is a documented limitation rather than a regression — but it should be called out explicitly in either a comment or a follow-up issue so it isn't rediscovered. Please add a // TODO: auto-detected provider path does not apply config defaults comment at media.rs:568 (the _ => (None, None) arm), or open a follow-up issue and reference it.

Tests: The three new unit tests in tts.rs correctly pin the unwrap_or resolution chain for both providers. They test the logic directly rather than through a network call, which is appropriate. The detectAudioMime JS tests in index.test.js are thorough and cover the magic-byte edge cases.

#[serde(default)] on TtsElevenLabsConfig is present at the struct level (:1284), so existing configs without output_format will default to "opus_48000_32". This is a behavior change for existing ElevenLabs users who previously got "mp3_44100_128" from the driver's hardcoded fallback — the librefang.toml.example update documents it, but a CHANGELOG entry under [Unreleased] for this user-visible default change would be appropriate.

After rebase + the _ => (None, None) comment + CHANGELOG entry: approve.

@houko houko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed against main. The core fix (chaining [tts.<provider>] config defaults into the MediaDriverCache path of tool_text_to_speech) is correct and well-targeted — the elevenlabs/openai media drivers genuinely do unwrap_or("alloy") / unwrap_or("mp3_44100_128") (crates/librefang-runtime-media/src/openai.rs:180-181, elevenlabs.rs:112-115), so the (config_default_voice, config_default_format) injection in crates/librefang-runtime/src/tool_runner/media.rs:558-578 is the right place to land it. However:

Blocking

  1. Clippy fails with 11 errors on --all-targets — CI gate cargo clippy --workspace --all-targets -- -D warnings will reject. The new tests in crates/librefang-runtime/src/tts.rs:398-505 all hit clippy::unnecessary_literal_unwrap because they call unwrap_or on literal Some("foo") / None. Rewrite the assertions to either bind through a real Option<&str> produced by code, or just assert_eq! against the constants directly (the test body already mirrors the resolution logic verbatim, so the indirection adds nothing).

  2. Scope: this PR duplicates ~100% of #5283. gh pr diff 5283 and gh pr diff 5284 are byte-identical in crates/librefang-types/src/config/types.rs, the synthesize_elevenlabs signature change, the URL format-string change, the librefang.toml.example block, the xtask/baselines/config.sha256 bump, the elevenlabs 3-tier tests, AND the whatsapp-gateway detectAudioMime + index.test.js cases. The PR body says "stacked on #5283" but the branch is opened off main, not off pr/audio-mime-fix, so the diff carries both payloads. Per CLAUDE.md "One PR ↔ one issue" — please either (a) rebase this branch onto pr/audio-mime-fix so only the media.rs delta + the new openai test remain, or (b) wait for #5283 to merge and rebase onto updated main. Right now reviewing the two PRs separately is impossible.

Should fix

  1. No integration / unit test for the media.rs resolution chain itself (tool_runner/media.rs:558-578). The new tests in tts.rs exercise the TtsEngine fallback path that already had internal fallbacks — they do NOT cover the new (config_default_voice, config_default_format) branch that's the actual subject of this PR. A targeted test under crates/librefang-runtime/src/tool_runner/tests/ (or against start_test_server* in librefang-api) that drives tool_text_to_speech with no voice / format args under a custom [tts.elevenlabs] and asserts the resolved MediaTtsRequest carries the config values would close this gap. CLAUDE.md mandates a #[tokio::test] for any route/wiring change — this is wiring.

Nits

  1. crates/librefang-runtime/src/tts.rs:404 and again at :506 reference "houko" / a reviewer handle in test doc-comments. Strip personal names from regression-test rationale; describe the bug instead.

  2. The comment block at crates/librefang-runtime/src/tool_runner/media.rs:553-557 is correct but quite long for a fallback note. Two lines about "google_tts handled in the branch above; do not duplicate" would suffice.

Please rebase onto main once #5283 lands (CONFLICTING currently — merge state DIRTY), fix the clippy errors, and add the media.rs resolution test. Happy to re-review.

@f-liva

f-liva commented May 25, 2026

Copy link
Copy Markdown
Contributor Author

This PR stacks on #5283. Since #5283 has been updated (rebased + all review findings addressed), this PR needs to be rebased onto #5283 once it merges to eliminate the duplication.

Regarding the 11 clippy unnecessary_literal_unwrap errors: #5283 already extracted the resolver into free functions (resolve_elevenlabs_format / elevenlabs_format_label) and uses std::hint::black_box for remaining test values. Once #5283 merges, rebase this PR onto main and the clippy errors will be resolved.

Merge order: #5283 first, then rebase #5284 onto the result and address any remaining review items (media.rs resolution chain integration test, long comment compression).

Will push the rebased version once #5283 lands.

@github-actions github-actions Bot added has-conflicts PR has merge conflicts that need resolution and removed has-conflicts PR has merge conflicts that need resolution labels May 28, 2026
@houko

houko commented May 30, 2026

Copy link
Copy Markdown
Contributor

Closing this as superseded by #5283.

The two PRs overlap heavily — both add the identical output_format field/default to TtsElevenLabsConfig, both change synthesize_elevenlabs(... format_override), both add the same detectAudioMime to the wa-gateway, and they edit the same tool_runner/media.rs region. They aren't independently mergeable.

#5283 is the strict superset: it additionally fixes the MediaDriver ElevenLabs default in librefang-runtime-media/src/elevenlabs.rs and extracts testable resolve_elevenlabs_format/elevenlabs_format_label free functions.

The logic in this PR is correct — thank you for it. If the [tts.openai] voice/format fallback path here isn't covered by #5283, please fold that specific delta into #5283 so nothing is lost.

@houko houko closed this May 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/runtime Agent loop, LLM drivers, WASM sandbox area/sdk JavaScript and Python SDKs has-conflicts PR has merge conflicts that need resolution needs-changes Changes requested by reviewer size/M 50-249 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants