Repository navigation
feat(media-use): add VoiceStudio as an explicit-choice TTS provider - #5658
Closed
leokung4325 wants to merge 1 commit into
Closed
leokung4325 wants to merge 1 commit into
leokung4325 wants to merge 1 commit into
Conversation
VoiceStudio (github.com/debpalash/VoiceStudio) is a local, fully open-source TTS/voice-cloning server — adds a free/local option alongside HeyGen, ElevenLabs, and Kokoro for users who want real voice cloning without a cloud dependency. Verified live against a running VoiceStudio 0.5.7 backend (native macOS process on Apple Silicon, MPS acceleration): generated real audio through the exact /v1/audio/speech payload shape this function sends, confirmed playable. 20/20 unit tests pass (14 existing + 6 new), oxlint/oxfmt clean. Deliberately explicit-choice-only (--provider voicestudio), same as Gemini, never added to the silent auto-detect chain: - VoiceStudio has no credential to probe synchronously (it's a local server, not a cloud API) — the existing chain (heygenAvailable/elevenlabsAvailable) is sync-only by design, and "is the backend running" can only be answered with a network call. - More importantly: VoiceStudio's default engine (omnivoice, backed by k2-fsa/OmniVoice) ships CC-BY-NC with restricted commercial inference. Silently auto-picking VoiceStudio would silently inherit whichever engine happens to be active on the user's backend, with real licence consequences. This PR pins VOICESTUDIO_ENGINE to VoxCPM2 (Apache-2.0, still supports voice cloning) by default, but that choice should always be explicit. Scope: this is the simple synthesize-one-line path, matching what the other providers already do here. Voice cloning (VoiceStudio's /generate endpoint with a ref_audio sample) is a separate per-project provisioning step, closer to heygen-voice.mjs's clone/list/delete pattern than to this per-line engine — a natural follow-up, kept out of this PR to stay focused. Co-Authored-By: Claude Sonnet 5 <[email protected]>
6 tasks done
miguel-heygen
requested changes
Oct 24, 2026
miguel-heygen
left a comment
miguel-heygen
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for this, and for keeping VoiceStudio out of the automatic fallback chain. The reasoning in the description (no credential to check synchronously, and engine licences that matter for commercial use) is sound, and the provider follows the existing Gemini pattern closely.
One thing to fix before this can go in:
tts.test.mjs:237("pickProvider never auto-picks voicestudio or gemini") depends on the machine it runs on. It clears five env vars, butheygenCredential()also readsHEYGEN_ACCESS_TOKEN,HEYGEN_API_BASE,HEYGEN_CONFIG_DIRand~/.heygen/credentials(heygen.mjs:109-126). On a machine that is signed in to HeyGen, the test fails withexpected 'kokoro', got 'heygen'; I reproduced that at this head, and it also fails with justHEYGEN_ACCESS_TOKENset. PointingHEYGEN_CONFIG_DIRat an empty temp dir and clearing those two variables (like the "unreadable credentials path" test a few lines up) fixes it.
Smaller suggestions:
references/tts.md(the provider table and the Gemini note) and the request comment ataudio.mjs:18still list only the existing four providers. Agents choose providers from those docs, so without an entry there VoiceStudio is hard to find, and the "pin a commercially usable engine" guidance only lives in a code comment.- The
synthesizeVoicestudiotests all passlang: null, butaudio.mjsalways passes a language ("en"by default), so the request body that is actually sent, withlanguage, is never asserted. One case withlang: "en"would cover it. - The note about verifying against 0.5.7 natively on macOS and the Docker image (around
tts.mjs:393) reads as history; CONTRIBUTING's "Comments" section asks for that kind of detail in the PR description instead.
CI has not run on this head yet, so I'm going on reading the code and running the test file locally.
Verdict: REQUEST_CHANGES
Reasoning: A new test fails on any machine signed in to HeyGen (reproduced at this head); the provider code itself looks fine once that and the docs gap are addressed.
Collaborator
|
Thanks for putting this together. We're keeping the TTS providers in the skills to ones we maintain or officially support, so we're not going to recommend VoiceStudio here. Closing, but it would make a good community skill in heygen-com/hyperframes-community-skills. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds VoiceStudio (a local, fully
open-source ElevenLabs-alternative TTS/voice-cloning server) as a provider in
skills/media-use/audio/scripts/lib/tts.mjs, alongside HeyGen, ElevenLabs,Kokoro, and Gemini — gives users real voice cloning with no cloud dependency
and no account.
Why explicit-choice-only, not auto-detected
Deliberately not added to the silent auto-detect fallback chain, for two
reasons:
heygenAvailable(),elevenlabsAvailable())is synchronous by design. VoiceStudio is a local server with no credential
to check — "is it running" can only be answered with a network call.
(
omnivoice, backed byk2-fsa/OmniVoice) ships CC-BY-NC withcommercial_inference: restrictedper VoiceStudio's own model registry.Silently auto-picking VoiceStudio would silently inherit whichever engine
the user's backend happens to have active, with real licence consequences
for anyone generating narration for monetized content. This PR pins
VOICESTUDIO_ENGINEto VoxCPM2 (Apache-2.0, still supports real voicecloning — unlike the other commercially-safe engines, Kokoro/KittenTTS,
which are fixed-preset only) by default, but that choice should always be
deliberate, never silent.
geminialready follows this sameexplicit-only pattern for its own reasons; this PR follows the precedent.
What's in scope
The simple synthesize-one-line path (
/v1/audio/speech, OpenAI-compatible),matching what every other provider does in this file. No word timestamps in
the response (same as ElevenLabs/Kokoro) — caller chains
transcribeWav().Out of scope, left as a follow-up: voice cloning itself (VoiceStudio's
native
/generateendpoint with aref_audioreference sample) is aper-project provisioning step, closer to
heygen-voice.mjs'sclone/list/delete pattern than to this per-line synthesis engine. Keeping
this PR focused on the provider-chain addition.
Verification
(Apple Silicon, MPS acceleration — the published Docker image is
linux/amd64-only with no GPU access in that container, so native is theright path there), installed VoxCPM2, and generated real audio through the
exact
/v1/audio/speechpayload shape this function sends. Confirmedplayable (
ffprobe-verified wav, correct duration).node --test skills/media-use/audio/scripts/lib/tts.test.mjs. New testscover:
pickProvideraccepting the explicit choice,pickProviderneverauto-selecting it into the fallback chain, a successful synth writing wav
bytes directly (no transcode step, unlike HeyGen), a model-licence refusal
being surfaced verbatim (not swallowed — the actual error VoiceStudio
returns when a model's licence isn't accepted), a plain HTTP failure, and a
connection failure naming the backend URL + the
bun run dev:apihint.oxlintandoxfmt --checkboth clean on both changedfiles.
Test plan
node --test skills/media-use/audio/scripts/lib/tts.test.mjs— 20/20 passbunx oxlint skills/media-use/audio/scripts/lib/tts.mjs skills/media-use/audio/scripts/lib/tts.test.mjs— cleanbunx oxfmt --checkon both files — clean🤖 Generated with Claude Code