Repository navigation
Conversation
…tamps
The ElevenLabs adapter hard-coded eleven_multilingual_v2 through the Python
SDK and then ran a Whisper pass for word timings. It now calls
/v1/text-to-speech/{voice}/with-timestamps directly, defaults to eleven_v4
(override with --tts-model, tts_model or ELEVENLABS_MODEL_ID), sends the
line's language as language_code, and builds voices[].words from the
character alignment with audio tags left out. ElevenLabs TTS no longer needs
Python or the elevenlabs package.
kumanday
force-pushed
the
fix/media-use-elevenlabs-v4-timestamps
branch
from
October 16, 2026 17:46
4f1d5a2 to
fa123fa
Compare
miguel-heygen
requested changes
Oct 24, 2026
miguel-heygen
left a comment
miguel-heygen
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for this. Calling /with-timestamps directly is a nice simplification: it removes the Python dependency and the Whisper pass for ElevenLabs in one go. The adapter follows the HeyGen one closely and the tests are solid. I ran tts.test.mjs at this head (18 pass). Breaking the tag filter or the language retry makes the new tests fail, so they pin the behaviour.
One thing needs fixing before merge:
- The CLI still requires the Python
elevenlabspackage.packages/cli/src/audio/providers.ts:82still checksELEVENLABS_API_KEY && hasPythonModules(["elevenlabs"]), and its test says it mirrors this skill's provider order. That check driveshyperframes auth status, whichrequirements.mdtells people to run to see which engine they'll get. With this PR, someone with onlyELEVENLABS_API_KEYset and nopip install elevenlabsis voiced by ElevenLabs.auth statusstill tells them Kokoro, plus a pip hint if Kokoro deps are missing. Dropping the Python probe there, updating the two comments that mention it (:7,:41), and adjustingproviders.test.tswould keep the two in step.
Also needed:
- Rebase. The branch now conflicts with main in
skills/media-use/audio/references/requirements.md(the same provider table row changed on main) andskills-manifest.json, so the manifest needs a regen after the rebase. CI hasn't run on this head yet either.
Optional:
tts.mjs:302: if a response comes back withoutalignmentornormalized_alignment,elevenlabsWordsreturns[].audio.mjsthen skips transcription for that line and it ends up with no word timings. Returningnullin that case would let the existing Whisper fallback cover it.- The new tests call
synthesizeElevenLabsdirectly, so nothing checks thatsynthesizeOnepasseslangandmodelthrough. One case throughsynthesizeOnewould pin that.
I didn't make a live ElevenLabs call, so the eleven_v4 model id and its timestamp support rest on your manual run.
Verdict: REQUEST_CHANGES
Reasoning: The new engine and the CLI's auth status now disagree on when ElevenLabs is available (providers.ts:82 still requires the Python package), and the branch conflicts with main.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| const modelId = model || process.env.ELEVENLABS_MODEL_ID || ELEVENLABS_DEFAULT_MODEL; | ||
| const url = `${ELEVENLABS_TTS_URL}/${encodeURIComponent(voiceId)}/with-timestamps?output_format=mp3_44100_128`; | ||
| const post = (body) => | ||
| fetchImpl(url, { |
| fetchImpl(url, { | ||
| method: "POST", | ||
| headers: { "xi-api-key": key, "Content-Type": "application/json" }, | ||
| body: JSON.stringify(body), |
| } | ||
| } else { | ||
| mkdirSync(dirname(wavAbs), { recursive: true }); | ||
| writeFileSync(wavAbs, bytes); |
4 tasks done
Contributor
|
This PR has had no activity for 14 days, so it is marked stale and will close in 7 days. Push a commit or leave a comment to keep it open. A maintainer can add |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The media-use ElevenLabs TTS adapter now calls
POST /v1/text-to-speech/{voice_id}/with-timestampsdirectly:eleven_v4by default, overridable with--tts-model,tts_modelin the request, or$ELEVENLABS_MODEL_ID.pip install elevenlabs.Why
skills/media-use/audio/scripts/lib/tts.mjshard-codedmodel_id="eleven_multilingual_v2"in the Python helper, with no option to change it. Every workflow that voices through the shared audio engine therefore got the older model:faceless-explainer,product-launch-video,pr-to-videoand others.The provider table in
references/tts.mdlisted ElevenLabs as having no word timestamps. Because of that, every ElevenLabs line was transcribed again with Whisper, which costs time and gives estimated timings./with-timestampsreturns exact character timings with the audio.Related work
tts.cleanup.test.mjsnow cover Kokoro only, since ElevenLabs no longer writes a temp text file.lib/tts.mjs. The two changes touch different branches ofsynthesizeOne.How
synthesizeElevenLabs: a new function inlib/tts.mjs, following thesynthesizeHeygenpattern: injectablefetchandtranscodeToWav, never throws, and returns{ ok, words, error }with the reason on failure.language_codefrom the line'slang. If a model rejects it with a 400 that mentions language, the call is retried once without it..mp3output.elevenlabsWords: turns the alignment into[{text,start,end}]and skips bracketed audio tags ([warm],[slightly wry]), so tags never reach captions.elevenlabsAvailable()now checks only for$ELEVENLABS_API_KEY, since no Python import is involved.references/tts.md):eleven_multilingual_v2voice. It now says the pin applies only to docs-site films. Switching that house voice is a separate call for maintainers.references/tts-to-captions.mdandreferences/requirements.mdreflect the new path.skills-manifest.json: regenerated.Test plan
lib/tts.test.mjscoversmodel_idandxi-api-keyheader;language_coderetry;.mp3output into a new directory;node --testoverskills/media-use/**/*.test.mjs,skills/faceless-explainer/**/*.test.mjsandskills/product-launch-video/**/*.test.mjswith thepackages/cli/scripts/test-home.mjsimport: 366 pass.bun packages/cli/scripts/gen-skills-manifest.ts --check,bun scripts/lint-skills.ts,oxfmt --checkandoxlintare clean on the changed files.synthesizeOne({ provider: "elevenlabs", lang: "es", text: "[curious] ¿Y si el grafo aprende solo?" })returned a 2.08 s, 44.1 kHz mono wav and six words with timings. The[curious]tag was excluded.