Skip to content

Commit 01601d1

Browse files
authored
feat: add Gemini TTS models and service-account auth to media skills (#4377)
* feat: add opt-in Gemini 3.8 TTS to media skills * feat: support Gemini service accounts and older TTS models * fix(audio): isolate Gemini auth and narrow OAuth scope * fix(audio): pin Gemini Python helpers to UTF-8
1 parent 86fa10c commit 01601d1

19 files changed

Lines changed: 948 additions & 21 deletions

File tree

‎docs/guides/authentication.mdx‎

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -74,9 +74,37 @@ Run `npx hyperframes doctor` to check which local dependencies are installed.
7474
The media workflows run `hyperframes auth status` before generation and tell
7575
you which path they will use.
7676

77+
For Gemini narration, explicitly ask your agent to use Gemini TTS. The shared
78+
audio workflow supports 3.8 Flash and Flash-Lite, 3.1 Flash Preview, and 2.5 Pro
79+
and Flash Preview. Gemini remains an explicit choice; configuring credentials
80+
does not change the automatic voice-provider order above.
81+
82+
Choose one authentication method:
83+
84+
- **API key:** set `GEMINI_API_KEY` or `GOOGLE_API_KEY` in your environment or
85+
project `.env`. This path needs no Python dependencies.
86+
- **Service account:** install `google-auth` and `requests` in your Python 3
87+
environment, then set `GOOGLE_APPLICATION_CREDENTIALS` to the absolute path
88+
of your service-account JSON file. Alternatively, inject the JSON through
89+
`GCS_CREDS` from your secret manager. Set `GOOGLE_CLOUD_PROJECT` to the quota
90+
project when it differs from the service account's project.
91+
92+
Credentials resolve in that order: Gemini key, Google key, service-account file,
93+
then JSON environment variable. Unset API keys when choosing service-account
94+
OAuth. The helper obtains a fresh token for each generation using the
95+
`generative-language.retriever` scope. Keys and tokens
96+
are never placed in compositions or saved audio metadata.
97+
98+
Both methods call the Gemini Developer Interactions API. Your project needs
99+
access to that API and the requested model. Cloud Text-to-Speech and Vertex AI
100+
have separate model availability. This helper accepts service-account JSON,
101+
not user ADC files or metadata-server credentials.
102+
103+
See [Gemini voiceover](/prompting/media-and-audio#gemini-voiceover).
104+
77105
<Note>
78106
`npx hyperframes tts` itself is the local Kokoro CLI. Hosted HeyGen and
79-
ElevenLabs voices are selected by the bundled media workflow helpers, not by
107+
ElevenLabs and Gemini voices are selected by the bundled media workflow helpers, not by
80108
that command.
81109
</Note>
82110

@@ -112,7 +140,9 @@ to a shared space with `--space`.
112140
| `HEYGEN_API_URL` | API base URL (default `https://api.heygen.com`). |
113141
| `HEYGEN_CONFIG_DIR` | Credentials directory (default `~/.heygen`). |
114142
| `ELEVENLABS_API_KEY` | ElevenLabs TTS, used when no HeyGen credential is present. |
115-
| `GEMINI_API_KEY` / `GOOGLE_API_KEY` | Lyria music generation (and capture descriptions). |
143+
| `GEMINI_API_KEY` / `GOOGLE_API_KEY` | Explicit Gemini TTS selection and Lyria music generation; capture descriptions use `GEMINI_API_KEY`. |
144+
| `GOOGLE_APPLICATION_CREDENTIALS` / `GCS_CREDS` | Gemini TTS service-account JSON file path / injected JSON. Used when neither API key is set. |
145+
| `GOOGLE_CLOUD_PROJECT` / `GCLOUD_PROJECT_ID` | Gemini OAuth quota project; first set value wins, then the service account project. |
116146
| `OPENROUTER_API_KEY` | Capture descriptions; takes priority over Gemini for that step. |
117147

118148
## Related topics

‎docs/prompting/media-and-audio.mdx‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,26 @@ agent choose.
3838
- ❌ `add a voice`
3939
- ✅ `warm, unhurried female narration of the quoted script` — tone and pace are what actually change the delivery
4040

41+
### Gemini voiceover
42+
43+
Choose Gemini when you want to direct a narrator's delivery with a description.
44+
Configure [Gemini credentials](/guides/authentication#providers-used-by-agent-workflows),
45+
then ask your agent:
46+
47+
> Use Gemini 3.8 Flash TTS with the Kore voice to narrate this script. Make the
48+
> delivery warm and unhurried. Use the media-use audio engine, transcribe the
49+
> generated audio, and sync the captions and visual reveals to the spoken words.
50+
51+
The shared audio workflow supports 3.8 Flash and Flash-Lite TTS,
52+
3.1 Flash TTS Preview, and 2.5 Pro and Flash TTS Preview. Gemini is an explicit choice; it does not replace
53+
the automatic provider order above. Keep the script separate from delivery
54+
instructions. The helper uses structured style metadata for 3.8 and a delivery
55+
prompt for older models; review the generated speech for accidental directions.
56+
57+
The agent saves the voiceover locally and obtains word timings through a
58+
separate transcription pass. Check names, numbers, and pauses against the
59+
audio before the final render. A new narration take needs new timings.
60+
4161
## Background music
4262

4363
Music resolves from a large catalog by mood. It should almost always sit *under*

‎skills-manifest.json‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
"files": 142
77
},
88
"faceless-explainer": {
9-
"hash": "d3e32414b9e86678",
9+
"hash": "995968cc3ba6faee",
1010
"files": 27
1111
},
1212
"figma": {
@@ -54,8 +54,8 @@
5454
"files": 1
5555
},
5656
"media-use": {
57-
"hash": "d15857ed6f884a1f",
58-
"files": 91
57+
"hash": "b35de041b87e763d",
58+
"files": 98
5959
},
6060
"motion-graphics": {
6161
"hash": "ab9d2e9c205ce1b1",
@@ -66,11 +66,11 @@
6666
"files": 169
6767
},
6868
"pr-to-video": {
69-
"hash": "148de3a4692bc44d",
69+
"hash": "b1473fb853c2ca2c",
7070
"files": 33
7171
},
7272
"product-launch-video": {
73-
"hash": "104014221c41ae13",
73+
"hash": "6677e7785909bf98",
7474
"files": 34
7575
},
7676
"remotion-to-hyperframes": {

‎skills/faceless-explainer/scripts/audio.mjs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,8 @@ function runGenerate(argv) {
173173
: { mode: "retrieve", query, blob: g.message || "", arc: g.arc || "" },
174174
};
175175
if (userVoice) request.voice = userVoice;
176+
if (flag(argv, "tts-model", null)) request.tts_model = flag(argv, "tts-model", null);
177+
if (flag(argv, "style", null)) request.style = flag(argv, "style", null);
176178

177179
const neutral = neutralPath(outPath);
178180
runEngine({ request, hyperframesDir, neutral, only: "tts,bgm" }, die);

‎skills/media-use/audio/references/tts-to-captions.md‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,13 +14,22 @@ node skills/media-use/audio/scripts/heygen-tts.mjs \
1414

1515
`narration.words.json` is already in the `[{ id, text, start, end }]` shape the captions pipeline consumes — no separate transcribe pass.
1616

17-
## Path B — ElevenLabs / Kokoro (TTS → Whisper)
17+
## Path B — Gemini / ElevenLabs / Kokoro (TTS → transcription)
1818

19-
These providers don't return word data. Generate the audio, then transcribe:
19+
These adapters supply audio without word data. The shared audio engine runs
20+
transcription automatically when timings are absent. For Gemini, use the
21+
request in [Text to speech](tts.md#gemini-narration), then consume
22+
`audio_meta.json` → `voices[].words`.
23+
24+
For a standalone local Kokoro generation, generate the audio, then transcribe:
2025

2126
```bash
2227
npx hyperframes tts script.txt --voice af_heart --output narration.wav
2328
npx hyperframes transcribe narration.wav --model small.en # voice af_heart is American English
2429
```
2530

2631
Whisper extracts precise word boundaries from the generated audio, so caption timing matches delivery without hand-tuning. Match `--model` to the voice's language (use `small.en` for `a`/`b` prefixes, `small --language <code>` otherwise). Then consume `transcript.json` via the caption references in `captions/`.
32+
33+
For Gemini, verify that transcription preserved the script, especially names,
34+
numbers, and delivery pauses. If `words` is empty, resolve the transcription
35+
failure before captioning. Generate and align again after changing the read.

‎skills/media-use/audio/references/tts.md‎

Lines changed: 84 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,10 @@ Use another voice only for a documented reason, and write the reason down.
3636

3737
## Available routes
3838

39+
Gemini is an explicit alternative to the automatic provider order below. A
40+
request to use Gemini already chooses the provider; do not redirect that user
41+
to HeyGen sign-in. Read the Gemini section for its credential requirement.
42+
3943
| Order | Provider | Env trigger | Voice IDs | Word timestamps | Audio format |
4044
| ----- | ----------------- | ------------------------------------------- | ------------------------------------------- | ----------------------------------------- | -------------------- |
4145
| 1 | HeyGen (Starfish) | `$HEYGEN_API_KEY` / `~/.heygen/credentials` | UUIDs from `GET /v3/voices?engine=starfish` | **Yes** (`word_timestamps[]` in response) | mp3 → wav via ffmpeg |
@@ -83,12 +87,86 @@ node skills/media-use/audio/scripts/heygen-tts.mjs --list # public starfish vo
8387

8488
## When to use which provider
8589

86-
| Goal | Use |
87-
| --------------------------------------------------------- | --------------------------------------------------- |
88-
| Best voice quality + word timestamps in one call | **HeyGen** |
89-
| Drop-in cloud TTS, big voice catalog | **ElevenLabs** |
90-
| Offline, no API key, fast iteration | **Kokoro** |
91-
| Non-English multilingual with deterministic phonemization | **Kokoro** (`ef_dora`, `jf_alpha`, `zf_xiaobei`, …) |
90+
| Goal | Use |
91+
| --------------------------------------------------------- | --------------------------------------------------------- |
92+
| Best voice quality + word timestamps in one call | **HeyGen** |
93+
| Drop-in cloud TTS, big voice catalog | **ElevenLabs** |
94+
| Offline, no API key, fast iteration | **Kokoro** |
95+
| Directed delivery with Gemini prebuilt or custom voices | **Gemini** (explicit selection; transcription for timing) |
96+
| Non-English multilingual with deterministic phonemization | **Kokoro** (`ef_dora`, `jf_alpha`, `zf_xiaobei`, …) |
97+
98+
## Gemini narration
99+
100+
Use the shared audio engine, not `hyperframes tts`. Authenticate with either:
101+
102+
- `GEMINI_API_KEY` or `GOOGLE_API_KEY` (first set key wins).
103+
- A service-account JSON file at `GOOGLE_APPLICATION_CREDENTIALS`, or injected
104+
JSON in `GCS_CREDS`. Install `google-auth requests` in the Python 3 environment
105+
used by the helper. A configured file takes precedence over injected JSON.
106+
107+
API keys take precedence over service accounts: unset both key variables to use
108+
OAuth. Never put credentials in a request file or composition. The helper
109+
obtains a fresh OAuth token for each generation with the
110+
`generative-language.retriever` scope. The quota project resolves from
111+
`GOOGLE_CLOUD_PROJECT`, then `GCLOUD_PROJECT_ID`, then the service-account JSON.
112+
User ADC files and metadata-server authentication are not supported.
113+
114+
Both routes call the Gemini Developer Interactions API, not Cloud TTS or Vertex
115+
AI. Those APIs have separate model catalogs and access requirements.
116+
117+
Save this as `audio_request.json` in the project:
118+
119+
```json
120+
{
121+
"provider": "gemini",
122+
"tts_model": "gemini-3.8-flash-tts",
123+
"voice": "Kore",
124+
"lang": "en",
125+
"style": "Warm, clear, conversational. Leave a short pause between sentences.",
126+
"lines": [
127+
{ "id": "intro", "text": "Every word has a moment. Let the picture follow the voice." }
128+
],
129+
"bgm": { "mode": "none" }
130+
}
131+
```
132+
133+
```bash
134+
node <SKILL_DIR>/audio/scripts/audio.mjs \
135+
--request ./audio_request.json --hyperframes . --out ./audio_meta.json --only tts
136+
```
137+
138+
The engine saves `assets/voice/intro.wav`, measures its duration, and transcribes
139+
it into `voices[].words` in `audio_meta.json`. Check that every requested line
140+
has audio and nonempty word timings before building a captioned video. Review
141+
the timings against the actual audio; transcription is estimated alignment,
142+
not native TTS timestamps. Do not distribute words evenly across a clip.
143+
144+
- **Models:** `gemini-3.8-flash-tts` (default), `gemini-3.8-flash-lite-tts`,
145+
`gemini-3.1-flash-tts-preview`, `gemini-2.5-pro-preview-tts`, and
146+
`gemini-2.5-flash-preview-tts`. Use these exact Developer API IDs; Cloud TTS
147+
aliases such as `gemini-2.5-flash-tts` are not accepted.
148+
- **Voice:** `Kore` by default; pass another prebuilt voice or an existing custom
149+
voice ID for 3.8. Older models require prebuilt voices. Creating or replicating
150+
voices is outside this helper.
151+
- **Delivery:** Put directions in `style`, not in spoken `text`. Each line can
152+
override `style`. 3.8 uses structured annotations; older models receive a
153+
delivery prompt before the transcript. Check that directions were not spoken.
154+
Use style for pacing; numeric `speed` must be omitted or 1.
155+
- **Audio:** 3.8 returns a complete WAV. Older models return mono 16-bit PCM,
156+
which the helper wraps as WAV at the returned sample rate without resampling.
157+
No generation calls run during playback or rendering.
158+
- **Timing:** This adapter requests no native word timestamps. It uses the
159+
existing transcription pass. `lang` selects transcription language; Gemini
160+
infers speech language from the text.
161+
- **Workflow adapters:** Product-launch, faceless-explainer, and PR-video audio
162+
scripts accept `--provider gemini --voice Kore --tts-model gemini-3.8-flash-tts
163+
--style "Warm and clear"`.
164+
165+
An API error is reported as a failed line; an explicitly chosen Gemini voice
166+
never silently falls back to another provider. Check the engine's anomalies
167+
and output metadata, not only its exit code.
168+
169+
API contract: [Google's speech generation guide](https://ai.google.dev/gemini-api/docs/speech-generation).
92170

93171
## ffmpeg requirement
94172

‎skills/media-use/audio/scripts/audio.mjs‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
//
1616
// ── audio_request.json (input) ────────────────────────────────────────────────
1717
// {
18-
// "provider": "auto", // auto|heygen|elevenlabs|kokoro (override: --provider)
18+
// "provider": "auto", // auto|heygen|elevenlabs|kokoro|gemini (override: --provider)
1919
// "lang": "en", "speed": 1.0,
2020
// "lines": [ // one TTS unit each; id joins back to the caller's model
2121
// { "id": "01", "text": "...", "sfx": ["whoosh", "ui click"] }
@@ -113,7 +113,6 @@ const speed = Number(speedOverride ?? request.speed ?? 1.0) || 1.0;
113113
// ── env + HeyGen availability (the single switch) ─────────────────────────────
114114
loadEnvFromDir(hyperframesDir);
115115
const heygenOK = heygenCredential() !== null;
116-
const headers = heygenOK ? heygenAuthHeaders() : null;
117116

118117
// ── merge base: preserve sections not selected by --only ──────────────────────
119118
const audioMeta = openAudioMeta(outPath);
@@ -153,6 +152,8 @@ if (only.has("tts") && lines.length) {
153152
voiceId,
154153
lang,
155154
speed,
155+
model: flag("tts-model", request.tts_model),
156+
style: flag("style", line.style ?? request.style),
156157
wavAbs: abs,
157158
hyperframesDir,
158159
});
@@ -210,7 +211,12 @@ if (only.has("bgm")) {
210211
console.error(`· bgm: disabled`);
211212
} else if (mode === "retrieve") {
212213
try {
213-
bgm = await retrieveBgm({ query: request.bgm?.query, headers, hyperframesDir, hasVoice });
214+
bgm = await retrieveBgm({
215+
query: request.bgm?.query,
216+
headers: heygenAuthHeaders(),
217+
hyperframesDir,
218+
hasVoice,
219+
});
214220
if (bgm) {
215221
bgmFields.bgm_provider = "heygen";
216222
bgmFields.bgm_mode = "retrieve";
@@ -262,6 +268,7 @@ if (only.has("sfx")) {
262268
.map((name) => ({ id: String(l.id), name: String(name).trim() }))
263269
.filter((c) => c.name),
264270
);
271+
const headers = heygenOK && cues.length ? heygenAuthHeaders() : null;
265272
const res = await resolveSfx({ cues, heygenOK, headers, hyperframesDir, sfxLibDir });
266273
sfx = res.sfx;
267274
anomalies.push(...res.anomalies);

0 commit comments

Comments
 (0)