Skip to content

Commit acd5c30

Browse files
feat(cli): read audio stems through your HeyGen account (#6993)
* feat(cli): audio --read organises the reading by stem Measures where each stem is heard, leaves the drums' recurring hits to the music while it plays, names the stem of each remaining effect, writes a voice-free mix next to the stems, and describes each audible stem and the whole recording with Gemini. * feat(cli): audio --read reads stems as vocals, drums, bass and melodic Piano, guitar and other are mixed into melodic.wav next to the stems; levels, spans, descriptions, effects and pieces are read per group, and a group heard for under two seconds or 5% of the recording is left out of the reading. * feat(cli): audio --read describes the six stems and writes a BGM prompt from them Stems are read one by one again; a stem heard under two seconds or 5% of the recording is still left out. Each stem's description gives the times of its changes and names the effects it hears. The overall and the BGM prompt are written from those descriptions and the measured spans, not by listening to the mix again; the BGM prompt orders the arrangement without seconds. * fix(cli): audio --read writes no BGM prompt for a recording without music Foley and effects in a rhythm are not music; the reading says so instead of a prompt that would make a remix generate a track the reference never had. * fix(cli): audio --read prompts name no example sounds * fix(cli): describe audio through HeyGen and keep effects measured * docs(cli): document measured effect events without sound kinds * fix(cli): clear audio description review checks Use portable stem path assertions and simplify response validation. Keep auth-derived URLs out of cache hashes and trim redundant comments. — Jerrai --------- Co-authored-by: Xuanru Li <[email protected]>
1 parent 0bcd816 commit acd5c30

14 files changed

Lines changed: 1154 additions & 148 deletions

File tree

‎docs/guides/voice-and-audio.mdx‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,32 @@ npx hyperframes transcribe interview.mp4 --language es
5656
Pass `--language` when the speech isn't English. It also imports an existing `.srt` or `.vtt`. For a talking-head clip that needs
5757
captions or a tighter cut, see [Captions and recuts](/guides/captions-and-recuts).
5858

59+
## Read a reference soundtrack
60+
61+
To understand an existing recording before making a new soundtrack, run:
62+
63+
```bash
64+
npx hyperframes audio reference.mp4 --read --quality best
65+
```
66+
67+
With a HeyGen credential, the recording is separated into six stems: vocals,
68+
drums, bass, guitar, piano and other. The reading describes each audible stem,
69+
then gives an overall description. `voice-free mix:` points to a mix without the
70+
vocal stem. `bgm prompt:` describes the music for a music generator, or says
71+
there is no music. These descriptions use HeyGen; a separate Google or Gemini
72+
API key is not used. Without a HeyGen credential, or if the description service
73+
fails, a warning explains why descriptions are unavailable instead of guessing.
74+
75+
Timed effects are named `effect` and grouped as `S1`, `S2` and so on, without
76+
assigning sound kinds. Each keeps its source stem, time, frequency band, attack,
77+
tail, peak level and level above the surrounding music. Model descriptions can
78+
explain what a sound resembles; their timing is approximate. The timed lines
79+
carry the measurements.
80+
81+
Sign in with `npx hyperframes auth login`, or set `HEYGEN_API_KEY`, before using
82+
the HeyGen services. See [audio analysis](/packages/cli#audio) for transcript,
83+
range and local-analysis options.
84+
5985
## Get captions right
6086

6187
Read the transcript before it goes on screen: names, product terms and numbers are where recognition slips. Keep

‎docs/packages/cli.mdx‎

Lines changed: 14 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -818,19 +818,20 @@ npx hyperframes audio refs/episode.mp4 --read --range 60,70 # only what overla
818818
the `pause` kind and say what still plays through them: the effects in them,
819819
the music or ambience with its times, the recording's level when no bed is
820820
read, or `silent` under -55 dBFS.
821-
- **effects**: each sound effect with a kind (`riser`, `boom`, `thump`, `whoosh`,
822-
`click`, `tick`, `ding`, `hit`), the group of identical sounds it belongs to
823-
(`S1` is the most used), band, attack, tail, its peak, and where it falls in a
824-
pause. The peak is the loudest 20 ms of the whole recording at the effect, in
825-
dBFS, so it does not depend on how a stem was separated or encoded; the figure
826-
after it names its reference: `+4 dB over music` against the music's level in
827-
the second before the effect (after it, when the music starts with it), `+12 dB
828-
over what played before` against the recording's median level in the half
829-
second before, or `out of silence`. A sound ends where the next one starts, so
830-
a run of drum hits is not read as one long riser. Quick runs of one sound are a
831-
single line with a count; hits on the music's beat are the music's own drums,
832-
and sounds more than 20 dB under the recording are separation residue nobody
833-
hears; both are left out.
821+
- **effects**: every event is labelled `effect`, without assigning a sound kind.
822+
Similar sounds belong to groups `S1`, `S2` and so on (`S1` is the most used).
823+
Each event keeps its time, source stem, frequency band, attack, tail, peak,
824+
level above the surrounding music, and position within a pause. The peak is
825+
the loudest 20 ms of the whole recording at the effect, in dBFS, so it does
826+
not depend on how a stem was separated or encoded; the figure after it names
827+
its reference: `+4 dB over music` against the music's level in the second
828+
before the effect (after it, when the music starts with it), `+12 dB over what
829+
played before` against the recording's median level in the half second
830+
before, or `out of silence`. A sound ends where the next one starts. Quick
831+
runs of one sound are a single line with a count; hits on the music's beat
832+
are the music's own drums, and sounds more than 20 dB under the recording are
833+
separation residue nobody hears; both are left out. Group summaries give the
834+
group ids and counts, not sound kinds.
834835
- **music**: the bed split at its cut-outs, each part naming its piece, the
835836
instruments that carry it and its tempo, whether it starts, resumes after a
836837
break, returns or is new, and where it fades. Music and ambience must be

‎packages/cli/src/audio/analysis/providers/shared.ts‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -98,22 +98,22 @@ async function heygenHeaders(): Promise<Record<string, string> | string> {
9898

9999
/** An explicit service URL requires its own key; never forward the HeyGen credential to it. */
100100
export async function serviceAccess(
101-
env: { url: string; key: string; timeout: string },
101+
env: { url: string; key: string; timeout: string } | null,
102102
path: string,
103103
defaultMs: number,
104104
): Promise<ServiceAccess | string> {
105-
const timeout = Number(process.env[env.timeout]);
105+
const timeout = env ? Number(process.env[env.timeout]) : 0;
106106
const timeoutMs = timeout > 0 ? timeout : defaultMs;
107-
const url = process.env[env.url]?.trim();
108-
const key = process.env[env.key]?.trim();
109-
if (url === "") return `${env.url} is empty, which turns this service off`;
107+
const url = env ? process.env[env.url]?.trim() : undefined;
108+
const key = env ? process.env[env.key]?.trim() : undefined;
109+
if (env && url === "") return `${env.url} is empty, which turns this service off`;
110110
if (key)
111111
return {
112112
url: url || `${resolveCloudBaseUrl()}${path}`,
113113
headers: { "x-api-key": key },
114114
timeoutMs,
115115
};
116-
if (url) return `${env.url} is set but ${env.key} is not`;
116+
if (env && url) return `${env.url} is set but ${env.key} is not`;
117117
const headers = await heygenHeaders();
118118
if (typeof headers === "string") return headers;
119119
return { url: `${resolveCloudBaseUrl()}${path}`, headers, timeoutMs };

0 commit comments

Comments
 (0)