Skip to content

Commit 2c1d9d5

Browse files
authored
feat(cli): read a whole recording as voice, effect and music lines (#6056)
* feat(cli): read a whole recording as voice, effect and music lines `hyperframes audio <file> --read` turns an episode or reference video into time-stamped lines an agent can read or grep: sentences with rate, pitch and ending; pauses; sound effects grouped by identical sound, with band, attack, tail and level over the music; and the music bed split at its cut-outs into pieces that start, resume, return or are new. Effects and music come from stems separated by the service at HYPERFRAMES_AUDIO_STEMS_URL (vocals, drums, bass, guitar, piano, other). Without stems, effects are read from the mix outside speech; with neither stems nor a transcript, voice and effects are not read, and each case is reported in a warning. Results are cached per file content. * refactor(cli): name the service error parse instead of commenting it * fix(cli): call HeyGen's audio services with the CLI's HeyGen credential Stems for `audio --read` and the `api` analysis provider only ran when HYPERFRAMES_AUDIO_STEMS_URL / HYPERFRAMES_AUDIO_API_URL and HYPERFRAMES_AUDIO_API_KEY were set, so someone with HEYGEN_API_KEY or a saved `hyperframes auth login --api-key` still got "not set" and had to know the internal route. Both now default to /v3/audio/stems and /v3/audio/analysis on the HeyGen API (HEYGEN_API_URL when set) with the credential the other HeyGen commands use. The URL variables still point at another service, which then needs its own HYPERFRAMES_AUDIO_API_KEY, so the HeyGen credential never goes to a host it was not issued for. * fix(cli): keep audio on the machine unless --quality best is asked for `hyperframes audio` defaulted to --quality best, so anyone with a HeyGen credential uploaded their audio, and could be billed, for an ordinary beats or --read run. tts, transcribe and remove-background never leave the machine unasked. The default is now local for both measuring and --read. best is unchanged and still uploads to /v3/audio/analysis and /v3/audio/stems. When a capability goes unanswered because a service was left out, or --read reads without stems, the warning says --quality best would add it. * fix(cli): bound stem transfers and preserve recording timestamps * fix(cli): an empty audio service URL keeps the service off HyperFrames Desktop blanks HYPERFRAMES_AUDIO_API_URL and its key while the analysis route is behind a flag, but gives runs HEYGEN_API_KEY and HEYGEN_API_URL for its gateway. Falling back to that credential would let a run that passes --quality best reach the route anyway. An empty URL now turns the service off, whatever credential is present. * fix(ci): capture package name in release-age exclusion (cherry picked from commit f9c6ba3) * fix(cli): resend the stems request with its key until the service answers The stems service answers once the stems are made, which can outlast the connection, and answers 409 request_in_progress to a resend while the first request runs. A failed job's reason is failure_message. * fix(cli): retry incomplete stem separations * test: validate audio mock route handlers before dispatch * test: trace premature Windows CLI runner exits * test: remove temporary Windows runner diagnostics
1 parent 2faf660 commit 2c1d9d5

23 files changed

Lines changed: 2424 additions & 78 deletions

‎docs/packages/cli.mdx‎

Lines changed: 72 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -583,6 +583,50 @@ first, and the result warns when the cut part held the file's loudest moment.
583583

584584
Word timing is not here: use [`transcribe`](#transcribe).
585585

586+
#### Reading a whole recording
587+
588+
`--read` turns one recording — an episode, a reference video — into
589+
time-stamped lines an agent can read or grep, instead of measurements of one sound:
590+
591+
```bash
592+
npx hyperframes audio refs/episode.mp4 --read # the whole recording
593+
npx hyperframes audio refs/episode.mp4 --read --range 60,70 # only what overlaps 60–70 s
594+
```
595+
596+
- **voice**: one line per sentence with its text, speaking rate against the
597+
recording's mean, pitch when it is 3 semitones or more off the median, and a
598+
rising or falling end; pauses of 0.4 s or more between sentences.
599+
- **effects**: each sound effect with a kind (`riser`, `boom`, `thump`, `whoosh`,
600+
`click`, `tick`, `ding`, `hit`), the group of identical sounds it belongs to
601+
(`S1` is the most used), band, attack, tail, how far it stands over the music,
602+
and where it falls in a pause. Quick runs of one sound are a single line with a
603+
count; hits on the music's beat are the music's own drums, and sounds more than
604+
20 dB under the recording are separation residue nobody hears; both are left out.
605+
- **music**: the bed split at its cut-outs, each part naming its piece, the
606+
instruments that carry it and its tempo, whether it starts, resumes after a
607+
break, returns or is new, and where it fades.
608+
609+
Sentences come from a transcript: `--transcript <file>`, or
610+
`<recording>.transcript.json` beside the recording. Without one, voice lines
611+
carry no words.
612+
613+
Effects and music are read from stems. At `--quality best` the
614+
recording is separated into vocals, drums, bass, guitar, piano and other by
615+
HeyGen's `/v3/audio/stems`, with your HeyGen credential (see
616+
[audio services](#audio-services)): it is uploaded as 192 kb/s MP3 through the
617+
same origin's direct-upload endpoints. The separation request waits for the
618+
stems; an in-progress response or dropped connection retries the same request
619+
with the same idempotency key (`HYPERFRAMES_AUDIO_STEMS_TIMEOUT_MS`, 30 minutes
620+
by default). Effects are read
621+
from the drums stem and the music from the other four. Without stems, effects are
622+
read from the mix outside speech and the music is not read; with neither stems nor
623+
a transcript, voice and effects are not read either, because the mix alone does
624+
not tell speech from music. Each case says so in a warning.
625+
626+
The result is computed once per file content and kept in `.hyperframes/read-audio`
627+
under the project, with the stems; the text output ends with the path of the
628+
whole recording's text, and `--json` returns the analysis.
629+
586630
Every measurement comes from a provider ladder, tried in this order until one
587631
answers:
588632

@@ -593,13 +637,12 @@ answers:
593637
| sections | `api`, `apple-mu`, `signal` |
594638
| describe | `api`, `apple-sound` |
595639

596-
- `api` is an audio analysis service, used when `HYPERFRAMES_AUDIO_API_URL` and
597-
`HYPERFRAMES_AUDIO_API_KEY` are both set. The audio is uploaded as 16 kHz mono
640+
- `api` is HeyGen's `/v3/audio/analysis`, used when a HeyGen credential is
641+
found (see [audio services](#audio-services)). The audio is uploaded as 16 kHz mono
598642
WAV through the same origin’s public asset direct-upload and completion endpoints.
599643
Each decoded signal is uploaded once; analysis requests send its finalized asset ID as JSON.
600-
The API key needs asset upload and audio-analysis permissions. Channels are averaged unless cancellation removes over 90% of the loudest
601-
channel’s energy, in which case that channel is used. Requests carry the key in an `x-api-key` header and a new `Idempotency-Key` on each
602-
request; the answer may come wrapped in `{ "data": … }`. Every beat,
644+
The credential needs asset upload and audio permissions. Channels are averaged unless cancellation removes over 90% of the loudest
645+
channel’s energy, in which case that channel is used. Each request carries a new `Idempotency-Key`; the answer may come wrapped in `{ "data": … }`. Every beat,
603646
downbeat or section boundary it answers moves onto the nearest beat signal
604647
processing measured, when one is within a quarter beat; the rest are listed in
605648
`snap.unsnapped`, and `snap.medianShiftMs` says how far the moved ones went.
@@ -621,13 +664,34 @@ answers:
621664
helper compiled once with the Xcode command line tools.
622665

623666
A provider that cannot run, or fails, is passed over with a warning that says
624-
why. `--quality fast|local|best` limits which providers may run (`local` keeps
625-
the audio on this machine, `fast` allows only signal processing),
626-
`--provider beats=signal` pins one, and `--full` shows each ladder step.
667+
why. `--quality local|best|fast` limits which providers may run: `local`, the
668+
default, keeps the audio on this machine; `best` also sends it to the audio
669+
services below; `fast` allows only signal processing. When a capability goes
670+
unanswered because a service was left out, a warning says `--quality best` could
671+
answer it. `--provider beats=signal` pins a capability to one provider, and
672+
`--full` shows each ladder step.
627673
Tune the analysis with `--onset-method`, `--threshold`, `--band lo,hi`,
628674
`--hop-ms`, `--onset-ratio`, `--tail-db` and `--silence-db`. Decoding needs
629675
FFmpeg.
630676

677+
#### Audio services
678+
679+
Only at `--quality best` is audio uploaded. Stems and the `api` provider then
680+
call HeyGen, which may bill your account, with the same credential as the other
681+
HeyGen commands: `HEYGEN_API_KEY`, `HYPERFRAMES_API_KEY`, `HEYGEN_ACCESS_TOKEN`,
682+
or a credential saved by `hyperframes auth login`, on the API at
683+
`HEYGEN_API_URL` when set. API keys are sent as `x-api-key`; OAuth credentials
684+
are sent as bearer tokens. If the service refuses a request, analysis falls
685+
back with a warning. Without a credential, stems are skipped and `api` is
686+
passed over. At `local`, the default, no credential is looked up and nothing
687+
leaves the machine.
688+
689+
To use another service, set `HYPERFRAMES_AUDIO_STEMS_URL` or
690+
`HYPERFRAMES_AUDIO_API_URL` to its full URL together with
691+
`HYPERFRAMES_AUDIO_API_KEY`, which is sent as `x-api-key`. Your HeyGen
692+
credential is never sent to a URL set this way. Setting either URL to an empty
693+
string turns that service off, whatever credential is present.
694+
631695
## Look at it
632696

633697
### `normalize-audio`

‎packages/cli/src/audio/analysis/analyze.test.ts‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,34 @@ describe("analyzeSignal", () => {
8383
});
8484
});
8585

86+
describe("analyzeSignal quality", () => {
87+
const service: AudioAnalysisProvider<AudioSignal> = {
88+
id: "service",
89+
local: false,
90+
capabilities: ["describe"],
91+
analyze: async () => ({ tags: [{ label: "speech", score: 0.9 }] }),
92+
};
93+
94+
it("keeps audio on this machine unless best is asked for, and says what best would add", async () => {
95+
const local = await analyzeSignal(tone, { capabilities: ["describe"] }, [service]);
96+
expect(local.values.describe).toBeNull();
97+
expect(local.ladder.describe).toEqual([
98+
{
99+
provider: "service",
100+
outcome: "excluded",
101+
reason: "quality local keeps audio on this machine",
102+
},
103+
]);
104+
expect(local.warnings.at(-1)).toContain("--quality best");
105+
106+
const best = await analyzeSignal(tone, { capabilities: ["describe"], quality: "best" }, [
107+
service,
108+
]);
109+
expect(best.provenance.describe).toBe("service");
110+
expect(best.warnings).toEqual([]);
111+
});
112+
});
113+
86114
describe("envelopeHopMs", () => {
87115
it("keeps short sounds at 5 ms and coarsens long ones to about 2,000 values", () => {
88116
expect(envelopeHopMs(1)).toBe(5);

‎packages/cli/src/audio/analysis/analyze.ts‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ function audioAnalysisProviders(): AudioAnalysisProvider<AudioSignal>[] {
3636

3737
export interface AnalyzeOptions {
3838
capabilities: readonly AudioCapability[];
39+
/** Defaults to local: audio leaves the machine only when best is asked for. */
3940
quality?: AnalysisQuality;
4041
/** Pin a capability to one provider, skipping the ladder. */
4142
provider?: Partial<Record<AudioCapability, string>>;
@@ -103,6 +104,7 @@ export async function analyzeSignal(
103104
request: AnalyzeOptions,
104105
providers: readonly AudioAnalysisProvider<AudioSignal>[] = audioAnalysisProviders(),
105106
): Promise<Analysis> {
107+
const quality = request.quality ?? "local";
106108
const { beats, bpm } = request.provider ?? {};
107109
if (beats && bpm && beats !== bpm) throw new Error("beats and bpm must use the same provider");
108110
const beatProvider = beats ?? bpm;
@@ -120,7 +122,7 @@ export async function analyzeSignal(
120122
if (!result) {
121123
const withContext = capability === "onsets" && framing.cut;
122124
result = runLadder(capability, withContext ? signal : framing.windowed, providers, {
123-
quality: request.quality,
125+
quality,
124126
provider: capability === "beats" ? beatProvider : request.provider?.[capability],
125127
options: optionsFor(capability, request, analysis.analyzed),
126128
});
@@ -140,9 +142,23 @@ export async function analyzeSignal(
140142
const said = new Set(analysis.warnings);
141143
analysis.warnings.push(...result.warnings.filter((w) => !said.has(w)));
142144
}
145+
analysis.warnings.push(...serviceHint(quality, analysis, request.capabilities));
143146
return analysis;
144147
}
145148

149+
function serviceHint(
150+
quality: AnalysisQuality,
151+
analysis: Analysis,
152+
capabilities: readonly AudioCapability[],
153+
): string[] {
154+
const leftToService = (capability: AudioCapability) =>
155+
analysis.values[capability] == null &&
156+
(analysis.ladder[capability] ?? []).some((s) => s.outcome === "excluded");
157+
return quality === "local" && capabilities.some(leftToService)
158+
? ["a service could have answered what is missing: --quality best uploads the audio to HeyGen"]
159+
: [];
160+
}
161+
146162
/** Tempo is read off the beats of whichever provider placed them, so the two never disagree. */
147163
function tempoFromBeats(beats: LadderResult): LadderResult {
148164
const times = (beats.value as Beats | null)?.times;

‎packages/cli/src/audio/analysis/decode.ts‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,11 @@ export interface DecodeWindow {
5555
seconds?: number;
5656
}
5757

58+
export interface DecodeFormat {
59+
sampleRate?: number;
60+
mono?: boolean;
61+
}
62+
5863
export interface Decoded {
5964
signal: AudioSignal;
6065
stream: AudioStreamInfo;
@@ -65,9 +70,15 @@ export async function decodeAudio(
6570
file: string,
6671
window: DecodeWindow = {},
6772
probed?: AudioStreamInfo,
73+
format: DecodeFormat = {},
6874
): Promise<Decoded | null> {
69-
const info = probed ?? (await probeAudio(file));
70-
if (!info) return null;
75+
const probedInfo = probed ?? (await probeAudio(file));
76+
if (!probedInfo) return null;
77+
const info = {
78+
...probedInfo,
79+
sampleRate: format.sampleRate ?? probedInfo.sampleRate,
80+
channels: format.mono ? 1 : probedInfo.channels,
81+
};
7182
const seek = window.from ? ["-ss", String(window.from)] : [];
7283
const length = window.seconds === undefined ? [] : ["-t", String(window.seconds)];
7384
const { stdout } = await run(
@@ -81,6 +92,8 @@ export async function decodeAudio(
8192
...length,
8293
"-map",
8394
"0:a:0",
95+
...(format.mono ? ["-ac", "1"] : []),
96+
...(format.sampleRate ? ["-ar", String(format.sampleRate)] : []),
8497
"-f",
8598
"f32le",
8699
"-acodec",

0 commit comments

Comments
 (0)