Skip to content

Commit 8e340d2

Browse files
authored
feat(cli): audio answers beats, downbeats, bpm, sections and describe through a provider ladder (#5632)
* feat(cli): audio answers beats, downbeats, bpm, sections and describe through a provider ladder Beats and downbeats try api, apple-mu, beat-this, then signal; sections try api, apple-mu, then signal; describe tries api, then apple-sound. bpm is read off whichever provider placed the beats. Each provider says whether it can run here and why not, and the ladder warns when it passes one over. The signal rung is a TypeScript tracker: log-band spectral flux, autocorrelation tempo with a log-normal prior, Ellis dynamic-programming beats with a windowed local tempo, a low-band bar phase, and Foote novelty sections snapped to beats. It reports no steady beat when the onsets do not repeat. The api rung posts the audio to HYPERFRAMES_AUDIO_API_URL with HYPERFRAMES_AUDIO_API_KEY and moves each answered time onto the nearest signal beat within a quarter beat, reporting what moved and what did not. beat-this runs Beat This! from a configured Python and local weights and never downloads. apple-sound runs macOS's sound classifier through a Swift helper. apple-mu reports that Music Understanding needs macOS 27. words are no longer accepted: they come from hyperframes transcribe. * docs(cli): audio's beats, downbeats, bpm, sections, describe and their provider ladder * fix(cli): remove redundant audio context comment * fix(core): keep flat audio novelty in one section * fix(cli): honor BPM provider pins before analysis
1 parent db845d9 commit 8e340d2

19 files changed

Lines changed: 2014 additions & 82 deletions

‎docs/packages/cli.mdx‎

Lines changed: 51 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -540,9 +540,57 @@ For a file not in the film yet, `--place-at <s>` answers as if it started at tha
540540
film second; `--skip-file-seconds <s>` cuts that much from the start of the file
541541
first, and the result warns when the cut part held the file's loudest moment.
542542

543-
Every measurement comes from a provider ladder. Today the only provider is
544-
local signal processing; `--quality fast|local|best` limits which providers may
545-
run, `--provider onsets=signal` pins one, and `--full` shows each ladder step.
543+
`--only` also asks for the music's structure, which is not measured unless asked:
544+
545+
- **beats**: beat times, or none with a `reason` (`no steady beat`, or too short
546+
to tell) when the music has no beat to follow.
547+
- **downbeats**: the first beat of each bar, with the `meter` (3 or 4). From
548+
signal processing these are a guess (`confidence: "low"`).
549+
- **bpm**: the tempo read off the beats (median interval), with a `curve` every
550+
4 s for music that speeds up or slows down. It always comes from the provider
551+
that answered `beats`.
552+
- **sections**: spans where the sound stays alike, each with a letter; a part
553+
that returns gets its earlier letter.
554+
- **describe**: labels for what the audio holds, each with a score.
555+
556+
Word timing is not here: use [`transcribe`](#transcribe).
557+
558+
Every measurement comes from a provider ladder, tried in this order until one
559+
answers:
560+
561+
| Capability | Ladder |
562+
| --- | --- |
563+
| moments, onsets, loudness, silence, envelope, kind | `signal` |
564+
| beats, downbeats (and so bpm) | `api`, `apple-mu`, `beat-this`, `signal` |
565+
| sections | `api`, `apple-mu`, `signal` |
566+
| describe | `api`, `apple-sound` |
567+
568+
- `api` is an audio analysis service, used when `HYPERFRAMES_AUDIO_API_URL` and
569+
`HYPERFRAMES_AUDIO_API_KEY` are both set. The audio is uploaded. Every beat,
570+
downbeat or section boundary it answers moves onto the nearest beat signal
571+
processing measured, when one is within a quarter beat; the rest are listed in
572+
`snap.unsnapped`, and `snap.medianShiftMs` says how far the moved ones went.
573+
- `apple-mu` is Apple's Music Understanding, which needs macOS 27. It is not
574+
available yet.
575+
- `beat-this` is [Beat This!](https://github.com/CPJKU/beat_this), used when
576+
`HYPERFRAMES_BEAT_THIS_PYTHON` names a Python with `beat-this` installed and
577+
its `small0` weights are on disk (`HYPERFRAMES_BEAT_THIS_CHECKPOINT`, or
578+
`beat_this-small0.ckpt` in the Torch hub cache). The CLI never downloads them;
579+
to set it up once:
580+
581+
```bash
582+
uv init --bare ~/.beat-this
583+
uv add --project ~/.beat-this "beat-this @ git+https://github.com/CPJKU/beat_this" torch torchaudio soundfile
584+
~/.beat-this/.venv/bin/python -c "from beat_this.inference import load_checkpoint; load_checkpoint('small0')"
585+
export HYPERFRAMES_BEAT_THIS_PYTHON=~/.beat-this/.venv/bin/python
586+
```
587+
- `apple-sound` is macOS's built-in sound classifier, through a small Swift
588+
helper compiled once with the Xcode command line tools.
589+
590+
A provider that cannot run, or fails, is passed over with a warning that says
591+
why. `--quality fast|local|best` limits which providers may run (`local` keeps
592+
the audio on this machine, `fast` allows only signal processing),
593+
`--provider beats=signal` pins one, and `--full` shows each ladder step.
546594
Tune the analysis with `--onset-method`, `--threshold`, `--band lo,hi`,
547595
`--hop-ms`, `--onset-ratio`, `--tail-db` and `--silence-db`. Decoding needs
548596
FFmpeg.

‎packages/cli/src/audio/analysis/analyze.test.ts‎

Lines changed: 42 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -38,10 +38,48 @@ describe("analyzeSignal", () => {
3838
expect(analysis.values.moments).toEqual({ onset: 1.1, steepest: 1.1, peak: 1.1, tailEnd: 1.2 });
3939
});
4040

41-
it("warns, and answers nothing, for a capability no provider offers yet", async () => {
42-
const analysis = await analyzeSignal(tone, { capabilities: ["beats"] });
43-
expect(analysis.values.beats).toBeNull();
44-
expect(analysis.warnings).toEqual(["beats: no provider answered (none offers it)"]);
41+
it("warns, and answers nothing, for a capability no provider offers", async () => {
42+
const analysis = await analyzeSignal(tone, { capabilities: ["describe"] }, [signalProvider]);
43+
expect(analysis.values.describe).toBeNull();
44+
expect(analysis.warnings).toEqual(["describe: no provider answered (none offers it)"]);
45+
});
46+
47+
it("reads tempo off the beats of whichever provider placed them, in file seconds", async () => {
48+
const grid: AudioAnalysisProvider<AudioSignal> = {
49+
id: "grid",
50+
local: true,
51+
capabilities: ["beats"],
52+
analyze: async () => ({ times: [0, 0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4] }),
53+
};
54+
const analysis = await analyzeSignal(tone, { capabilities: ["bpm", "beats"], fileOffset: 10 }, [
55+
grid,
56+
signalProvider,
57+
]);
58+
expect(analysis.provenance).toEqual({ bpm: "grid", beats: "grid" });
59+
expect(analysis.values.bpm).toEqual({
60+
value: 120,
61+
curve: [
62+
{ t: 10, bpm: 120 },
63+
{ t: 14, bpm: 120 },
64+
],
65+
});
66+
expect((analysis.values.beats as { times: number[] }).times[0]).toBe(10);
67+
});
68+
69+
it("keeps a service off the ladder at quality fast", async () => {
70+
const service: AudioAnalysisProvider<AudioSignal> = {
71+
id: "service",
72+
local: false,
73+
capabilities: ["beats"],
74+
analyze: async () => ({ times: [1] }),
75+
};
76+
const analysis = await analyzeSignal(
77+
tone,
78+
{ capabilities: ["beats"], quality: "fast", options: {} },
79+
[service, signalProvider],
80+
);
81+
expect(analysis.provenance.beats).toBe("signal");
82+
expect(analysis.warnings).toEqual([]);
4583
});
4684
});
4785

‎packages/cli/src/audio/analysis/analyze.ts‎

Lines changed: 40 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,19 @@ import {
22
runLadder,
33
signalDuration,
44
signalProvider,
5+
tempoOf,
56
type AnalysisQuality,
67
type AudioAnalysisProvider,
78
type AudioCapability,
89
type AudioSignal,
10+
type Beats,
11+
type LadderResult,
912
type LadderStep,
1013
} from "@hyperframes/core/audio-analysis";
14+
import { apiProvider } from "./providers/api.js";
15+
import { appleMuProvider, appleSoundProvider } from "./providers/apple.js";
16+
import { beatThisProvider } from "./providers/beatThis.js";
17+
import { shiftedTimes } from "./fileTime.js";
1118

1219
/** What `hyperframes audio` asks for when `--only` is not given. */
1320
export const DEFAULT_CAPABILITIES: readonly AudioCapability[] = [
@@ -19,9 +26,12 @@ export const DEFAULT_CAPABILITIES: readonly AudioCapability[] = [
1926
"envelope",
2027
];
2128

22-
/** Every provider the CLI knows, in default ladder order. Model and service providers join this list. */
29+
/**
30+
* Every provider the CLI knows, in default ladder order: each capability tries the ones that offer it in this order.
31+
* beats, downbeats: api, apple-mu, beat-this, signal. sections: api, apple-mu, signal. describe: api, apple-sound.
32+
*/
2333
function audioAnalysisProviders(): AudioAnalysisProvider<AudioSignal>[] {
24-
return [signalProvider];
34+
return [apiProvider, appleMuProvider, beatThisProvider, appleSoundProvider, signalProvider];
2535
}
2636

2737
export interface AnalyzeOptions {
@@ -54,36 +64,6 @@ export function envelopeHopMs(seconds: number, asked?: number): number {
5464

5565
const round = (t: number) => Math.round(t * 1000) / 1000;
5666

57-
/** Move a capability's times from signal seconds to file seconds. */
58-
function toFileTime(capability: AudioCapability, value: unknown, offset: number): unknown {
59-
if (value === null || typeof value !== "object") return value;
60-
const at = (t: number) => round(t + offset);
61-
switch (capability) {
62-
case "moments": {
63-
const m = value as Record<string, number>;
64-
return Object.fromEntries(Object.entries(m).map(([k, t]) => [k, at(t)]));
65-
}
66-
case "onsets": {
67-
const o = value as { events: { t: number; strength: number }[] };
68-
return { ...o, events: o.events.map((e) => ({ ...e, t: at(e.t) })) };
69-
}
70-
case "silence":
71-
return (value as { start: number; end: number }[]).map((s) => ({
72-
start: at(s.start),
73-
end: at(s.end),
74-
}));
75-
case "loudness": {
76-
const l = value as { clipping: { count: number; firstAt: number | null } };
77-
const firstAt = l.clipping.firstAt === null ? null : at(l.clipping.firstAt);
78-
return { ...l, clipping: { ...l.clipping, firstAt } };
79-
}
80-
case "envelope":
81-
return { startsAt: round(offset), ...(value as object) };
82-
default:
83-
return value;
84-
}
85-
}
86-
8767
function within(value: unknown, from: number, to: number): unknown {
8868
if (!value || typeof value !== "object" || !("events" in value)) return value;
8969
const onsets = value as { events: { t: number }[] };
@@ -123,6 +103,9 @@ export async function analyzeSignal(
123103
request: AnalyzeOptions,
124104
providers: readonly AudioAnalysisProvider<AudioSignal>[] = audioAnalysisProviders(),
125105
): Promise<Analysis> {
106+
const { beats, bpm } = request.provider ?? {};
107+
if (beats && bpm && beats !== bpm) throw new Error("beats and bpm must use the same provider");
108+
const beatProvider = beats ?? bpm;
126109
const framing = frame(signal, request.context);
127110
const analysis: Analysis = {
128111
analyzed: round(signalDuration(framing.windowed)),
@@ -131,20 +114,37 @@ export async function analyzeSignal(
131114
ladder: {},
132115
warnings: [],
133116
};
117+
const ladders = new Map<AudioCapability, Promise<LadderResult>>();
118+
const ladderFor = (capability: AudioCapability) => {
119+
let result = ladders.get(capability);
120+
if (!result) {
121+
const withContext = capability === "onsets" && framing.cut;
122+
result = runLadder(capability, withContext ? signal : framing.windowed, providers, {
123+
quality: request.quality,
124+
provider: capability === "beats" ? beatProvider : request.provider?.[capability],
125+
options: optionsFor(capability, request, analysis.analyzed),
126+
});
127+
ladders.set(capability, result);
128+
}
129+
return result;
130+
};
134131
for (const capability of request.capabilities) {
135-
// Onsets see the context, so a window's edge is not mistaken for a hit, then keep only the window's own.
136132
const withContext = capability === "onsets" && framing.cut;
137-
const result = await runLadder(capability, withContext ? signal : framing.windowed, providers, {
138-
quality: request.quality,
139-
provider: request.provider?.[capability],
140-
options: optionsFor(capability, request, analysis.analyzed),
141-
});
133+
const result =
134+
capability === "bpm" ? tempoFromBeats(await ladderFor("beats")) : await ladderFor(capability);
142135
const value = withContext ? within(result.value, framing.before, framing.end) : result.value;
143136
const offset = (request.fileOffset ?? 0) - (withContext ? framing.before : 0);
144-
analysis.values[capability] = toFileTime(capability, value, offset);
137+
analysis.values[capability] = shiftedTimes(capability, value, offset);
145138
analysis.provenance[capability] = result.provenance;
146139
analysis.ladder[capability] = result.steps;
147-
analysis.warnings.push(...result.warnings);
140+
const said = new Set(analysis.warnings);
141+
analysis.warnings.push(...result.warnings.filter((w) => !said.has(w)));
148142
}
149143
return analysis;
150144
}
145+
146+
/** Tempo is read off the beats of whichever provider placed them, so the two never disagree. */
147+
function tempoFromBeats(beats: LadderResult): LadderResult {
148+
const times = (beats.value as Beats | null)?.times;
149+
return { ...beats, value: times ? tempoOf(times) : null };
150+
}
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
import type {
2+
AudioCapability,
3+
Onsets,
4+
SnapReport,
5+
Span,
6+
Tempo,
7+
} from "@hyperframes/core/audio-analysis";
8+
9+
const round = (t: number) => Math.round(t * 1000) / 1000;
10+
11+
type At = (t: number) => number;
12+
13+
/** A snap report's kept times are times too. */
14+
function withSnap<V extends { snap?: SnapReport }>(value: V, at: At): V {
15+
return value.snap
16+
? { ...value, snap: { ...value.snap, unsnapped: value.snap.unsnapped.map(at) } }
17+
: value;
18+
}
19+
20+
const SHIFTS: Partial<Record<AudioCapability, (value: never, at: At, offset: number) => unknown>> =
21+
{
22+
moments: (m: Record<string, number>, at) =>
23+
Object.fromEntries(Object.entries(m).map(([k, t]) => [k, at(t)])),
24+
onsets: (o: Onsets, at) => ({ ...o, events: o.events.map((e) => ({ ...e, t: at(e.t) })) }),
25+
silence: (spans: Span[], at) => spans.map((s) => ({ start: at(s.start), end: at(s.end) })),
26+
loudness: (l: { clipping: { count: number; firstAt: number | null } }, at) => ({
27+
...l,
28+
clipping: {
29+
...l.clipping,
30+
firstAt: l.clipping.firstAt === null ? null : at(l.clipping.firstAt),
31+
},
32+
}),
33+
envelope: (e: object, _at, offset) => ({ startsAt: round(offset), ...e }),
34+
beats: (b: { times: number[]; snap?: SnapReport }, at) =>
35+
withSnap({ ...b, times: b.times.map(at) }, at),
36+
downbeats: (b: { times: number[]; snap?: SnapReport }, at) =>
37+
withSnap({ ...b, times: b.times.map(at) }, at),
38+
bpm: (tempo: Tempo, at) => ({
39+
...tempo,
40+
curve: tempo.curve.map((p) => ({ ...p, t: at(p.t) })),
41+
}),
42+
sections: (s: { spans: Span[]; snap?: SnapReport }, at) =>
43+
withSnap(
44+
{
45+
...s,
46+
spans: s.spans.map((span) => ({ ...span, start: at(span.start), end: at(span.end) })),
47+
},
48+
at,
49+
),
50+
};
51+
52+
/** Move a capability's times from signal seconds to file seconds. */
53+
export function shiftedTimes(capability: AudioCapability, value: unknown, offset: number): unknown {
54+
if (value === null || typeof value !== "object") return value;
55+
const shift = SHIFTS[capability];
56+
return shift ? shift(value as never, (t) => round(t + offset), offset) : value;
57+
}

‎packages/cli/src/audio/analysis/format.ts‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,15 @@
11
import type {
2+
Beats,
3+
Downbeats,
24
Envelope,
35
KindGuess,
46
Loudness,
57
Moments,
68
Onsets,
9+
Sections,
10+
SnapReport,
711
Span,
12+
Tempo,
813
} from "@hyperframes/core/audio-analysis";
914
import type { AudioItem, AudioReport } from "./run.js";
1015
import { filmTime } from "./targets.js";
@@ -60,13 +65,62 @@ function silenceLine(silence: Span[], at: At): string | null {
6065
: null;
6166
}
6267

68+
function snapNote(snap: SnapReport | undefined): string {
69+
if (!snap) return "";
70+
const total = snap.snapped + snap.unsnapped.length;
71+
const median = snap.medianShiftMs === null ? "" : `, median ${snap.medianShiftMs} ms`;
72+
return `; ${snap.snapped}/${total} snapped to ${snap.reference}${median}`;
73+
}
74+
75+
const listed = (times: number[], at: At) => {
76+
const shown = times.slice(0, ONSETS_SHOWN).map(at).join(" ");
77+
return times.length > ONSETS_SHOWN ? `${shown} +${times.length - ONSETS_SHOWN}` : shown;
78+
};
79+
80+
function beatsLine(beats: Beats & { snap?: SnapReport }, at: At): string {
81+
if (!beats.times.length) return ` beats none${beats.reason ? ` (${beats.reason})` : ""}`;
82+
return ` beats ${beats.times.length}: ${listed(beats.times, at)}${snapNote(beats.snap)}`;
83+
}
84+
85+
function downbeatsLine(downbeats: Downbeats & { snap?: SnapReport }, at: At): string | null {
86+
if (!downbeats.times.length) return null;
87+
const meter = downbeats.meter ? ` in ${downbeats.meter}` : "";
88+
const low = downbeats.confidence === "low" ? ", a guess" : "";
89+
return ` downbeats${meter}${low}: ${listed(downbeats.times, at)}${snapNote(downbeats.snap)}`;
90+
}
91+
92+
function tempoLine(tempo: Tempo): string | null {
93+
if (tempo.value === null) return null;
94+
const bpms = tempo.curve.map((p) => p.bpm);
95+
const spread =
96+
bpms.length && Math.max(...bpms) - Math.min(...bpms) >= 2
97+
? ` (${Math.min(...bpms)}–${Math.max(...bpms)})`
98+
: "";
99+
return ` tempo ${tempo.value} BPM${spread}`;
100+
}
101+
102+
function sectionsLine(sections: Sections, at: At): string {
103+
const spans = sections.spans.map((s) => `${s.label} ${at(s.start)}–${at(s.end)}`).join(" ");
104+
return ` sections ${spans}${snapNote(sections.snap)}`;
105+
}
106+
107+
function describeLine(description: { text?: string; tags: { label: string; score: number }[] }) {
108+
const tags = description.tags.map((t) => `${t.label} ${t.score.toFixed(2)}`).join(", ");
109+
return ` sounds like ${[description.text, tags].filter(Boolean).join(" — ")}`;
110+
}
111+
63112
/** One line per measurement the item holds, in reading order. */
64113
const LINES: [string, (value: never, at: At) => string | null][] = [
65114
["moments", momentsLine],
66115
["onsets", hitsLine],
67116
["loudness", loudnessLine],
68117
["silence", silenceLine],
69118
["envelope", (envelope: Envelope) => ` ${sparkline(envelope)}`],
119+
["describe", describeLine],
120+
["bpm", tempoLine],
121+
["beats", beatsLine],
122+
["downbeats", downbeatsLine],
123+
["sections", sectionsLine],
70124
];
71125

72126
function headline(item: AudioItem, where: string): string {

0 commit comments

Comments
 (0)