Skip to content

Commit 9f09861

Browse files
perf(studio): zooming a long timeline re-decodes and redraws less (#5142)
* perf(studio): reuse filmstrip frames when a zoom changes a strip's width A zoom that changed a strip's frame count decoded the whole strip again on the main thread, and showed the clip's single poster frame until it landed. Frames are now sampled at each slice's left edge, so a strip twice as long holds every frame of the shorter one; decoded frames are shared per source and time, so zooming out decodes nothing and zooming in decodes only the new half. The strip on screen stays until the new width is ready. * perf(studio): redraw only the waveforms on or near the screen after a zoom A zoom resizes every mounted clip, and every mounted waveform redrew in that same frame, on screen or not. A waveform now draws only while it is on or near the screen (one shared observer, a screen of margin each side) and draws when it comes near if a zoom changed it meanwhile. * perf(studio): never start a beat-analysis stage while the timeline zooms Analyzing a long music track runs for over a second on the main thread, and when it started late it landed inside a zoom. The analysis now awaits an optional pause before each long stage; Studio's waits for the timeline to rest and the browser to go idle. Results are unchanged. The tempo detector's stage is one call, so a zoom that starts inside it still waits for it. * fix(studio): reuse a filmstrip frame only where its own strip would show it A shared frame was keyed by the time a strip asked for, not the time it was decoded at, so a coarse strip's keyframe-early frame could land in a finer strip and run its frames backwards. A frame now carries its decoded time and is reused only when the strip would have accepted that keyframe itself. The frame-count ladder now tops out at a power of two (32), so its top step shares frames with the one below instead of none. * feat(studio): show the clip's last frame at the end of every filmstrip Left-edge sampling never showed a clip's final slice, so an end card or a fade-out was missing from its strip. Each strip now also decodes the clip's last frame (at its own time, never an earlier keyframe), shared across zoom steps like the others, and the last tile shows it. The strip on screen is now held per clip and range, not per props: a clip crossing between on and off screen keeps it, and a clip narrowed to one frame lets it go, so widening again never re-decodes a dropped width. * docs(studio): keep the filmstrip files at or under main's comment share The comment ratchet allows no file a higher share of comment lines than on main; the added explanations move to the PR description. * fix(studio): keep a held filmstrip leased so a full cache cannot evict it mid-zoom The hold leased the strip on screen only once a new width was requested, so for one commit it had no lease; with the cache over budget it was evicted, the poster showed, and the old width decoded again. The strip on screen is now leased for as long as it is the clip's, and shown only while held. A frame that fails to decode takes the frame before it, so every slot keeps its time and the last tile still shows the end. * refactor(studio): one owner for which clip a held filmstrip belongs to The hold checked the clip twice, in its lease and again before showing it; the lease alone decides now, and the cross-clip test pins it. A test now also pins that a filled frame is not charged to the cache. * fix(core): hold the tempo detector until idle after a slow import * fix(studio): scope shared video frames to timeline sessions * chore(core): keep beat analysis comments within the ratchet
1 parent dbb8222 commit 9f09861

13 files changed

Lines changed: 862 additions & 119 deletions

‎packages/core/src/beats/beatDetection.test.ts‎

Lines changed: 71 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
import { describe, expect, it } from "vitest";
1+
import { describe, expect, it, vi } from "vitest";
22

3-
import { loadBpmDetective } from "./beatDetection";
3+
import { analyzeMusicFromBuffer, loadBpmDetective } from "./beatDetection";
44

55
describe("loadBpmDetective", () => {
66
it("resolves to null when the import is unavailable", async () => {
@@ -39,3 +39,72 @@ describe("loadBpmDetective", () => {
3939
expect(detect).toBe(fn);
4040
});
4141
});
42+
43+
describe("analyzeMusicFromBuffer", () => {
44+
it("waits on the caller's pause before each long stage", async () => {
45+
const sampleRate = 8000;
46+
const channel = new Float32Array(sampleRate * 2);
47+
for (let beat = 0; beat < 4; beat++) channel.fill(0.9, beat * 4000, beat * 4000 + 80);
48+
const audio = {
49+
getChannelData: () => channel,
50+
sampleRate,
51+
duration: 2,
52+
} as unknown as AudioBuffer;
53+
const held: (() => void)[] = [];
54+
const pause = () => new Promise<void>((resolve) => held.push(resolve));
55+
let done = false;
56+
const analysis = analyzeMusicFromBuffer(audio, { pause }).then((result) => {
57+
done = true;
58+
return result;
59+
});
60+
for (let stage = 1; stage <= 3; stage++) {
61+
await vi.waitFor(() => expect(held).toHaveLength(stage));
62+
expect(done).toBe(false);
63+
held[stage - 1]!();
64+
}
65+
await expect(analysis).resolves.toMatchObject({ sampleRate });
66+
});
67+
68+
it("rechecks pause after a slow tempo-detector import", async () => {
69+
let releaseImport!: () => void;
70+
const importGate = new Promise<void>((resolve) => (releaseImport = resolve));
71+
let importStarted = false;
72+
let moving = false;
73+
const movingAtDetect: boolean[] = [];
74+
const detect = vi.fn(() => {
75+
movingAtDetect.push(moving);
76+
return 120;
77+
});
78+
vi.resetModules();
79+
vi.doMock("bpm-detective", async () => {
80+
importStarted = true;
81+
await importGate;
82+
return { default: detect };
83+
});
84+
try {
85+
const { analyzeMusicFromBuffer: analyze } = await import("./beatDetection");
86+
const sampleRate = 8000;
87+
const audio = {
88+
getChannelData: () => new Float32Array(sampleRate * 2),
89+
sampleRate,
90+
duration: 2,
91+
} as unknown as AudioBuffer;
92+
const waiters: (() => void)[] = [];
93+
const pause = () =>
94+
moving ? new Promise<void>((resolve) => waiters.push(resolve)) : Promise.resolve();
95+
const analysis = analyze(audio, { pause });
96+
await vi.waitFor(() => expect(importStarted).toBe(true));
97+
moving = true;
98+
releaseImport();
99+
await vi.waitFor(() => expect(waiters).toHaveLength(1));
100+
expect(detect).not.toHaveBeenCalled();
101+
moving = false;
102+
waiters.shift()!();
103+
await analysis;
104+
expect(movingAtDetect).toEqual([false]);
105+
} finally {
106+
vi.doUnmock("bpm-detective");
107+
vi.resetModules();
108+
}
109+
});
110+
});

‎packages/core/src/beats/beatDetection.ts‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -212,18 +212,27 @@ function gateBeatsBySilence(
212212
return { times, strengths, peak };
213213
}
214214

215+
export interface MusicAnalysisOptions {
216+
pause?: () => Promise<void>;
217+
}
218+
215219
// fallow-ignore-next-line complexity
216-
export async function analyzeMusicFromBuffer(audioBuffer: AudioBuffer): Promise<MusicBeatAnalysis> {
220+
export async function analyzeMusicFromBuffer(
221+
audioBuffer: AudioBuffer,
222+
{ pause }: MusicAnalysisOptions = {},
223+
): Promise<MusicBeatAnalysis> {
217224
const channelData = audioBuffer.getChannelData(0);
218225
const sampleRate = audioBuffer.sampleRate;
219226
const duration = audioBuffer.duration;
220227

228+
await pause?.();
221229
const rawBeats = await detectBeats(audioBuffer);
222230
const onsetBpm = computeBpmFromBeats(rawBeats);
223231

224232
let detectiveBpm: number | null = null;
233+
const detect = await loadBpmDetective();
234+
await pause?.();
225235
try {
226-
const detect = await loadBpmDetective();
227236
if (detect) detectiveBpm = detect(audioBuffer);
228237
} catch {
229238
// Not enough peaks or browser context unavailable
@@ -261,6 +270,7 @@ export async function analyzeMusicFromBuffer(audioBuffer: AudioBuffer): Promise<
261270
regularizeBpm = detectiveBpm;
262271
}
263272

273+
await pause?.();
264274
const gridBeats =
265275
regularizeBpm !== null ? regularizeBeats(rawBeats, regularizeBpm, duration) : rawBeats;
266276
const gated = gateBeatsBySilence(gridBeats, channelData, sampleRate);
@@ -288,13 +298,16 @@ export async function detectBeatsFromUrl(url: string): Promise<number[]> {
288298
}
289299
}
290300

291-
export async function analyzeMusicFromUrl(url: string): Promise<MusicBeatAnalysis> {
301+
export async function analyzeMusicFromUrl(
302+
url: string,
303+
options: MusicAnalysisOptions = {},
304+
): Promise<MusicBeatAnalysis> {
292305
const audioContext = new AudioContext();
293306
try {
294307
const response = await fetch(url);
295308
const arrayBuffer = await response.arrayBuffer();
296309
const audioBuffer = await audioContext.decodeAudioData(arrayBuffer);
297-
return analyzeMusicFromBuffer(audioBuffer);
310+
return analyzeMusicFromBuffer(audioBuffer, options);
298311
} finally {
299312
await audioContext.close();
300313
}

‎packages/studio/src/hooks/useMusicBeatAnalysis.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ import { useEffect, useMemo, useRef } from "react";
22
import { usePlayerStore } from "../player/store/playerStore";
33
import { resolveBeatSourceTrack } from "../utils/timelineInspector";
44
import { analyzeMusicFromUrl } from "@hyperframes/core/beats";
5+
import { whenTimelineIdle } from "../player/components/timelineMotion";
56
import { useFileManagerContextOptional } from "../contexts/FileManagerContext";
67
import { mergeUserBeats } from "../utils/beatEditing";
78
import { getTimelineElementIndexes } from "../player/lib/timelineElementIndexes";
@@ -79,7 +80,7 @@ async function loadBeatAnalysis(
7980
): Promise<{ analysis: MusicAnalysis; times: number[]; strengths: number[] } | null> {
8081
let promise = analysisCache.get(musicSrc);
8182
if (!promise) {
82-
promise = analyzeMusicFromUrl(musicSrc);
83+
promise = analyzeMusicFromUrl(musicSrc, { pause: whenTimelineIdle });
8384
cacheAnalysis(musicSrc, promise);
8485
}
8586
try {

‎packages/studio/src/player/components/AudioWaveform.test.tsx‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,29 @@ vi.mock("../../hooks/useThumbnailLease", () => ({
1111
useThumbnailLease: leaseSpy,
1212
}));
1313

14+
// One observer serves every waveform; it reports each one near as it is watched, unless a test
15+
// reports by hand.
16+
const nearScreen = {
17+
auto: true,
18+
watched: [] as Element[],
19+
report: (_target: Element, _near: boolean) => {},
20+
};
21+
globalThis.IntersectionObserver = class {
22+
constructor(callback: IntersectionObserverCallback) {
23+
nearScreen.report = (target, near) =>
24+
callback(
25+
[{ target, isIntersecting: near } as IntersectionObserverEntry],
26+
this as unknown as IntersectionObserver,
27+
);
28+
}
29+
observe(target: Element) {
30+
nearScreen.watched.push(target);
31+
if (nearScreen.auto) nearScreen.report(target, true);
32+
}
33+
unobserve() {}
34+
disconnect() {}
35+
} as unknown as typeof IntersectionObserver;
36+
1437
const EMPTY_STRIP = { width: 0, height: 0, inViewStart: 0, inViewEnd: 0 };
1538
const watchGap = vi.hoisted(() => vi.fn());
1639
const strip = vi.hoisted(() => ({
@@ -52,6 +75,7 @@ afterEach(() => {
5275
leaseSpy.mockReset();
5376
leaseSpy.mockImplementation(() => ({ status: "loading" as const }));
5477
strip.size = EMPTY_STRIP;
78+
nearScreen.auto = true;
5579
watchGap.mockClear();
5680
document.body.innerHTML = "";
5781
});
@@ -141,6 +165,23 @@ describe("AudioWaveform", () => {
141165
expect(watchGap.mock.calls.filter(([gap]) => gap)).toEqual([]);
142166
});
143167

168+
it("leaves a waveform off screen undrawn through a zoom, and draws it once it comes near", () => {
169+
nearScreen.auto = false;
170+
const getContext = vi.spyOn(HTMLCanvasElement.prototype, "getContext").mockReturnValue(null);
171+
setStrip({ width: 1000, height: 40, inViewStart: 0, inViewEnd: 1000 });
172+
renderReadyWaveform();
173+
const root = nearScreen.watched.at(-1)!;
174+
expect(getContext).not.toHaveBeenCalled();
175+
act(() => nearScreen.report(root, true));
176+
const drawn = getContext.mock.calls.length;
177+
expect(drawn).toBeGreaterThan(0);
178+
act(() => nearScreen.report(root, false));
179+
setStrip({ width: 2000, height: 40, inViewStart: 0, inViewEnd: 2000 });
180+
expect(getContext.mock.calls.length).toBe(drawn);
181+
act(() => nearScreen.report(root, true));
182+
expect(getContext.mock.calls.length).toBe(drawn + 1);
183+
});
184+
144185
it("redraws a short clip, drawn whole, when a zoom changes its width", () => {
145186
const getContext = vi.spyOn(HTMLCanvasElement.prototype, "getContext").mockReturnValue(null);
146187
setStrip({ width: 1000, height: 40, inViewStart: 0, inViewEnd: 1000 });

‎packages/studio/src/player/components/AudioWaveform.tsx‎

Lines changed: 35 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import { memo, useCallback, useContext, useEffect, useMemo, useRef } from "react";
1+
import { memo, useCallback, useContext, useEffect, useMemo, useRef, useState } from "react";
22
import { fadeGain } from "@hyperframes/core/audio-fade";
33
import { useThumbnailLease } from "../../hooks/useThumbnailLease";
44
import { useThumbnailStripSize } from "../../hooks/useThumbnailStripSize";
@@ -158,6 +158,32 @@ async function decodeWaveformPeaks(url: string, signal: AbortSignal): Promise<nu
158158
}
159159

160160
/** Bounded waveform subscriber; cache, cancellation and dedupe live in one scheduler. */
161+
const NEAR_SCREEN: IntersectionObserverInit & { scrollMargin: string } = {
162+
rootMargin: "50% 100%",
163+
scrollMargin: "50% 100%",
164+
};
165+
const nearScreenListeners = new Map<Element, (near: boolean) => void>();
166+
let nearScreen: IntersectionObserver | null = null;
167+
168+
function useNearScreen(element: Element | null): boolean {
169+
const [near, setNear] = useState(() => typeof IntersectionObserver === "undefined");
170+
useEffect(() => {
171+
if (!element || typeof IntersectionObserver === "undefined") return;
172+
nearScreen ??= new IntersectionObserver(
173+
(entries) =>
174+
entries.forEach((entry) => nearScreenListeners.get(entry.target)?.(entry.isIntersecting)),
175+
NEAR_SCREEN,
176+
);
177+
nearScreenListeners.set(element, setNear);
178+
nearScreen.observe(element);
179+
return () => {
180+
nearScreenListeners.delete(element);
181+
nearScreen?.unobserve(element);
182+
};
183+
}, [element]);
184+
return near;
185+
}
186+
161187
export const AudioWaveform = memo(function AudioWaveform({
162188
audioUrl,
163189
waveformUrl,
@@ -171,7 +197,7 @@ export const AudioWaveform = memo(function AudioWaveform({
171197
muted = false,
172198
labelInset = 16,
173199
}: AudioWaveformProps) {
174-
const rootRef = useRef<HTMLDivElement | null>(null);
200+
const [root, setRoot] = useState<HTMLDivElement | null>(null);
175201
const canvasRef = useRef<HTMLCanvasElement | null>(null);
176202
const cacheKey = waveformUrl ?? audioUrl;
177203
const request = useMemo(
@@ -218,7 +244,10 @@ export const AudioWaveform = memo(function AudioWaveform({
218244
span,
219245
);
220246
}, [fades, from, muted, peaks, to, trimEndFraction, trimStartFraction]);
221-
useEffect(draw, [draw, strip.width, strip.height]);
247+
const near = useNearScreen(root);
248+
useEffect(() => {
249+
if (near) draw();
250+
}, [draw, near, strip.width, strip.height]);
222251

223252
useEffect(() => {
224253
const root = document.documentElement;
@@ -231,15 +260,15 @@ export const AudioWaveform = memo(function AudioWaveform({
231260
}, [draw]);
232261

233262
useEffect(() => {
234-
const clip = rootRef.current?.closest(".timeline-clip");
263+
const clip = root?.closest(".timeline-clip");
235264
if (!(clip instanceof HTMLElement)) return;
236265
if (muted) clip.setAttribute("data-audio-muted", "true");
237266
else clip.removeAttribute("data-audio-muted");
238267
return () => clip.removeAttribute("data-audio-muted");
239-
}, [muted]);
268+
}, [muted, root]);
240269

241270
return (
242-
<div ref={rootRef} className="absolute inset-0">
271+
<div ref={setRoot} className="absolute inset-0">
243272
<div
244273
ref={setStripRef}
245274
className="absolute inset-0 overflow-hidden"

0 commit comments

Comments
 (0)