Skip to content

Commit b7157bd

Browse files
authored
feat(studio): scrub full-quality video with decoded preview frames (#7379)
* feat(studio): scrub full-quality video with decoded preview frames * fix(studio): release scrub decoders before native playback * fix(studio): bound paused fallback and tolerate decoder load failure * fix(studio): recover stalled native preview handoffs * fix(runtime): recognize cross-realm preview video elements * fix(studio): bound Play handoffs and drain transparent preview samples * fix(preview): recover retained video without chasing playback clock * fix(studio): drain finite alpha queries before disposing readers
1 parent b87a01f commit b7157bd

57 files changed

Lines changed: 4474 additions & 153 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎bun.lock‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/packages/studio.mdx‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,31 @@ depends on the browser playing in real time, while render captures frames one at
106106
time. A heavy project can therefore stutter during preview and still render every
107107
frame.
108108

109+
Paused video scrubbing decodes and stages the visible video layers before
110+
advancing the composition's animations. Completed frames appear during a drag,
111+
with newer pointer positions coalesced into the next decode. Canvas and GPU
112+
effects consume the same staged pixels. Continuous playback uses native video;
113+
Studio keeps the paused picture visible while those videos catch up before Play.
114+
A native handoff waits for ready frames for up to ten seconds; Pause or a new
115+
seek cancels it. At the deadline, the rest plays while each unready video keeps
116+
its last full-quality picture. During playback, native frame presentation or
117+
playback progress releases that picture; normal transport sync corrects drift.
118+
Paused recovery still waits for its requested frame. A skipped source keeps its
119+
picture on subsequent Play requests until it recovers.
120+
A paused fallback that outlasts its wait retains the presented clock and picture,
121+
while paused sync continues. It can still publish when the video becomes ready;
122+
a new gesture cancels that recovery. Studio requests paused-decoder release
123+
before preparing native playback and reopens readers when paused scrubbing resumes.
124+
Transparent video queries drain their alpha merges before input disposal. If one
125+
never settles, its cleanup can outlast the Play deadline; the remaining playback
126+
still starts while that video's last picture stays visible.
127+
Scrubbing and paused frame steps decode the original video at source resolution
128+
throughout. Automatic playback quality reductions do not reduce these frames,
129+
and the frame decoder does not substitute re-encoded seek copies. Playback
130+
quality settings do not reduce these paused frames. Browsers or
131+
media that cannot use the frame decoder fall back to native seeking. Long-GOP
132+
sources can take longer to seek, especially backward or across distant cuts.
133+
109134
## Tailwind preset
110135

111136
```typescript

‎packages/core/src/compiler/htmlDocument.test.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -469,6 +469,14 @@ describe("insertRuntimeTag", () => {
469469
expect(out.indexOf("window.author")).toBeLessThan(out.indexOf(runtime));
470470
});
471471

472+
it("keeps the tracker and runtime ahead of author scripts for Studio previews", () => {
473+
const out = insertRuntimeTag(page('data-motion-blur="on"'), runtime, true);
474+
const tracker = out.indexOf("data-hf-accel-tracker");
475+
expect(tracker).toBeGreaterThan(-1);
476+
expect(tracker).toBeLessThan(out.indexOf(runtime));
477+
expect(out.indexOf(runtime)).toBeLessThan(out.indexOf("window.author"));
478+
});
479+
472480
it("adds the tracker once, however many times the runtime is placed", () => {
473481
const once = insertRuntimeTag(page('data-motion-blur="on"'), runtime);
474482
const twice = insertRuntimeTag(once, runtime);

‎packages/core/src/compiler/htmlDocument.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -393,8 +393,8 @@ function insertTagInHead(html: string, tag: string): string {
393393
const ACCEL_TRACKER_ATTR = "data-hf-accel-tracker";
394394
const OPTS_INTO_MOTION_BLUR = /\sdata-motion-blur\s*=\s*["']?on[\s"'>]/i;
395395

396-
export function insertRuntimeTag(html: string, tag: string): string {
397-
const withRuntime = insertTagInHead(html, tag);
396+
export function insertRuntimeTag(html: string, tag: string, atHeadStart = false): string {
397+
const withRuntime = atHeadStart ? injectTagsAtHeadStart(html, tag) : insertTagInHead(html, tag);
398398
if (!OPTS_INTO_MOTION_BLUR.test(html) || html.includes(ACCEL_TRACKER_ATTR)) return withRuntime;
399399
const tracker = `<script ${ACCEL_TRACKER_ATTR}>${acceleratedCanvasTrackerSource()}</script>`;
400400
return injectTagsAtHeadStart(withRuntime, tracker);

‎packages/core/src/index.ts‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -308,6 +308,11 @@ export {
308308
type PreviewQuality,
309309
type PreviewSourceFacts,
310310
} from "./previewProxyBox";
311+
export type {
312+
PausedVideoFrameProvider,
313+
PausedVideoFrameRequest,
314+
PausedVideoFrame,
315+
} from "./runtime/pausedVideoFrames";
311316

312317
// Inline scripts
313318
export {

‎packages/core/src/previewProxyBox.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,8 @@ export type PreviewSourceFacts = {
5555
bitDepth: number;
5656
bitrate: number;
5757
codec?: string;
58+
frameRate?: number;
59+
variableFrameRate?: boolean;
5860
};
5961

6062
export const HEAVY_PIXEL_SHARE = 0.5;
Lines changed: 238 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,238 @@
1+
import { afterEach, expect, it, vi } from "vitest";
2+
import { createPausedVideoFrames, getPausedVideoFrame } from "../pausedVideoFrames";
3+
import {
4+
patchCanvasVideoFrameCompat,
5+
patchExternalVideoTextureCompat,
6+
patchVideoTextureCompat,
7+
patchWebGLVideoTextureCompat,
8+
} from "./video-texture-compat";
9+
10+
const sessions: ReturnType<typeof createPausedVideoFrames>[] = [];
11+
afterEach(() => {
12+
for (const session of sessions.splice(0)) session.destroy();
13+
document.body.innerHTML = "";
14+
vi.restoreAllMocks();
15+
vi.unstubAllGlobals();
16+
});
17+
18+
function surfaces() {
19+
const draw = vi.fn();
20+
const offscreenDraw = vi.fn();
21+
const image = vi.fn();
22+
const subImage = vi.fn();
23+
const copy = vi.fn();
24+
const imported = vi.fn((_descriptor: unknown) => ({ texture: true }));
25+
const constructed = vi.fn();
26+
class Context2D {
27+
drawImage(...args: unknown[]) {
28+
draw.apply(this, args);
29+
}
30+
}
31+
class Offscreen2D {
32+
drawImage(...args: unknown[]) {
33+
offscreenDraw.apply(this, args);
34+
}
35+
}
36+
class GL {
37+
texImage2D(...args: unknown[]) {
38+
image.apply(this, args);
39+
}
40+
texSubImage2D(...args: unknown[]) {
41+
subImage.apply(this, args);
42+
}
43+
}
44+
class GL2 extends GL {}
45+
class Queue {
46+
copyExternalImageToTexture(...args: unknown[]) {
47+
copy.apply(this, args);
48+
}
49+
}
50+
class Device {
51+
importExternalTexture(descriptor: unknown) {
52+
return imported.call(this, descriptor);
53+
}
54+
}
55+
class Frame {
56+
close = vi.fn();
57+
constructor(source: unknown, options: unknown) {
58+
constructed(source, options);
59+
}
60+
}
61+
vi.stubGlobal("CanvasRenderingContext2D", Context2D);
62+
vi.stubGlobal("OffscreenCanvasRenderingContext2D", Offscreen2D);
63+
vi.stubGlobal("WebGLRenderingContext", GL);
64+
vi.stubGlobal("WebGL2RenderingContext", GL2);
65+
vi.stubGlobal("GPUQueue", Queue);
66+
vi.stubGlobal("GPUDevice", Device);
67+
vi.stubGlobal("VideoFrame", Frame);
68+
vi.spyOn(HTMLCanvasElement.prototype, "getContext").mockReturnValue({
69+
clearRect: vi.fn(),
70+
drawImage: vi.fn(),
71+
} as unknown as CanvasRenderingContext2D);
72+
return {
73+
Context2D,
74+
Offscreen2D,
75+
GL,
76+
GL2,
77+
Queue,
78+
Device,
79+
draw,
80+
offscreenDraw,
81+
image,
82+
subImage,
83+
copy,
84+
imported,
85+
constructed,
86+
};
87+
}
88+
89+
async function presented() {
90+
const session = createPausedVideoFrames();
91+
sessions.push(session);
92+
const video = document.createElement("video");
93+
video.src = "/original.mp4";
94+
document.body.append(video);
95+
Object.defineProperties(video, {
96+
videoWidth: { value: 1920 },
97+
videoHeight: { value: 1080 },
98+
});
99+
const source = document.createElement("canvas");
100+
source.width = 320;
101+
source.height = 180;
102+
session.setProvider({
103+
decode: async (requests) =>
104+
requests.map(({ time }) => ({
105+
source,
106+
timestamp: time,
107+
duration: 1 / 30,
108+
width: 1920,
109+
height: 1080,
110+
release: vi.fn(),
111+
})),
112+
dispose: vi.fn(),
113+
});
114+
const show = async (time: number) => {
115+
const batch = await session.prepare(
116+
[{ video, source: video.src, time }],
117+
new AbortController().signal,
118+
);
119+
expect(batch?.present()).toBe(true);
120+
batch!.refresh();
121+
};
122+
await show(3);
123+
const canvas = getPausedVideoFrame(video)!;
124+
expect([canvas.width, canvas.height]).toEqual([video.videoWidth, video.videoHeight]);
125+
return { session, video, canvas, show };
126+
}
127+
128+
it.each([
129+
["three arguments", [11, 17]],
130+
["five arguments", [11, 17, 640, 360]],
131+
["nine arguments", [120, 60, 960, 540, 11, 17, 640, 360]],
132+
])("preserves %s and source coordinates for both canvas surfaces", async (_name, coordinates) => {
133+
const { Context2D, Offscreen2D, draw, offscreenDraw } = surfaces();
134+
const { session, video, canvas } = await presented();
135+
patchCanvasVideoFrameCompat();
136+
patchCanvasVideoFrameCompat();
137+
const context = new Context2D();
138+
const offscreen = new Offscreen2D();
139+
context.drawImage(video, ...coordinates);
140+
offscreen.drawImage(video, ...coordinates);
141+
expect(draw).toHaveBeenCalledExactlyOnceWith(canvas, ...coordinates);
142+
expect(offscreenDraw).toHaveBeenCalledExactlyOnceWith(canvas, ...coordinates);
143+
expect(draw.mock.contexts[0]).toBe(context);
144+
expect(offscreenDraw.mock.contexts[0]).toBe(offscreen);
145+
session.clear();
146+
context.drawImage(video, ...coordinates);
147+
expect(draw).toHaveBeenLastCalledWith(video, ...coordinates);
148+
const image = document.createElement("img");
149+
offscreen.drawImage(image, ...coordinates);
150+
expect(offscreenDraw).toHaveBeenLastCalledWith(image, ...coordinates);
151+
});
152+
153+
it("uploads paused pixels through both WebGL versions without changing crop or upload parameters", async () => {
154+
const { GL, GL2, image, subImage } = surfaces();
155+
const { session, video, canvas } = await presented();
156+
patchWebGLVideoTextureCompat();
157+
for (const Context of [GL, GL2]) {
158+
const gl = new Context();
159+
const imageArgs = [0x0de1, 2, 0x1908, 0x1908, 0x1401];
160+
const subArgs = [0x0de1, 2, 13, 29, 0x1908, 0x1401];
161+
gl.texImage2D(...imageArgs, video);
162+
gl.texSubImage2D(...subArgs, video);
163+
expect(image).toHaveBeenLastCalledWith(...imageArgs, canvas);
164+
expect(subImage).toHaveBeenLastCalledWith(...subArgs, canvas);
165+
expect(image.mock.contexts.at(-1)).toBe(gl);
166+
const pixels = new Uint8Array([10, 20, 30, 255]);
167+
gl.texImage2D(0x0de1, 0, 0x1908, 1, 1, 0, 0x1908, 0x1401, pixels);
168+
expect(image).toHaveBeenLastCalledWith(0x0de1, 0, 0x1908, 1, 1, 0, 0x1908, 0x1401, pixels);
169+
}
170+
session.clear();
171+
new GL().texImage2D(0x0de1, 0, 0x1908, 0x1908, 0x1401, video);
172+
expect(image.mock.lastCall?.at(-1)).toBe(video);
173+
});
174+
175+
it("preserves WebGPU crop, orientation, destination and full intrinsic copy extent", async () => {
176+
const { Queue, copy } = surfaces();
177+
const { session, video, canvas } = await presented();
178+
patchVideoTextureCompat();
179+
const queue = new Queue();
180+
const source = { source: video, origin: { x: 37, y: 41 }, flipY: true };
181+
const destination = { texture: {}, origin: [5, 7], mipLevel: 1, premultipliedAlpha: true };
182+
const extent = {
183+
width: video.videoWidth - 37,
184+
height: video.videoHeight - 41,
185+
depthOrArrayLayers: 1,
186+
};
187+
queue.copyExternalImageToTexture(source, destination, extent);
188+
expect(copy).toHaveBeenCalledExactlyOnceWith({ ...source, source: canvas }, destination, extent);
189+
expect(copy.mock.contexts[0]).toBe(queue);
190+
expect(source.source).toBe(video);
191+
expect(copy.mock.calls[0]?.[1]).toBe(destination);
192+
expect(copy.mock.calls[0]?.[2]).toBe(extent);
193+
session.clear();
194+
queue.copyExternalImageToTexture(source, destination, extent);
195+
expect(copy).toHaveBeenLastCalledWith(source, destination, extent);
196+
expect(copy.mock.lastCall?.[0]).toBe(source);
197+
});
198+
199+
it("reuses an external VideoFrame until presentation changes, then closes it on replacement and clear", async () => {
200+
const { Device, imported, constructed } = surfaces();
201+
const { session, video, canvas, show } = await presented();
202+
patchExternalVideoTextureCompat();
203+
const once = Device.prototype.importExternalTexture;
204+
patchExternalVideoTextureCompat();
205+
expect(Device.prototype.importExternalTexture).toBe(once);
206+
const device = new Device();
207+
const descriptor = { source: video, colorSpace: "srgb", label: "authored texture" };
208+
const texture = device.importExternalTexture(descriptor);
209+
const first = imported.mock.lastCall?.[0] as {
210+
source: { close: ReturnType<typeof vi.fn> };
211+
colorSpace: string;
212+
label: string;
213+
};
214+
expect(texture).toEqual({ texture: true });
215+
expect(constructed).toHaveBeenCalledExactlyOnceWith(canvas, { timestamp: 3_000_000 });
216+
expect(first).toEqual({ ...descriptor, source: first.source });
217+
expect(first.source).not.toBe(video);
218+
expect(imported.mock.contexts[0]).toBe(device);
219+
expect(descriptor.source).toBe(video);
220+
device.importExternalTexture(descriptor);
221+
expect(imported.mock.lastCall?.[0]).toEqual(first);
222+
expect(constructed).toHaveBeenCalledTimes(1);
223+
expect(first.source.close).not.toHaveBeenCalled();
224+
await show(4);
225+
expect(first.source.close).toHaveBeenCalledTimes(1);
226+
device.importExternalTexture(descriptor);
227+
const second = imported.mock.lastCall?.[0] as typeof first;
228+
expect(second.source).not.toBe(first.source);
229+
expect(constructed).toHaveBeenLastCalledWith(getPausedVideoFrame(video), {
230+
timestamp: 4_000_000,
231+
});
232+
session.clear();
233+
expect(second.source.close).toHaveBeenCalledTimes(1);
234+
device.importExternalTexture(descriptor);
235+
expect(imported).toHaveBeenLastCalledWith(descriptor);
236+
expect(imported.mock.lastCall?.[0]).toBe(descriptor);
237+
expect(constructed).toHaveBeenCalledTimes(2);
238+
});

0 commit comments

Comments
 (0)