Skip to content

Commit 5a77dbc

Browse files
perf(core): the preview loads only the media near the playhead (#4682)
* perf(core): the preview loads only the media on screen or due next Studios preview bound every timed video and audio at preload=auto and called * perf(core): a far clip in the preview stays unloaded until it comes due A trimmed clip outside the window now waits at preload=none, reloaded so a fetch the parser started stops too; at metadata, 60 clips' header reads still queued on the origin under a slow link. An untrimmed clip keeps metadata, since its window is read from the element's duration. * fix(core): a far preview clip makes no request before it comes due Chromium keeps fetching a clip once it has started, whatever preload says after, so the reload at preload=none from the bind only restarted the fetch. In Studio's preview the runtime now sets preload=none on each trimmed clip after the first frame as it is parsed, and reloads only a clip the window armed when it leaves. * perf(core): a playing clip loads the next clip on its track At 3 MB/s a cold 1080p clip needed about 3.3 s to show a frame, longer than the 2 s look-ahead, so every clip on a one-track film was still empty when it came due. While the film plays, a clip that starts now arms the next clip on the same track (authored track index within one composition host, read as the timeline payload reads it), and the window pass keeps that clip armed. Paused or scrubbing, only the look-ahead applies. * chore(core): say in one line why the preview parses later clips at preload none * fix(core): the Studio timeline stops waiting on clips the preview left unloaded Studio's timeline readiness scans every media element for canplay, so each far clip the preview deferred held it until the 8 s timeout. In a document with the Studio preview mark, a clip the runtime has not set to buffer now counts only if it is on the first frame, as a lazy image already does. * fix(core): a clip leaving the preview window closes its fetch on a slow link Chromium keeps a fetch in flight when preload drops to none and load() runs, so on a 3 MB/s link each played clip kept its connection and the armed next clip queued behind them with no bytes. The release now drops src around the reload and puts the same attribute back, which ends the fetch without a new one. * refactor(core): name the authored-length check for what it reads * fix(player): a clip the preview stops preloading drops its parent-frame copy The player adopts each iframe clip whose preload turns auto into a parent-frame proxy that fetches the whole file, but never let go when the runtime set the clip back to none. On a 3 MB/s link those proxies and the thumbnail reads held all six connections, so the armed next clip got no bytes. A demoted clip now releases its proxy through the same path a removed clip uses. * chore(player): keep the proxy adoption note in one place * perf(studio): pause timeline thumbnails while playing preview media waits A playing preview clip that fires waiting now holds every thumbnail job and aborts the ones in flight until the transport stops; they re-run afterwards. An aborted job frees its slot at once, and a filmstrip decode closes its range requests on abort. * Revert "perf(studio): pause timeline thumbnails while playing preview media waits" This reverts commit 36926eb. * fix(core): preview keeps loop-start clips loaded and drops arms a jump leaves behind Studio's loop tick now tells the runtime where a wrap lands (window.__hf.setLoopStart), and while playing the preview keeps the clips due there loaded, so a wrap no longer starts cold. A next-on-track arm lasts only while the clip that set it plays on screen, so a jump away releases it. A clip with no authored length stays out of the window and loads as in a render, so no reload drops its duration. A clip is held until 4 s ahead after it is armed at 2 s, so a scrub or a pause at the edge does not refetch it. The parse-time watcher is installed only in the preview, once per page. One helper stops a clip's download, <source> children included, for released and detached media alike. * fix(core): a loop wrap also keeps the loop-in clip's next clip loaded The loop start's clips are held out to the 4 s release distance, as a clip being left is, so the wrap no longer reloads the clip after the loop-in one from nothing. * fix(core): the preview leaves clips on <source> children out of its release Putting a clip's <source> children back after the reload started a new download and moved them ahead of its other children, which a scene swap reads as an edit. Release now stops only a src clip's fetch; detached media is dropped as on main. Tests cover the loop hold's playing check, its time check, and the untrimmed guard on the next-clip arm. * style(core): drop a release comment the commit message already carries * fix(core): arm a nested clip timed in root seconds by its real start * fix(core): keep a clip loaded for a step back across its end in the preview
1 parent 6d00d94 commit 5a77dbc

15 files changed

Lines changed: 606 additions & 25 deletions

‎packages/core/src/compositionReadiness.test.ts‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import {
88
settleFirstFrameCompositionReadiness,
99
} from "./compositionReadiness.js";
1010
import { createRuntimeStartTimeResolver } from "./runtime/startResolver.js";
11+
import { STUDIO_PREVIEW_MARK_META } from "./studioPreviewMark.js";
1112
import { isRuntimeElementVisibleAt } from "./runtime/timeline.js";
1213

1314
function docWith(bodyHtml: string): Document {
@@ -179,6 +180,23 @@ describe("mediaReadinessInput", () => {
179180
await expect(pending).resolves.toBeUndefined();
180181
});
181182

183+
it("waits on a later clip in Studio's preview only once the runtime has set it to buffer", () => {
184+
const html =
185+
'<video id="deferred" data-start="30" data-duration="5" preload="none" src="a.mp4"></video>' +
186+
'<video id="untrimmed" data-start="30" preload="metadata" src="b.mp4"></video>' +
187+
'<video id="due" data-start="30" data-duration="5" preload="auto" src="c.mp4"></video>';
188+
const preview = docWith(html);
189+
preview.head
190+
.appendChild(preview.createElement("meta"))
191+
.setAttribute("name", STUDIO_PREVIEW_MARK_META);
192+
const plain = docWith(html);
193+
const ids = (doc: Document) =>
194+
scanPendingCompositionAssets(doc, { scope: "all" }).pendingMedia.map((el) => el.id);
195+
196+
expect(ids(preview)).toEqual(["due"]);
197+
expect(ids(plain)).toEqual(["deferred", "untrimmed", "due"]);
198+
});
199+
182200
it("waits on a later scene's image in the full scan unless it is lazy", () => {
183201
const doc = docWith(
184202
'<div data-start="30" data-duration="5"><img id="later" src="later.png">' +

‎packages/core/src/compositionReadiness.ts‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import { createRuntimeStartTimeResolver } from "./runtime/startResolver.js";
22
import { isRuntimeElementVisibleAt } from "./runtime/timeline.js";
33
import type { RuntimeTimelineLike } from "./runtime/types.js";
4+
import { STUDIO_PREVIEW_MARK_META } from "./studioPreviewMark.js";
45

56
/** A composition is "ready" once every declared input settles, not just once
67
* its duration is known. Each input returns null (nothing to wait on) or a
@@ -103,9 +104,18 @@ export function scanPendingCompositionAssets(
103104
timelineRegistry: runtimeWindow?.__timelines,
104105
includeAuthoredTimingAttrs: true,
105106
});
107+
// Studio's preview buffers a clip off the first frame only once it comes due, as with a lazy image.
108+
const defersMedia = doc.querySelector(`meta[name="${STUDIO_PREVIEW_MARK_META}"]`) !== null;
106109
const pendingMedia = Array.from(doc.querySelectorAll("video, audio"))
107110
.filter(isRealmHtmlMediaElement)
108-
.filter((el) => shouldIncludeAsset(el, scope, resolver, runtimeWindow?.__timelines ?? {}))
111+
.filter((el) =>
112+
shouldIncludeAsset(
113+
el,
114+
defersMedia && el.preload !== "auto" ? FIRST_FRAME_READINESS_SCOPE : scope,
115+
resolver,
116+
runtimeWindow?.__timelines ?? {},
117+
),
118+
)
109119
.filter((el) => el.readyState < HAVE_FUTURE_DATA);
110120
// A lazy image off the first frame loads only once shown, so no scope can wait on it.
111121
const pendingImages = Array.from(doc.querySelectorAll("img"))

‎packages/core/src/runtime/entry.test.ts‎

Lines changed: 325 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -303,6 +303,331 @@ describe("runtime entry", () => {
303303
expect(document.querySelectorAll("style[data-hf-first-pass-hide]")).toHaveLength(0);
304304
});
305305

306+
describe("media preload window", () => {
307+
const loads: HTMLMediaElement[] = [];
308+
const spyLoad = () => {
309+
loads.length = 0;
310+
vi.spyOn(HTMLMediaElement.prototype, "load").mockImplementation(
311+
function (this: HTMLMediaElement) {
312+
loads.push(this);
313+
},
314+
);
315+
};
316+
const loadsOf = (...els: HTMLMediaElement[]) =>
317+
els.map((el) => loads.filter((loaded) => loaded === el).length);
318+
const videos = (root: HTMLElement, ...starts: string[]) =>
319+
starts.map((start) => timed(root, "video", start));
320+
const armed = (...els: HTMLMediaElement[]) => els.map((el) => el.preload === "auto");
321+
afterEach(() => vi.restoreAllMocks());
322+
323+
it("loads a preview's media only on screen or due within the look-ahead, and frees what the playhead left", async () => {
324+
spyLoad();
325+
servePreview();
326+
const [now, soon, later] = videos(mountRoot(), "0", "1.5", "9");
327+
328+
await evaluateRuntime();
329+
expect(armed(now, soon, later)).toEqual([true, true, false]);
330+
expect(later.preload).toBe("none");
331+
expect(loadsOf(now, soon, later)).toEqual([1, 1, 0]);
332+
333+
now.setAttribute("src", "now.mp4");
334+
const srcWrites = new MutationObserver(() => {});
335+
srcWrites.observe(now, { attributeFilter: ["src"] });
336+
window.__player?.seek(7.6);
337+
expect(armed(now, soon, later)).toEqual([false, false, true]);
338+
// The two that left are reloaded empty with src dropped for the reload, ending any fetch in flight.
339+
expect([now.preload, soon.preload]).toEqual(["none", "none"]);
340+
expect(loadsOf(now, soon, later)).toEqual([2, 2, 1]);
341+
expect(srcWrites.takeRecords()).toHaveLength(2);
342+
expect(now.getAttribute("src")).toBe("now.mp4");
343+
});
344+
345+
it("loads a clip with no authored length as a render does, so no reload drops its duration", async () => {
346+
spyLoad();
347+
servePreview();
348+
const [untrimmed, later] = videos(mountRoot(), "0", "8");
349+
untrimmed.removeAttribute("data-duration");
350+
later.removeAttribute("data-duration");
351+
Object.defineProperty(untrimmed, "readyState", { value: 1 });
352+
Object.defineProperty(untrimmed, "duration", { value: 2 });
353+
354+
await evaluateRuntime();
355+
window.__player?.seek(3.6);
356+
window.__player?.seek(0.5);
357+
window.__player?.seek(7.5);
358+
expect(armed(untrimmed, later)).toEqual([true, true]);
359+
expect(loadsOf(untrimmed, later)).toEqual([1, 1]);
360+
});
361+
362+
it("arms a clip a far jump lands on without reloading it under the seek", async () => {
363+
spyLoad();
364+
servePreview();
365+
const [now, later] = videos(mountRoot(), "0", "8");
366+
367+
await evaluateRuntime();
368+
const beforeJump = loadsOf(later)[0];
369+
window.__player?.seek(8.4);
370+
expect(armed(now, later)).toEqual([false, true]);
371+
expect(loadsOf(later)).toEqual([beforeJump]);
372+
});
373+
374+
// jsdom's media elements cannot play; the runtime only needs play() to settle.
375+
const stubPlayback = () => {
376+
vi.spyOn(HTMLMediaElement.prototype, "play").mockResolvedValue(undefined);
377+
vi.spyOn(HTMLMediaElement.prototype, "pause").mockImplementation(() => {});
378+
};
379+
const tracks = (root: HTMLElement) => {
380+
stubPlayback();
381+
const [now, next, afterNext, otherTrack] = videos(root, "0", "4", "7", "3");
382+
otherTrack.setAttribute("data-track-index", "2");
383+
return { now, next, afterNext, otherTrack };
384+
};
385+
386+
it("arms the next clip on a clip's track when it starts playing, and nothing else", async () => {
387+
spyLoad();
388+
servePreview();
389+
const { next, afterNext, otherTrack } = tracks(mountRoot());
390+
391+
await evaluateRuntime();
392+
window.__player?.play();
393+
expect(armed(next, afterNext, otherTrack)).toEqual([true, false, false]);
394+
expect(loadsOf(next, afterNext, otherTrack)).toEqual([1, 0, 0]);
395+
});
396+
397+
it("arms nothing ahead of a paused playhead, and keeps what it armed through a pause", async () => {
398+
spyLoad();
399+
servePreview();
400+
const { next, afterNext } = tracks(mountRoot());
401+
402+
await evaluateRuntime();
403+
window.__player?.seek(0.5);
404+
expect(armed(next)).toEqual([false]);
405+
window.__player?.play();
406+
window.__player?.pause();
407+
expect(armed(next, afterNext)).toEqual([true, false]);
408+
expect(loadsOf(next)).toEqual([1]);
409+
});
410+
411+
it("holds a clip past the look-ahead edge a scrub crosses, and frees it once well clear", async () => {
412+
spyLoad();
413+
servePreview();
414+
const [, later] = videos(mountRoot(), "0", "5");
415+
416+
await evaluateRuntime();
417+
window.__player?.seek(3.5);
418+
window.__player?.seek(2.5);
419+
window.__player?.seek(3.5);
420+
expect(armed(later)).toEqual([true]);
421+
expect(loadsOf(later)).toEqual([1]);
422+
window.__player?.seek(0.5);
423+
expect(armed(later)).toEqual([false]);
424+
});
425+
426+
it("keeps the clip a step back across a cut returns to loaded", async () => {
427+
spyLoad();
428+
servePreview();
429+
const [outgoing, incoming] = videos(mountRoot(), "0", "2");
430+
431+
await evaluateRuntime();
432+
window.__player?.seek(2.1);
433+
expect(armed(outgoing, incoming)).toEqual([true, true]);
434+
window.__player?.seek(1.9);
435+
expect(armed(outgoing)).toEqual([true]);
436+
expect(loadsOf(outgoing)).toEqual([1]);
437+
});
438+
439+
it("arms a clip behind a playhead shuttling back before the playhead reaches it", async () => {
440+
spyLoad();
441+
servePreview();
442+
const [earlier, current] = videos(mountRoot(), "0", "2");
443+
current.setAttribute("data-duration", "6");
444+
445+
await evaluateRuntime();
446+
window.__player?.seek(6.5);
447+
expect(armed(earlier)).toEqual([false]);
448+
let armedAt: number | undefined;
449+
for (let tenths = 45; tenths >= 20 && armedAt === undefined; tenths--) {
450+
window.__player?.seek(tenths / 10);
451+
if (armed(earlier)[0]) armedAt = tenths / 10;
452+
}
453+
// Within the 2 s look-ahead of its end, as a clip ahead is armed within 2 s of its start.
454+
expect(armedAt).toBe(4);
455+
});
456+
457+
it("still frees a clip that ended well behind the playhead", async () => {
458+
spyLoad();
459+
servePreview();
460+
const [earlier, current] = videos(mountRoot(), "0", "2");
461+
current.setAttribute("data-duration", "6");
462+
463+
await evaluateRuntime();
464+
window.__player?.seek(2.1);
465+
window.__player?.seek(5.9);
466+
expect(armed(earlier)).toEqual([true]);
467+
expect(loadsOf(earlier)).toEqual([1]);
468+
window.__player?.seek(6.1);
469+
expect(armed(earlier)).toEqual([false]);
470+
expect(earlier.preload).toBe("none");
471+
expect(loadsOf(earlier)).toEqual([2]);
472+
});
473+
474+
it("drops an arm once the clip that set it leaves the screen, as a jump while playing does", async () => {
475+
spyLoad();
476+
servePreview();
477+
const { next } = tracks(mountRoot());
478+
window.__timelines = { main: pausedTimeline(20) };
479+
480+
await evaluateRuntime();
481+
window.__player?.play();
482+
window.__player?.seek(10.5, { keepPlaying: true });
483+
expect(armed(next)).toEqual([false]);
484+
});
485+
486+
it("stops holding a clip it armed once that clip has played", async () => {
487+
spyLoad();
488+
servePreview();
489+
const { next } = tracks(mountRoot());
490+
window.__timelines = { main: pausedTimeline(20) };
491+
492+
await evaluateRuntime();
493+
window.__player?.play();
494+
window.__player?.seek(4.5, { keepPlaying: true });
495+
window.__player?.seek(10.5, { keepPlaying: true });
496+
expect(armed(next)).toEqual([false]);
497+
});
498+
499+
it("arms a clip that starts exactly where the playing one ends", async () => {
500+
spyLoad();
501+
servePreview();
502+
stubPlayback();
503+
// 1.1 + 2.2 is 3.3000000000000003: back to back only within the boundary tolerance.
504+
const [first, second] = videos(mountRoot(), "1.1", "3.3");
505+
first.setAttribute("data-duration", "2.2");
506+
507+
await evaluateRuntime();
508+
window.__player?.seek(1.2);
509+
window.__player?.play();
510+
expect(armed(second)).toEqual([true]);
511+
});
512+
513+
it("arms the next clip only in the same host and on an authored track", async () => {
514+
spyLoad();
515+
servePreview();
516+
stubPlayback();
517+
const root = mountRoot();
518+
const [now] = videos(root, "0");
519+
const host = timed(root, "div", "0");
520+
host.setAttribute("data-composition-id", "inner");
521+
host.setAttribute("data-duration", "10");
522+
const [otherHost] = videos(host, "4");
523+
const [untracked, untrackedNext] = videos(root, "0", "4");
524+
for (const el of [untracked, untrackedNext]) el.removeAttribute("data-track-index");
525+
now.setAttribute("data-track-index", "1");
526+
527+
await evaluateRuntime();
528+
window.__player?.play();
529+
expect(armed(otherHost, untrackedNext)).toEqual([false, false]);
530+
});
531+
532+
it("arms a short nested clip whose start is already root time before it is due", async () => {
533+
spyLoad();
534+
servePreview();
535+
const host = timed(mountRoot(), "div", "5");
536+
host.setAttribute("data-composition-id", "inner");
537+
host.setAttribute("data-duration", "5");
538+
const [clip] = videos(host, "6");
539+
clip.setAttribute("data-hf-media-start-basis", "global");
540+
clip.setAttribute("data-duration", "0.4");
541+
542+
await evaluateRuntime();
543+
window.__player?.seek(4.5);
544+
expect(armed(clip)).toEqual([true]);
545+
});
546+
547+
it("keeps the clips due at the loop start loaded while the loop plays", async () => {
548+
spyLoad();
549+
servePreview();
550+
const { now, next, afterNext } = tracks(mountRoot());
551+
window.__timelines = { main: pausedTimeline(20) };
552+
553+
await evaluateRuntime();
554+
window.__hf?.setLoopStart?.(0);
555+
window.__player?.play();
556+
window.__player?.seek(19.5, { keepPlaying: true });
557+
expect(armed(now, next, afterNext)).toEqual([true, true, false]);
558+
expect(loadsOf(now, next)).toEqual([1, 1]);
559+
window.__hf?.setLoopStart?.(null);
560+
window.__player?.seek(19.6, { keepPlaying: true });
561+
expect(armed(now)).toEqual([false]);
562+
});
563+
564+
it("holds the loop start only while playing, and for a loop start that is a time", async () => {
565+
spyLoad();
566+
servePreview();
567+
const { now } = tracks(mountRoot());
568+
569+
await evaluateRuntime();
570+
window.__hf?.setLoopStart?.(0);
571+
window.__player?.play();
572+
window.__player?.seek(9.5, { keepPlaying: true });
573+
window.__player?.pause();
574+
expect(armed(now)).toEqual([false]);
575+
window.__hf?.setLoopStart?.(-1);
576+
window.__player?.play();
577+
expect(armed(now)).toEqual([false]);
578+
});
579+
580+
it("never arms a next clip with no authored length, so it is not reloaded", async () => {
581+
spyLoad();
582+
servePreview();
583+
const { next } = tracks(mountRoot());
584+
next.removeAttribute("data-duration");
585+
586+
await evaluateRuntime();
587+
const atBind = loadsOf(next)[0];
588+
window.__player?.play();
589+
expect(loadsOf(next)).toEqual([atBind]);
590+
});
591+
592+
it("keeps an armed next clip through a window pass that finds it outside the look-ahead", async () => {
593+
spyLoad();
594+
servePreview();
595+
const { next } = tracks(mountRoot());
596+
597+
await evaluateRuntime();
598+
window.__player?.play();
599+
window.__player?.seek(0.5, { keepPlaying: true });
600+
expect(armed(next)).toEqual([true]);
601+
expect(loadsOf(next)).toEqual([1]);
602+
});
603+
604+
it("gives each parsed clip after the first frame preload none before it can fetch", async () => {
605+
servePreview();
606+
const root = mountRoot();
607+
timed(root, "div", "0");
608+
await evaluateRuntime();
609+
const [first, later] = videos(root, "0", "5");
610+
const untrimmed = timed(root, "audio", "5");
611+
untrimmed.removeAttribute("data-duration");
612+
await Promise.resolve();
613+
expect([first.preload, later.preload, untrimmed.preload]).toEqual(["", "none", ""]);
614+
});
615+
616+
it("keeps a render loading every media element at bind", async () => {
617+
spyLoad();
618+
const [now, soon, later] = videos(mountRoot(), "0", "1.5", "5");
619+
620+
await evaluateRuntime();
621+
window.__player?.seek(3.6);
622+
expect(armed(now, soon, later)).toEqual([true, true, true]);
623+
expect(loadsOf(now, soon, later)).toEqual([1, 1, 1]);
624+
const parsedLater = timed(document.body, "video", "5");
625+
await Promise.resolve();
626+
expect(parsedLater.preload).not.toBe("none");
627+
expect((window as { __hfMediaDeferral?: unknown }).__hfMediaDeferral).toBeUndefined();
628+
});
629+
});
630+
306631
it("grades media inside a clip once the first pass shows the clip, with no seek", async () => {
307632
const getContext = vi.spyOn(HTMLCanvasElement.prototype, "getContext").mockReturnValue(null);
308633
const scene = timed(mountRoot(), "div", "0");

0 commit comments

Comments
 (0)