Skip to content

Commit 2ab7b86

Browse files
feat(engine): adaptive per-frame sample count for motion blur (#4031)
* feat(engine): adaptive per-frame sample count for motion blur Measured against the accuracy bar set in #4010 (72.4dB mean PSNR vs the 960fps shutter integral), the fixed default of 16 samples misses it for hard-edge content while paying nothing extra for static/slow content is still important. This picks a sample count per frame from measured motion, floor 16 ceiling 64 (issue #4029). Two probe captures at the true shutter-window edges (never a sample that enters the average) measure motion magnitude, mapped through a monotone step table to a sample count. Frames confirmed to move only non-spatial properties (opacity, colour) skip the probe and use the floor directly, since a fade's shutter integral is a smooth blend that does not need many samples. MotionBlurPlan's window (from shutter angle/phase alone) is now independent of how many samples fill it, which is what lets the count vary per frame without moving the window; motionBlurWindowIsStatic is bounded by the widest sample inset any chosen K could produce so the static-dedup safety guarantee holds for every K in [16, 64]. * refactor(engine): simplify the adaptive motion-blur sample selection Two findings from an independent code-reuse/quality/efficiency review of the previous commit: - armStaticDedup ran the GSAP-timeline walk (computeStaticFrameSet) twice on the same page when adaptive motion blur and static dedup both apply, once for each purpose. Shares the one result instead. - The non-spatial floor path read the sample count from adaptiveSampleCount(0), which only worked because the step table happens to be sorted with the first bucket covering zero. Reads the now-exported DEFAULT_SAMPLES_PER_FRAME directly instead. * fix(engine): read raw options for the adaptive motion-blur gate Independent review caught it: every armStaticDedup call site runs before finalizeSessionInit resolves session.motionBlur, so the adaptive-classification gate's `session.motionBlur && session.motionBlur.fixedSamplesPerFrame === null` was always false at runtime. The non-spatial probe-skip described in the PR never engaged outside tests that hand-set motionBlurNonSpatialFrames directly, bypassing real init order. Reads session.options.motionBlur (set unconditionally at session creation) instead, which carries the same "adaptive requested" information the resolved plan would have. Adds a source-order lock test plus direct coverage of computeStaticFrameSet's nonSpatialOnlyFrameSet, mutation-proved against the exact regression. * fix(engine): add xPercent/yPercent to the spatial-property list Round 2 of the independent review: the spatial-property list deciding whether a frame's motion needs the adaptive probe omitted GSAP's percentage-based position properties (xPercent/yPercent) and two 3D transform properties (perspective/transformPerspective) that two other lists already in this repo (packages/parsers/src/gsapConstants.ts, packages/studio/.../gsapAnimatesProperty.ts) both carry. A composition positioned via xPercent/yPercent would have been silently floored to K=16 with no probe and no log line, the same failure shape as the round-1 bug through a data gap instead of a dead gate. Moves the list to an exported SPATIAL_TWEEN_PROPERTIES array in motionBlur.ts, passed into computeStaticFrameSet's page.evaluate as data rather than duplicated inline, so it stays a single list and is directly testable. Adds tests that execute the real page-side closure (the existing tests all mocked the whole result and never exercised the matching code), mutation-proved against the exact gap found. * fix(engine): add rotate to the spatial-property list Round 3 of the independent review, comparing SPATIAL_TWEEN_PROPERTIES side by side against the two sibling lists rather than trusting the prior commit's summary: GSAP's individual "rotate" transform property (distinct from "rotation", carried in packages/studio/.../gsapAnimatesProperty.ts's GSAP_TRANSFORM_PROPS) was still missing, same failure shape as the two prior rounds. The same pass also raised transformOrigin, which is deliberately NOT added: gsapConstants.ts's own classifyTweenPropertyGroup excludes it as a pivot-point modifier with no independent visual effect, and a test now locks that reasoning in rather than leaving it to be "fixed" again by a future pass that does not read that far.
1 parent 484a06f commit 2ab7b86

4 files changed

Lines changed: 611 additions & 83 deletions

File tree

‎packages/engine/src/services/frameCapture-motionBlur.test.ts‎

Lines changed: 244 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";
22
import {
33
captureFrame,
44
captureFrameToBuffer,
5+
computeStaticFrameSet,
56
resolveSessionMotionBlur,
67
type CaptureSession,
78
} from "./frameCapture.js";
@@ -73,7 +74,10 @@ function makeSession(overrides: Partial<CaptureSession> = {}): CaptureSession {
7374
totalMs: 0,
7475
frameMs: [],
7576
},
76-
motionBlur: resolveMotionBlurPlan({}) ?? undefined,
77+
// Fixed K=16 by default so the tests in this file that predate adaptive sampling
78+
// (accumulation, event suppression, dedup composition) keep testing exactly what
79+
// they did before — adaptive selection gets its own describe block below.
80+
motionBlur: resolveMotionBlurPlan({ samplesPerFrame: 16 }) ?? undefined,
7781
...overrides,
7882
} as unknown as CaptureSession;
7983
}
@@ -174,21 +178,30 @@ describe("sub-frame accumulation reaches the page with distinct sample times", (
174178
describe("accumulation composes with static-frame dedup", () => {
175179
// 720 degrees is the widest shutter After Effects offers, and the AD5 reference export
176180
// uses it, so it is the case a reader is most likely to check the frame ranges against.
177-
const wideShutter = resolveMotionBlurPlan({ shutterAngle: 720, shutterPhase: -360 }) ?? undefined;
178-
179-
it("pays nothing on a static frame instead of capturing K samples of the same instant", async () => {
181+
// Fixed K=16, same reason as makeSession's default above.
182+
const wideShutter =
183+
resolveMotionBlurPlan({ shutterAngle: 720, shutterPhase: -360, samplesPerFrame: 16 }) ??
184+
undefined;
185+
186+
/** Reuse claims frame 11's blurred output equals frame 10's, so the whole window
187+
* either side of it must be static for that claim to be true. */
188+
async function expectFrame11Reused(overrides: Partial<CaptureSession>) {
180189
const anchor = solidPng(42);
181190
const session = makeSession({
182-
// The whole window either side of frame 11 is static, which is what reuse claims.
183191
staticFrames: new Set([9, 10, 11, 12]),
184192
lastFrameBuffer: anchor,
185193
lastFrameAbsoluteIndex: 10,
194+
...overrides,
186195
});
187-
188196
const result = await captureFrameToBuffer(session, 11, 11 / 30);
189-
190197
expect(result.buffer).toBe(anchor);
191198
expect(vi.mocked(pageScreenshotCapture)).not.toHaveBeenCalled();
199+
return session;
200+
}
201+
202+
it("pays nothing on a static frame instead of capturing K samples of the same instant", async () => {
203+
const session = await expectFrame11Reused({});
204+
192205
expect(seeks).toHaveLength(0);
193206
expect(session.staticDedupCount).toBe(1);
194207
});
@@ -225,18 +238,7 @@ describe("accumulation composes with static-frame dedup", () => {
225238
});
226239

227240
it("still reuses at 720 degrees when the whole window is static", async () => {
228-
const anchor = solidPng(42);
229-
const session = makeSession({
230-
motionBlur: wideShutter,
231-
staticFrames: new Set([9, 10, 11, 12]),
232-
lastFrameBuffer: anchor,
233-
lastFrameAbsoluteIndex: 10,
234-
});
235-
236-
const result = await captureFrameToBuffer(session, 11, 11 / 30);
237-
238-
expect(result.buffer).toBe(anchor);
239-
expect(vi.mocked(pageScreenshotCapture)).not.toHaveBeenCalled();
241+
await expectFrame11Reused({ motionBlur: wideShutter });
240242
});
241243
});
242244

@@ -279,7 +281,10 @@ describe("resolveSessionMotionBlur rejects what accumulation cannot render", ()
279281
});
280282

281283
it("resolves a plan for a screenshot-mode PNG session", () => {
282-
expect(resolveSessionMotionBlur(withOptions({}))?.sampleTickOffsets).toHaveLength(16);
284+
const plan = resolveSessionMotionBlur(withOptions({}));
285+
expect(plan).not.toBeUndefined();
286+
// `options.motionBlur: {}` sets no explicit samplesPerFrame, so the plan is adaptive.
287+
expect(plan?.fixedSamplesPerFrame).toBeNull();
283288
});
284289

285290
it("rejects a capture mode whose sampling is not implemented yet", () => {
@@ -300,3 +305,222 @@ describe("resolveSessionMotionBlur rejects what accumulation cannot render", ()
300305
).toThrow(/video/);
301306
});
302307
});
308+
309+
describe("adaptive sample count (issue #4029)", () => {
310+
// No explicit samplesPerFrame → adaptive.
311+
const adaptivePlan = resolveMotionBlurPlan({}) ?? undefined;
312+
313+
it("probes the true window edges, escalates K on a large diff, and never lets a probe into the average", async () => {
314+
// First two screenshots are the probes (0, 255 — a large diff); everything after is
315+
// a constant 100, so if a probe leaked into the accumulator the average would move.
316+
let call = 0;
317+
vi.mocked(pageScreenshotCapture).mockImplementation(async () =>
318+
solidPng(call++ < 2 ? [0, 255][call - 1] : 100),
319+
);
320+
321+
const result = await captureFrameToBuffer(
322+
makeSession({ motionBlur: adaptivePlan }),
323+
10,
324+
10 / 30,
325+
);
326+
327+
// A 0/255 probe diff is well past the top step, so K=64.
328+
expect(vi.mocked(pageScreenshotCapture)).toHaveBeenCalledTimes(2 + 64);
329+
expect([...decodePng(result.buffer).data.slice(0, 4)]).toEqual([100, 100, 100, 255]);
330+
// seeks[0] is the one eventful seek; the two probe seeks come right after it, both
331+
// suppressed, and distinct from each other (the true window edges, not one point).
332+
const [probeA, probeB] = seeks.slice(1, 3);
333+
expect(probeA?.suppressEvents).toBe(true);
334+
expect(probeB?.suppressEvents).toBe(true);
335+
expect(probeA?.time).not.toBe(probeB?.time);
336+
});
337+
338+
it("skips the probe and uses the floor for a frame confirmed non-spatial", async () => {
339+
let calls = 0;
340+
vi.mocked(pageScreenshotCapture).mockImplementation(async () => {
341+
calls++;
342+
return solidPng(50);
343+
});
344+
345+
await captureFrameToBuffer(
346+
makeSession({ motionBlur: adaptivePlan, motionBlurNonSpatialFrames: new Set([10]) }),
347+
10,
348+
10 / 30,
349+
);
350+
351+
expect(calls).toBe(16); // floor only, no probe pair
352+
});
353+
354+
it("probes (rather than assuming the floor) for a frame outside the confirmed non-spatial set", async () => {
355+
let calls = 0;
356+
vi.mocked(pageScreenshotCapture).mockImplementation(async () => {
357+
calls++;
358+
return solidPng(50); // identical probes -> zero diff -> floor, but via the probe path
359+
});
360+
361+
await captureFrameToBuffer(
362+
makeSession({ motionBlur: adaptivePlan, motionBlurNonSpatialFrames: new Set([999]) }),
363+
10,
364+
10 / 30,
365+
);
366+
367+
expect(calls).toBe(2 + 16); // probed, diff was zero, landed on the floor anyway
368+
});
369+
});
370+
371+
describe("computeStaticFrameSet classifies spatial vs non-spatial intervals", () => {
372+
function makePage(intervals: Array<{ start: number; end: number; spatial: boolean }>) {
373+
return {
374+
evaluate: vi
375+
.fn()
376+
.mockResolvedValueOnce({
377+
intervals,
378+
tweenCount: intervals.length,
379+
duration: 10,
380+
hasVideo: false,
381+
hasCanvas: false,
382+
hasNonGsapAnim: false,
383+
hasUnresolvableClipStart: false,
384+
hasTimelineCall: false,
385+
})
386+
.mockResolvedValueOnce([]), // computeClipBoundaryFrames' own [data-start] scan
387+
} as unknown as Parameters<typeof computeStaticFrameSet>[0];
388+
}
389+
390+
it("marks a frame touched only by a non-spatial interval (e.g. opacity) as safe to skip the probe", async () => {
391+
const page = makePage([{ start: 0, end: 1, spatial: false }]);
392+
393+
const result = await computeStaticFrameSet(page, 30);
394+
395+
expect(result.eligible).toBe(true);
396+
expect(result.nonSpatialOnlyFrameSet.has(0)).toBe(true);
397+
});
398+
399+
it("does not mark a frame touched by any spatial interval (e.g. x), even alongside a non-spatial one", async () => {
400+
const page = makePage([
401+
{ start: 0, end: 1, spatial: false },
402+
{ start: 0, end: 1, spatial: true },
403+
]);
404+
405+
const result = await computeStaticFrameSet(page, 30);
406+
407+
expect(result.nonSpatialOnlyFrameSet.has(0)).toBe(false);
408+
});
409+
410+
it("reports an empty set when the analysis is ineligible, never a false 'confirmed non-spatial'", async () => {
411+
const page = {
412+
evaluate: vi
413+
.fn()
414+
.mockResolvedValueOnce({
415+
intervals: [{ start: 0, end: 1, spatial: false }],
416+
tweenCount: 0, // no GSAP tweens -> ineligible, per the existing "reasons" gate
417+
duration: 10,
418+
hasVideo: false,
419+
hasCanvas: false,
420+
hasNonGsapAnim: false,
421+
hasUnresolvableClipStart: false,
422+
hasTimelineCall: false,
423+
})
424+
.mockResolvedValueOnce([]), // computeClipBoundaryFrames' own [data-start] scan
425+
} as unknown as Parameters<typeof computeStaticFrameSet>[0];
426+
427+
const result = await computeStaticFrameSet(page, 30);
428+
429+
expect(result.eligible).toBe(false);
430+
expect(result.nonSpatialOnlyFrameSet.size).toBe(0);
431+
});
432+
});
433+
434+
describe("armStaticDedup reads the raw caller options for the adaptive gate, not the resolved plan", () => {
435+
// armStaticDedup always runs before finalizeSessionInit sets session.motionBlur, so a
436+
// gate reading that field is always false; it must read session.options.motionBlur.
437+
const source = readFileSync(new URL("./frameCapture.ts", import.meta.url), "utf8");
438+
439+
it("gates the adaptive classification block on session.options.motionBlur", () => {
440+
// Anchored on the `let` declaration, AFTER the explanatory comment above it, so this
441+
// reads only the functional `if (...)` condition — not prose that could name either
442+
// symbol without the code actually using it.
443+
const start = source.indexOf("let sharedStaticFrameStats");
444+
if (start < 0) throw new Error("sharedStaticFrameStats declaration not found");
445+
const conditionEnd = source.indexOf(") {", start);
446+
const condition = source.slice(start, conditionEnd);
447+
expect(condition).toContain("session.options.motionBlur");
448+
// Bare "session.motionBlur" (the resolved plan) not followed by a letter, so this
449+
// does not false-positive on the unrelated "session.motionBlurNonSpatialFrames".
450+
expect(condition).not.toMatch(/session\.motionBlur(?![A-Za-z])/);
451+
});
452+
453+
it("every armStaticDedup call site runs before finalizeSessionInit in the same init path", () => {
454+
// Confirms the premise: if a future refactor made these coincide, the raw-options
455+
// gate above would still be correct (session.options.motionBlur is set unconditionally
456+
// at session creation), but this pins today's actual ordering as documented context.
457+
const finalizeCalls = [...source.matchAll(/finalizeSessionInit\(session\)/g)].map(
458+
(m) => m.index as number,
459+
);
460+
const armCalls = [...source.matchAll(/armStaticDedup\(session,/g)].map(
461+
(m) => m.index as number,
462+
);
463+
expect(finalizeCalls.length).toBeGreaterThan(0);
464+
for (const finalizeAt of finalizeCalls) {
465+
const precedingArm = armCalls.filter((a) => a < finalizeAt).at(-1);
466+
expect(precedingArm).toBeDefined();
467+
}
468+
});
469+
});
470+
471+
describe("computeStaticFrameSet's real page-side property matching (not a canned mock)", () => {
472+
// Runs the ACTUAL closure passed to page.evaluate, against a stubbed window/document —
473+
// the other computeStaticFrameSet tests above mock the whole result and never exercise
474+
// isSpatial()/SPATIAL_PROPS at all, which is how xPercent/yPercent went unclassified.
475+
function makeTimeline(vars: Record<string, unknown>) {
476+
const child = { startTime: () => 0, duration: () => 1, totalDuration: () => 1, vars };
477+
return { getChildren: () => [child], duration: () => 1 };
478+
}
479+
480+
function makePage(timelines: Record<string, unknown>) {
481+
const root = globalThis as Record<string, unknown>;
482+
root.window = { __timelines: timelines, __hf: { duration: 1 } };
483+
root.document = {
484+
querySelector: () => null,
485+
querySelectorAll: () => [],
486+
getAnimations: undefined,
487+
};
488+
return {
489+
evaluate: async (fn: unknown, ...args: unknown[]) =>
490+
typeof fn === "function" ? (fn as (...a: unknown[]) => unknown)(...args) : undefined,
491+
} as unknown as Parameters<typeof computeStaticFrameSet>[0];
492+
}
493+
494+
afterEach(() => {
495+
const root = globalThis as Record<string, unknown>;
496+
delete root.window;
497+
delete root.document;
498+
});
499+
500+
it.each(["xPercent", "yPercent", "perspective", "transformPerspective", "rotate"])(
501+
"classifies a tween on %s as spatial",
502+
async (prop) => {
503+
const page = makePage({ main: makeTimeline({ [prop]: 50 }) });
504+
505+
const result = await computeStaticFrameSet(page, 30);
506+
507+
expect(result.nonSpatialOnlyFrameSet.has(0)).toBe(false);
508+
},
509+
);
510+
511+
it("does not classify transformOrigin alone as spatial (a pivot-point modifier, not independent motion)", async () => {
512+
const page = makePage({ main: makeTimeline({ transformOrigin: "center" }) });
513+
514+
const result = await computeStaticFrameSet(page, 30);
515+
516+
expect(result.nonSpatialOnlyFrameSet.has(0)).toBe(true);
517+
});
518+
519+
it("classifies a tween on opacity as non-spatial", async () => {
520+
const page = makePage({ main: makeTimeline({ opacity: 0.5 }) });
521+
522+
const result = await computeStaticFrameSet(page, 30);
523+
524+
expect(result.nonSpatialOnlyFrameSet.has(0)).toBe(true);
525+
});
526+
});

0 commit comments

Comments
 (0)