Skip to content

Commit 0bd2d73

Browse files
miguel-heygenmiga-heygen
authored andcommitted
feat(engine): sub-frame multi-sample motion blur, capture-level accumulation
Deliverable one of issue #4010: average K sub-frame captures into each output frame so rotation, scale and opacity blur the way After Effects does, which the translation-only catalog component (#4008) cannot do. Opt in per render with CaptureOptions.motionBlur; absent means the capture path is unchanged. The blocker was quantizeTimeToFrame flooring every seek onto the output frame grid, which collapsed all K samples onto one instant and made the averaged frame byte-identical to the unblurred one. quantizeSeekTime now owns that decision at all three quantize sites (runtime player, its no-timeline fallback, and the init transport) and refines the grid by an integer subdivision count, which can only contain the output grid, never move a frame-time instant off it. The finer grid rounds rather than floors: a sample time round-trips through the page bridge as a float, and past a few million ticks that error exceeds the floor epsilon. Accumulation sits inside captureFrameCore AFTER the static-dedup early return, so a static frame still costs one reused buffer and only moving frames pay K. It is not a before-capture hook, which would have disabled dedup outright. Callback invariant: one eventful seek per output frame at the frame time, then K suppressed sample seeks, then a suppressed restore to the frame time. The restore matters because a suppressed seek still moves the playhead, so without it the next frame's eventful seek would swallow the callbacks inside the shutter window. Scope of this deliverable: screenshot capture mode and PNG output. beginframe, drawelement and JPEG encoding are rejected with a named error rather than silently rendering unblurred. Video content is rejected for the same reason it is a documented limitation: injected frames are extracted per output frame and cannot follow a sub-frame seek. Blend defaults to sRGB, matching AE's non-linearised 8-bpc working space and the catalog component. Linear light is an explicit opt-in. Samples are folded into a running average as they are captured rather than held until the end, so peak memory is one accumulator plus one decoded sample instead of all K: at 1080p that is a constant 41 MB rather than 141 MB at 16 samples and 540 MB at the 64 maximum, on every frame of the render. PNG chunk CRC32 now has one owner in utils/crc32.ts, shared by the new encoder and ffprobe's metadata reader. The encoder previously imported zlib's crc32 by name, which throws at module evaluation on Node 22.0 and 22.1; ffprobe already carried the guarded version and the comment explaining why. The sRGB transfer function is likewise exported once and shared. Co-Authored-By: Miguel Ángel <[email protected]>
1 parent 2229d0d commit 0bd2d73

15 files changed

Lines changed: 1090 additions & 156 deletions

File tree

‎packages/core/src/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -269,6 +269,7 @@ export {
269269
export {
270270
MEDIA_VISUAL_STYLE_PROPERTIES,
271271
copyMediaVisualStyles,
272+
quantizeSeekTime,
272273
quantizeTimeToFrame,
273274
type MediaVisualStyleProperty,
274275
} from "./inline-scripts/parityContract";

‎packages/core/src/inline-scripts/parityContract.ts‎

Lines changed: 39 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -34,22 +34,52 @@ export type MediaVisualStyleProperty = (typeof MEDIA_VISUAL_STYLE_PROPERTIES)[nu
3434

3535
const FRAME_BOUNDARY_EPSILON = 1e-3;
3636

37+
function safeFps(fps: number): number {
38+
return Number.isFinite(fps) && fps > 0 ? fps : 30;
39+
}
40+
41+
function safeTime(timeSeconds: number): number {
42+
return Number.isFinite(timeSeconds) && timeSeconds > 0 ? timeSeconds : 0;
43+
}
44+
3745
export function quantizeTimeToFrame(timeSeconds: number, fps: number): number {
38-
const safeFps = Number.isFinite(fps) && fps > 0 ? fps : 30;
39-
const safeTime = Number.isFinite(timeSeconds) && timeSeconds > 0 ? timeSeconds : 0;
40-
const frameIndex = Math.floor(safeTime * safeFps + 1e-9);
41-
return frameIndex / safeFps;
46+
const grid = safeFps(fps);
47+
const frameIndex = Math.floor(safeTime(timeSeconds) * grid + 1e-9);
48+
return frameIndex / grid;
49+
}
50+
51+
/**
52+
* The grid a deterministic render seek lands on.
53+
*
54+
* With no `subFrameDivisions` this is the output frame grid, unchanged. Sub-frame
55+
* sampling (motion blur) passes an integer subdivision count, which can only refine
56+
* the output grid, never move a frame-time instant off it.
57+
*
58+
* The finer grid rounds where the frame grid floors. A sub-frame time makes a float
59+
* round trip (the caller builds `ticks / (fps * divisions)`, this recomputes
60+
* `time * fps * divisions`), and past a few million ticks that error exceeds the
61+
* floor epsilon, which would silently drop a sample one tick early.
62+
*/
63+
export function quantizeSeekTime(
64+
timeSeconds: number,
65+
fps: number,
66+
subFrameDivisions?: number,
67+
): number {
68+
const divisions = Number.isInteger(subFrameDivisions) ? (subFrameDivisions as number) : 1;
69+
if (divisions <= 1) return quantizeTimeToFrame(timeSeconds, fps);
70+
const grid = safeFps(fps) * divisions;
71+
return Math.round(safeTime(timeSeconds) * grid) / grid;
4272
}
4373

4474
/** Snap decimal noise near a frame boundary without moving genuinely fractional timing. */
4575
export function snapTimeToFrameBoundary(timeSeconds: number, fps: number): number {
46-
const safeFps = Number.isFinite(fps) && fps > 0 ? fps : 30;
47-
const safeTime = Number.isFinite(timeSeconds) && timeSeconds > 0 ? timeSeconds : 0;
48-
const framePosition = safeTime * safeFps;
76+
const grid = safeFps(fps);
77+
const time = safeTime(timeSeconds);
78+
const framePosition = time * grid;
4979
const nearestFrame = Math.round(framePosition);
5080
return Math.abs(framePosition - nearestFrame) <= FRAME_BOUNDARY_EPSILON
51-
? nearestFrame / safeFps
52-
: safeTime;
81+
? nearestFrame / grid
82+
: time;
5383
}
5484

5585
export function copyMediaVisualStyles(

‎packages/core/src/runtime/init.ts‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,11 @@ import {
5454
isMemberGroupHidden,
5555
} from "../audioGroups";
5656
import { clampNativeMediaVolume } from "../audioGain";
57-
import { quantizeTimeToFrame, snapTimeToFrameBoundary } from "../inline-scripts/parityContract";
57+
import {
58+
quantizeSeekTime,
59+
quantizeTimeToFrame,
60+
snapTimeToFrameBoundary,
61+
} from "../inline-scripts/parityContract";
5862
import { createManualEditGestureWatch } from "./manualEditGestureWatch";
5963
import type {
6064
RuntimeDeterministicAdapter,
@@ -3030,9 +3034,10 @@ export function initSandboxRuntimeModular(): void {
30303034
},
30313035
renderSeek: (timeSeconds, options) => {
30323036
renderCaptureSeekStarted = true;
3033-
const quantized = quantizeTimeToFrame(
3037+
const quantized = quantizeSeekTime(
30343038
Math.max(0, Number(timeSeconds) || 0),
30353039
state.canonicalFps,
3040+
options?.subFrameDivisions,
30363041
);
30373042
webAudio.stopAll();
30383043
clock.detachAudioSource();

‎packages/core/src/runtime/player.test.ts‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -652,3 +652,69 @@ describe("createRuntimePlayer", () => {
652652
});
653653
});
654654
});
655+
656+
describe("renderSeek sub-frame grid (issue #4010 motion blur)", () => {
657+
// The 16 sample times of one frame at the After Effects defaults (180 degree shutter,
658+
// phase -90): offsets -0.234375 .. +0.234375 frames on the engine's 4096-tick grid.
659+
const SUB_FRAME_DIVISIONS = 4096;
660+
const sampleTimes = (frameIndex: number, fps: number, samples: number): number[] => {
661+
const grid = fps * SUB_FRAME_DIVISIONS;
662+
return Array.from({ length: samples }, (_, k) => {
663+
const offsetFrames = -0.25 + ((k + 0.5) / samples) * 0.5;
664+
const ticks =
665+
frameIndex * SUB_FRAME_DIVISIONS + Math.round(offsetFrames * SUB_FRAME_DIVISIONS);
666+
return ticks / grid;
667+
});
668+
};
669+
670+
it("keeps 16 sub-frame sample times distinct instead of flooring them onto frame 10", () => {
671+
const timeline = createMockTimeline();
672+
const player = createRuntimePlayer(createMockDeps(timeline));
673+
674+
const landed = sampleTimes(10, 30, 16).map((t) => {
675+
player.renderSeek(t, { suppressEvents: true, subFrameDivisions: SUB_FRAME_DIVISIONS });
676+
return timeline.time();
677+
});
678+
679+
// Without the finer grid every sample floors onto frame 10 and the averaged frame is
680+
// byte-identical to the unblurred one, which reads as "the feature does nothing".
681+
expect(new Set(landed).size).toBe(16);
682+
expect(landed).toEqual([...landed].sort((a, b) => a - b));
683+
// The window straddles the frame instant: first sample before it, last after.
684+
expect(landed[0]).toBeLessThan(10 / 30);
685+
expect(landed[15]).toBeGreaterThan(10 / 30);
686+
});
687+
688+
it("recovers the intended tick exactly on a long composition", () => {
689+
const timeline = createMockTimeline();
690+
const player = createRuntimePlayer(createMockDeps(timeline));
691+
// Frame 18000 is 10 minutes in at 30fps, where the tick count is large enough that a
692+
// floor with a 1e-9 epsilon can land a tick early on the float round trip.
693+
const expected = sampleTimes(18000, 30, 16);
694+
695+
const landed = expected.map((t) => {
696+
player.renderSeek(t, { suppressEvents: true, subFrameDivisions: SUB_FRAME_DIVISIONS });
697+
return timeline.time();
698+
});
699+
700+
expect(landed).toEqual(expected);
701+
});
702+
703+
it("still floors onto the output frame grid when no subdivision is asked for", () => {
704+
const timeline = createMockTimeline();
705+
const player = createRuntimePlayer(createMockDeps(timeline));
706+
707+
player.renderSeek(10 / 30 + 0.017, {});
708+
709+
expect(timeline.time()).toBe(10 / 30);
710+
});
711+
712+
it("suppresses timeline events on a sub-frame sample seek", () => {
713+
const timeline = createMockTimeline();
714+
const player = createRuntimePlayer(createMockDeps(timeline));
715+
716+
player.renderSeek(10 / 30, { suppressEvents: true, subFrameDivisions: SUB_FRAME_DIVISIONS });
717+
718+
expect(timeline.totalTime).toHaveBeenCalledWith(10 / 30, true);
719+
});
720+
});

‎packages/core/src/runtime/player.ts‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import type { RuntimePlayer, RuntimeSeekOptions, RuntimeTimelineLike } from "./types";
2-
import { quantizeTimeToFrame } from "../inline-scripts/parityContract";
2+
import { quantizeSeekTime } from "../inline-scripts/parityContract";
33
import { swallow } from "./diagnostics";
44

55
/**
@@ -89,7 +89,7 @@ function seekTimelineDeterministically(
8989
canonicalFps: number,
9090
options?: RuntimeSeekOptions,
9191
): number {
92-
const quantized = quantizeTimeToFrame(timeSeconds, canonicalFps);
92+
const quantized = quantizeSeekTime(timeSeconds, canonicalFps, options?.subFrameDivisions);
9393
const suppressEvents = options?.suppressEvents === true;
9494
safeVoid(timeline, "pause");
9595
if (typeof timeline.totalTime === "function") {
@@ -248,7 +248,11 @@ export function createRuntimePlayer(deps: PlayerDeps): RuntimePlayer {
248248
activateSiblingTimelines(deps.getTimelineRegistry?.(), timeline);
249249
return seekTimelineDeterministically(timeline, timeSeconds, canonicalFps, options);
250250
})()
251-
: quantizeTimeToFrame(Math.max(0, Number(timeSeconds) || 0), canonicalFps);
251+
: quantizeSeekTime(
252+
Math.max(0, Number(timeSeconds) || 0),
253+
canonicalFps,
254+
options?.subFrameDivisions,
255+
);
252256
deps.onDeterministicSeek(quantized, options);
253257
deps.setIsPlaying(false);
254258
deps.onSyncMedia(quantized, false);

‎packages/core/src/runtime/types.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -257,6 +257,12 @@ export type RuntimePlayer = {
257257

258258
export type RuntimeSeekOptions = {
259259
suppressEvents?: boolean;
260+
/**
261+
* Subdivide the output frame grid this render seek quantizes onto. Integer >= 1;
262+
* 1 (or absent) is the output frame grid. Motion-blur sub-frame sampling passes the
263+
* engine's sub-frame tick count so a fractional sample time survives quantization.
264+
*/
265+
subFrameDivisions?: number;
260266
};
261267

262268
export type RuntimeTimelineChildLike = {

0 commit comments

Comments
 (0)