Skip to content

Commit b64a7a3

Browse files
feat(producer): make sub-frame motion blur reachable from a render (#4020)
* feat(producer): make sub-frame motion blur reachable from a render Second deliverable for issue #4010. The engine gained the accumulation path, but nothing set CaptureOptions.motionBlur, so the feature could not be switched on from a real render. RenderConfig now carries it and the producer threads it into the capture options. Doing that surfaced the defect that would have made it fail silently. "Does this render capture PNG frames" was spelled four times, every one of them re-derived from needsAlpha, which describes the OUTPUT container rather than the capture: probeStage picked the probe capture format renderOrchestrator picked the render capture format encodeStage built the encoder's frame_%06d input pattern, twice captureStage built the disk worker's frame extension Motion blur is a second, independent reason to capture PNG: the sub-frame samples are averaged pixel by pixel, so JPEG samples would be averaged after a lossy quantization. Add that reason to only one of those four and the encoder goes looking for frame_%06d.jpg against files written as .png, and the render finds no frames at all rather than reporting an error. So the rule now has one owner, resolveCaptureImageFormat, which names both reasons. Everyone downstream reads the resolved capture format instead of re-deriving it: the encoder and the disk worker share frameFileExtension with writeCapturedFrame, which is what actually names the files. needsAlpha keeps only the question it really answers, whether the output preserves alpha. captureStage no longer needs needsAlpha at all, which is the tell that the re-derivation was its only reason to know. No JPEG encoder is involved. The PNG capture path is already load-bearing, since every transparent-output render uses it and the streaming encoder already selects png as its input codec. Co-Authored-By: Miguel Ángel <[email protected]> * fix(engine): resolve the motion-blur plan on every initialization exit A real render produced no blur at all. The plan was correct, the producer wiring was correct, and the sub-frame sample times were genuinely distinct, but K=16 and K=64 rendered identically to no blur because neither ever ran. initializeSession has two exits: screenshot mode finishes its own block and returns, every other capture mode falls through to the end. The plan was resolved at the fall-through exit only, and screenshot is the one capture mode motion blur supports, so on exactly the path the feature runs on session.motionBlur was never assigned. captureFrameCore then read it as undefined and took the single-capture branch: one capture per output frame, no accumulation, no error. finalizeSessionInit is now the single owner of what finishing initialization means. It sets the plan and isInitialized together and is called at both exits, so a third exit cannot mark a session ready while forgetting the plan. Why the existing tests passed, which is the part worth keeping: the accumulation test set session.motionBlur directly on a stub session. It exercised the mechanism while bypassing the only code that decides whether the mechanism runs at all. A test that proves something works and never asks whether it is reachable is not end to end, however much behaviour it drives. Two tests. A structural guard that session.isInitialized = true appears exactly once and that the same place resolves the plan, which fails if the defect is reintroduced at either exit. And a sample-distinctness assertion that hashes the K buffers captured within one output frame and requires K distinct ones, which covers the different failure of every sub-frame seek landing on one instant. Co-Authored-By: Miguel Ángel <[email protected]> --------- Co-authored-by: Miguel Ángel <[email protected]>
1 parent e4b1832 commit b64a7a3

11 files changed

Lines changed: 192 additions & 14 deletions

File tree

‎packages/engine/src/index.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -273,6 +273,8 @@ export {
273273

274274
// ── Utilities ──────────────────────────────────────────────────────────────────
275275
export { quantizeTimeToFrame, MEDIA_VISUAL_STYLE_PROPERTIES } from "@hyperframes/core";
276+
export { frameFileExtension } from "./services/frameCapture.js";
277+
export type { MotionBlurOptions, MotionBlurBlendSpace } from "./services/motionBlur.js";
276278

277279
export {
278280
assertSwiftShader,

‎packages/engine/src/services/frameCapture-motionBlur.test.ts‎

Lines changed: 39 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ import {
55
resolveSessionMotionBlur,
66
type CaptureSession,
77
} from "./frameCapture.js";
8-
import { mkdtempSync } from "fs";
8+
import { mkdtempSync, readFileSync } from "fs";
99
import { tmpdir } from "os";
1010
import { join } from "path";
1111
import { resolveMotionBlurPlan } from "./motionBlur.js";
@@ -129,6 +129,23 @@ describe("sub-frame accumulation reaches the page with distinct sample times", (
129129
expect(seeks).toHaveLength(18);
130130
});
131131

132+
it("captures K distinct samples rather than K copies of one instant", async () => {
133+
// Guards the failure mode where every sub-frame seek lands on the same time: the
134+
// average of K identical frames is the unblurred frame, which looks like the feature
135+
// doing nothing rather than like an error.
136+
const captured: string[] = [];
137+
vi.mocked(pageScreenshotCapture).mockImplementation(async () => {
138+
const png = solidPng(16 * captured.length);
139+
captured.push(png.toString("base64"));
140+
return png;
141+
});
142+
143+
await captureFrameToBuffer(makeSession(), 10, 10 / 30);
144+
145+
expect(captured).toHaveLength(16);
146+
expect(new Set(captured).size).toBe(16);
147+
});
148+
132149
it("averages the captured samples into the output frame", async () => {
133150
const result = await captureFrameToBuffer(makeSession(), 10, 10 / 30);
134151

@@ -184,6 +201,27 @@ describe("accumulation composes with static-frame dedup", () => {
184201
});
185202
});
186203

204+
describe("every path that marks a session ready also resolves the plan", () => {
205+
// The defect this locks: `initializeSession` has two exits, and screenshot mode (the only
206+
// mode motion blur supports) returns from the first one. Resolving the plan at the other
207+
// exit left `session.motionBlur` undefined on exactly the path the feature runs on, so a
208+
// real render captured one frame per output frame and produced no blur at all, while the
209+
// plan, the producer wiring and every unit test stayed correct.
210+
const source = readFileSync(new URL("./frameCapture.ts", import.meta.url), "utf8");
211+
212+
it("assigns isInitialized in exactly one place", () => {
213+
const assignments = source.match(/session\.isInitialized = true/g) ?? [];
214+
expect(assignments).toHaveLength(1);
215+
});
216+
217+
it("resolves the motion-blur plan in that same place", () => {
218+
const start = source.indexOf("function finalizeSessionInit");
219+
const finalize = source.slice(start, start + 400);
220+
expect(finalize).toContain("resolveSessionMotionBlur");
221+
expect(finalize).toContain("session.isInitialized = true");
222+
});
223+
});
224+
187225
describe("resolveSessionMotionBlur rejects what accumulation cannot render", () => {
188226
const withOptions = (overrides: Partial<CaptureSession>): CaptureSession =>
189227
makeSession({

‎packages/engine/src/services/frameCapture.ts‎

Lines changed: 27 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2311,7 +2311,7 @@ export async function initializeSession(session: CaptureSession): Promise<void>
23112311

23122312
await armStaticDedup(session, session.page, logInitPhase);
23132313
await ensureRenderFrameSiblings(session.page);
2314-
session.isInitialized = true;
2314+
finalizeSessionInit(session);
23152315
return;
23162316
}
23172317

@@ -2512,9 +2512,7 @@ export async function initializeSession(session: CaptureSession): Promise<void>
25122512
const commitCdp = await getCdpSession(page);
25132513
await commitCdp.send("HeadlessExperimental.beginFrame", preparedBeginFrameTimeline.commitParams);
25142514

2515-
session.motionBlur = resolveSessionMotionBlur(session);
2516-
2517-
session.isInitialized = true;
2515+
finalizeSessionInit(session);
25182516
}
25192517

25202518
async function captureFrameErrorDiagnostics(
@@ -3748,6 +3746,20 @@ async function captureFrameSurface(
37483746
return { buffer: screenshotBuffer, quantizedTime, seekMs, beforeCaptureMs, screenshotMs };
37493747
}
37503748

3749+
/**
3750+
* Mark the session ready to capture.
3751+
*
3752+
* The single owner of what finishing initialization means, because `initializeSession`
3753+
* has two exits: screenshot mode returns early, and every other mode falls through the
3754+
* end. Resolving motion blur at only one of them marks the session ready with no plan,
3755+
* which silently renders unblurred rather than failing. Both fields are set here so a
3756+
* third exit cannot forget one.
3757+
*/
3758+
function finalizeSessionInit(session: CaptureSession): void {
3759+
session.motionBlur = resolveSessionMotionBlur(session);
3760+
session.isInitialized = true;
3761+
}
3762+
37513763
/**
37523764
* Capture one output frame as the average of `plan.samplesPerFrame` sub-frame captures.
37533765
*
@@ -3881,6 +3893,16 @@ export async function captureFrame(
38813893
return { frameIndex, time: quantizedTime, path: framePath, captureTimeMs };
38823894
}
38833895

3896+
/**
3897+
* File extension for a captured frame, keyed on the format the frames were actually
3898+
* captured in. The encoder's input pattern and the writer must agree, so both read this
3899+
* rather than re-deriving the answer from whether the OUTPUT needs alpha, which is a
3900+
* different question and stops being equivalent as soon as anything else forces PNG.
3901+
*/
3902+
export function frameFileExtension(format: "jpeg" | "png" | undefined): "png" | "jpg" {
3903+
return format === "png" ? "png" : "jpg";
3904+
}
3905+
38843906
/**
38853907
* Write an already-captured frame buffer to the session's output dir using the
38863908
* canonical `frame_NNNNNN.{jpg,png}` naming. `fileIndex` is the ENCODER-facing
@@ -3894,7 +3916,7 @@ export function writeCapturedFrame(
38943916
fileIndex: number,
38953917
buffer: Buffer,
38963918
): string {
3897-
const ext = session.options.format === "png" ? "png" : "jpg";
3919+
const ext = frameFileExtension(session.options.format);
38983920
const framePath = join(session.outputDir, `frame_${String(fileIndex).padStart(6, "0")}.${ext}`);
38993921
writeFileSync(framePath, buffer);
39003922
return framePath;

‎packages/producer/src/services/distributed/renderChunk.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -957,6 +957,7 @@ export async function renderChunk(
957957
width: plan.dimensions.width * encoder.deviceScaleFactor,
958958
height: plan.dimensions.height * encoder.deviceScaleFactor,
959959
needsAlpha: plan.dimensions.format !== "mp4",
960+
captureImageFormat: captureOptions.format ?? "jpeg",
960961
// Each chunk produces video only — audio is muxed once at assemble
961962
// time. Suppressing `hasAudio` skips the png-sequence audio sidecar
962963
// AND the mp4 audio mux.
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
import { describe, expect, it } from "bun:test";
2+
import { frameFileExtension } from "@hyperframes/engine";
3+
import { resolveCaptureImageFormat } from "./captureImageFormat.js";
4+
5+
describe("resolveCaptureImageFormat", () => {
6+
it("captures JPEG for an ordinary opaque render", () => {
7+
expect(resolveCaptureImageFormat({ needsAlpha: false, motionBlur: undefined })).toBe("jpeg");
8+
});
9+
10+
it("captures PNG when the output container keeps alpha", () => {
11+
expect(resolveCaptureImageFormat({ needsAlpha: true, motionBlur: undefined })).toBe("png");
12+
});
13+
14+
it("captures PNG for motion blur even when the output needs no alpha", () => {
15+
// The sub-frame samples are averaged pixel by pixel, so they cannot be JPEG.
16+
expect(resolveCaptureImageFormat({ needsAlpha: false, motionBlur: {} })).toBe("png");
17+
});
18+
19+
it("agrees with the name the capture writer gives a frame", () => {
20+
// The encoder's input pattern is built from this extension, so a disagreement here is
21+
// a render that captures frames and then finds none.
22+
for (const needsAlpha of [false, true]) {
23+
for (const motionBlur of [undefined, {}]) {
24+
const format = resolveCaptureImageFormat({ needsAlpha, motionBlur });
25+
expect(frameFileExtension(format)).toBe(format === "png" ? "png" : "jpg");
26+
}
27+
}
28+
});
29+
});
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
import type { MotionBlurOptions } from "@hyperframes/engine";
2+
3+
/**
4+
* The format a render captures its frames in.
5+
*
6+
* Two independent reasons force PNG, and conflating them is how the encoder ends up
7+
* hunting for `frame_%06d.jpg` against files written as `.png`:
8+
*
9+
* - the output container keeps alpha, which JPEG cannot carry;
10+
* - motion blur is on, because the sub-frame samples are averaged pixel by pixel and
11+
* JPEG samples would be averaged after a lossy quantization.
12+
*
13+
* Every consumer of "is this render PNG" reads the resolved capture format rather than
14+
* re-deriving it from one of the reasons.
15+
*/
16+
export function resolveCaptureImageFormat(input: {
17+
needsAlpha: boolean;
18+
motionBlur: MotionBlurOptions | undefined;
19+
}): "jpeg" | "png" {
20+
return input.needsAlpha || input.motionBlur ? "png" : "jpeg";
21+
}

‎packages/producer/src/services/render/stages/captureStage.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@
3838

3939
import { statfsSync } from "node:fs";
4040
import {
41+
frameFileExtension,
4142
type BeforeCaptureHook,
4243
type CaptureOptions,
4344
type CapturePerfSummary,
@@ -220,7 +221,7 @@ export async function runCaptureStage(input: CaptureStageInput): Promise<Capture
220221
} = input;
221222
let { probeSession } = input;
222223
let { workerCount } = plan;
223-
const { forceScreenshot, needsAlpha } = plan;
224+
const { forceScreenshot } = plan;
224225
let lastBrowserConsole: string[] = [];
225226
let captureBeyondViewport: boolean | undefined = probeSession?.options.captureBeyondViewport;
226227

@@ -271,7 +272,7 @@ export async function runCaptureStage(input: CaptureStageInput): Promise<Capture
271272
totalFrames,
272273
initialWorkerCount: workerCount,
273274
allowRetry: shouldAllowAdaptiveCaptureRetry(workerCount, job.config.workers !== undefined),
274-
frameExt: needsAlpha ? "png" : "jpg",
275+
frameExt: frameFileExtension(captureOptions.format),
275276
captureOptions,
276277
createBeforeCaptureHook: createRenderVideoFrameInjector,
277278
abortSignal,

‎packages/producer/src/services/render/stages/encodeStage.test.ts‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ mock.module("@hyperframes/engine", () => ({
3636
DEFAULT_CONFIG: { ffmpegEncodeTimeout: 600_000 },
3737
encodeFramesChunkedConcat: encodeFramesChunkedConcatMock,
3838
encodeFramesFromDir: encodeFramesFromDirMock,
39+
frameFileExtension: (format: string | undefined) => (format === "png" ? "png" : "jpg"),
3940
formatFfmpegError: (code: number | null, stderr: string) => `${String(code)} ${stderr}`,
4041
getEncoderPreset: () => ({
4142
codec: "h264",
@@ -94,6 +95,7 @@ function makeInput(overrides: Partial<EncodeStageInput> = {}): EncodeStageInput
9495
width: 2,
9596
height: 2,
9697
needsAlpha: false,
98+
captureImageFormat: "jpeg" as const,
9799
hasAudio: false,
98100
isPngSequence: false,
99101
isGif: false,
@@ -170,6 +172,38 @@ describe("gif encode args", () => {
170172
});
171173
});
172174

175+
describe("frame pattern follows the capture format, not the output's alpha need", () => {
176+
it("encodes frame_%06d.png for a motion-blur render whose output is opaque", async () => {
177+
const { runEncodeStage } = await import("./encodeStage.js");
178+
const paths = createFramesDir("png");
179+
encodeFramesFromDirMock.mockClear();
180+
181+
await runEncodeStage(
182+
makeInput({
183+
framesDir: paths.framesDir,
184+
outputPath: join(paths.root, "out.mp4"),
185+
videoOnlyPath: join(paths.root, "video-only.mp4"),
186+
// Motion blur forces PNG capture even though an mp4 output needs no alpha. Before
187+
// the capture format owned this, the encoder looked for frame_%06d.jpg against
188+
// files written as .png and the render found no frames at all.
189+
needsAlpha: false,
190+
captureImageFormat: "png",
191+
}),
192+
);
193+
194+
expect(encodeFramesFromDirMock.mock.calls[0]?.[1]).toBe("frame_%06d.png");
195+
});
196+
197+
it("still encodes frame_%06d.jpg for an ordinary opaque render", async () => {
198+
const { runEncodeStage } = await import("./encodeStage.js");
199+
encodeFramesFromDirMock.mockClear();
200+
201+
await runEncodeStage(makeInput());
202+
203+
expect(encodeFramesFromDirMock.mock.calls[0]?.[1]).toBe("frame_%06d.jpg");
204+
});
205+
});
206+
173207
describe("runEncodeStage config plumbing", () => {
174208
it("throws a typed retryable error when the GIF encoder is externally interrupted", async () => {
175209
const { EncoderInterruptedError } = await import("../encoderInterruption.js");
@@ -379,6 +413,7 @@ describe("runEncodeStage config plumbing", () => {
379413
videoOnlyPath: join(paths.root, "video-only.mp4"),
380414
isGif: true,
381415
needsAlpha: true,
416+
captureImageFormat: "png",
382417
}),
383418
);
384419

‎packages/producer/src/services/render/stages/encodeStage.ts‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ import {
5252
} from "./gifEncodeArgs.js";
5353
import { updateJobStatus } from "../shared.js";
5454
import { encoderFailureError } from "../encoderInterruption.js";
55+
import { frameFileExtension } from "@hyperframes/engine";
5556

5657
export interface EncodeStageInput {
5758
job: RenderJob;
@@ -67,6 +68,12 @@ export interface EncodeStageInput {
6768
height: number;
6869
/** True when the output format requires an alpha channel; selects frame extension. */
6970
needsAlpha: boolean;
71+
/**
72+
* Format the frames on disk were captured in. Drives the encoder's input pattern, which
73+
* must match what `writeCapturedFrame` named them. Not derivable from `needsAlpha`:
74+
* motion blur also forces PNG capture on an opaque output.
75+
*/
76+
captureImageFormat: "jpeg" | "png";
7077
/** True iff the composition has audio. Drives the sidecar copy. */
7178
hasAudio: boolean;
7279
/**
@@ -273,7 +280,7 @@ export async function runEncodeStage(input: EncodeStageInput): Promise<EncodeSta
273280
if (hasAudio) {
274281
log.warn("[Render] GIF output does not support audio; audio tracks will be ignored.");
275282
}
276-
const frameExt = needsAlpha ? "png" : "jpg";
283+
const frameExt = frameFileExtension(input.captureImageFormat);
277284
const framePattern = `frame_%06d.${frameExt}`;
278285
const loop = resolveGifLoop(job.config.gifLoop);
279286
const encodeResult = await encodeGifFromDir(framesDir, framePattern, outputPath, {
@@ -306,7 +313,7 @@ export async function runEncodeStage(input: EncodeStageInput): Promise<EncodeSta
306313
? { ...engineCfg, ffmpegEncodeTimeout: scaledEncodeTimeout }
307314
: engineCfg;
308315

309-
const frameExt = needsAlpha ? "png" : "jpg";
316+
const frameExt = frameFileExtension(input.captureImageFormat);
310317
const framePattern = `frame_%06d.${frameExt}`;
311318
const encoderOpts = {
312319
fps: job.config.fps,

‎packages/producer/src/services/render/stages/probeStage.ts‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ import {
6969
import type { RenderJob } from "../../renderOrchestrator.js";
7070
import { isActionableProbeFailure } from "./probeFailures.js";
7171
import { preflightCompositionAssetMediaTypes } from "../../assetMediaType.js";
72+
import { resolveCaptureImageFormat } from "../captureImageFormat.js";
7273

7374
export interface ProbeStageInput {
7475
projectDir: string;
@@ -328,14 +329,21 @@ export async function runProbeStage(input: ProbeStageInput): Promise<ProbeStageR
328329
});
329330
assertNotAborted();
330331

332+
// Motion blur averages the sub-frame samples pixel by pixel, so the frames have to be
333+
// captured losslessly even when the output container wants no alpha. This is the one
334+
// place a render picks its capture format; the encoder reads the result rather than
335+
// re-deriving it, so the two cannot disagree about what is on disk.
336+
const motionBlur = job.config.motionBlur;
337+
const captureFormat = resolveCaptureImageFormat({ needsAlpha, motionBlur });
331338
const captureOpts: CaptureOptions = {
332339
width,
333340
height,
334341
fps: job.config.fps,
335-
format: needsAlpha ? "png" : "jpeg",
336-
quality: needsAlpha ? undefined : 80,
342+
format: captureFormat,
343+
quality: captureFormat === "png" ? undefined : 80,
337344
variables: job.config.variables,
338345
deviceScaleFactor,
346+
motionBlur,
339347
};
340348

341349
const PROBE_MAX_ATTEMPTS = 2;

0 commit comments

Comments
 (0)