Skip to content

Commit ece9a20

Browse files
miguel-heygenmiga-heygen
authored andcommitted
feat(producer): make sub-frame motion blur reachable from a render
Second deliverable for issue #4010. The engine gained the accumulation path, but nothing set CaptureOptions.motionBlur, so the feature could not be switched on from a real render. RenderConfig now carries it and the producer threads it into the capture options. Doing that surfaced the defect that would have made it fail silently. "Does this render capture PNG frames" was spelled four times, every one of them re-derived from needsAlpha, which describes the OUTPUT container rather than the capture: probeStage picked the probe capture format renderOrchestrator picked the render capture format encodeStage built the encoder's frame_%06d input pattern, twice captureStage built the disk worker's frame extension Motion blur is a second, independent reason to capture PNG: the sub-frame samples are averaged pixel by pixel, so JPEG samples would be averaged after a lossy quantization. Add that reason to only one of those four and the encoder goes looking for frame_%06d.jpg against files written as .png, and the render finds no frames at all rather than reporting an error. So the rule now has one owner, resolveCaptureImageFormat, which names both reasons. Everyone downstream reads the resolved capture format instead of re-deriving it: the encoder and the disk worker share frameFileExtension with writeCapturedFrame, which is what actually names the files. needsAlpha keeps only the question it really answers, whether the output preserves alpha. captureStage no longer needs needsAlpha at all, which is the tell that the re-derivation was its only reason to know. No JPEG encoder is involved. The PNG capture path is already load-bearing, since every transparent-output render uses it and the streaming encoder already selects png as its input codec. Co-Authored-By: Miguel Ángel <[email protected]>
1 parent 462f9db commit ece9a20

10 files changed

Lines changed: 137 additions & 9 deletions

File tree

‎packages/engine/src/index.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -273,6 +273,8 @@ export {
273273

274274
// ── Utilities ──────────────────────────────────────────────────────────────────
275275
export { quantizeTimeToFrame, MEDIA_VISUAL_STYLE_PROPERTIES } from "@hyperframes/core";
276+
export { frameFileExtension } from "./services/frameCapture.js";
277+
export type { MotionBlurOptions, MotionBlurBlendSpace } from "./services/motionBlur.js";
276278

277279
export {
278280
assertSwiftShader,

‎packages/engine/src/services/frameCapture.ts‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3881,6 +3881,16 @@ export async function captureFrame(
38813881
return { frameIndex, time: quantizedTime, path: framePath, captureTimeMs };
38823882
}
38833883

3884+
/**
3885+
* File extension for a captured frame, keyed on the format the frames were actually
3886+
* captured in. The encoder's input pattern and the writer must agree, so both read this
3887+
* rather than re-deriving the answer from whether the OUTPUT needs alpha, which is a
3888+
* different question and stops being equivalent as soon as anything else forces PNG.
3889+
*/
3890+
export function frameFileExtension(format: "jpeg" | "png" | undefined): "png" | "jpg" {
3891+
return format === "png" ? "png" : "jpg";
3892+
}
3893+
38843894
/**
38853895
* Write an already-captured frame buffer to the session's output dir using the
38863896
* canonical `frame_NNNNNN.{jpg,png}` naming. `fileIndex` is the ENCODER-facing
@@ -3894,7 +3904,7 @@ export function writeCapturedFrame(
38943904
fileIndex: number,
38953905
buffer: Buffer,
38963906
): string {
3897-
const ext = session.options.format === "png" ? "png" : "jpg";
3907+
const ext = frameFileExtension(session.options.format);
38983908
const framePath = join(session.outputDir, `frame_${String(fileIndex).padStart(6, "0")}.${ext}`);
38993909
writeFileSync(framePath, buffer);
39003910
return framePath;

‎packages/producer/src/services/distributed/renderChunk.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -957,6 +957,7 @@ export async function renderChunk(
957957
width: plan.dimensions.width * encoder.deviceScaleFactor,
958958
height: plan.dimensions.height * encoder.deviceScaleFactor,
959959
needsAlpha: plan.dimensions.format !== "mp4",
960+
captureImageFormat: captureOptions.format ?? "jpeg",
960961
// Each chunk produces video only — audio is muxed once at assemble
961962
// time. Suppressing `hasAudio` skips the png-sequence audio sidecar
962963
// AND the mp4 audio mux.
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
import { describe, expect, it } from "bun:test";
2+
import { frameFileExtension } from "@hyperframes/engine";
3+
import { resolveCaptureImageFormat } from "./captureImageFormat.js";
4+
5+
describe("resolveCaptureImageFormat", () => {
6+
it("captures JPEG for an ordinary opaque render", () => {
7+
expect(resolveCaptureImageFormat({ needsAlpha: false, motionBlur: undefined })).toBe("jpeg");
8+
});
9+
10+
it("captures PNG when the output container keeps alpha", () => {
11+
expect(resolveCaptureImageFormat({ needsAlpha: true, motionBlur: undefined })).toBe("png");
12+
});
13+
14+
it("captures PNG for motion blur even when the output needs no alpha", () => {
15+
// The sub-frame samples are averaged pixel by pixel, so they cannot be JPEG.
16+
expect(resolveCaptureImageFormat({ needsAlpha: false, motionBlur: {} })).toBe("png");
17+
});
18+
19+
it("agrees with the name the capture writer gives a frame", () => {
20+
// The encoder's input pattern is built from this extension, so a disagreement here is
21+
// a render that captures frames and then finds none.
22+
for (const needsAlpha of [false, true]) {
23+
for (const motionBlur of [undefined, {}]) {
24+
const format = resolveCaptureImageFormat({ needsAlpha, motionBlur });
25+
expect(frameFileExtension(format)).toBe(format === "png" ? "png" : "jpg");
26+
}
27+
}
28+
});
29+
});
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
import type { MotionBlurOptions } from "@hyperframes/engine";
2+
3+
/**
4+
* The format a render captures its frames in.
5+
*
6+
* Two independent reasons force PNG, and conflating them is how the encoder ends up
7+
* hunting for `frame_%06d.jpg` against files written as `.png`:
8+
*
9+
* - the output container keeps alpha, which JPEG cannot carry;
10+
* - motion blur is on, because the sub-frame samples are averaged pixel by pixel and
11+
* JPEG samples would be averaged after a lossy quantization.
12+
*
13+
* Every consumer of "is this render PNG" reads the resolved capture format rather than
14+
* re-deriving it from one of the reasons.
15+
*/
16+
export function resolveCaptureImageFormat(input: {
17+
needsAlpha: boolean;
18+
motionBlur: MotionBlurOptions | undefined;
19+
}): "jpeg" | "png" {
20+
return input.needsAlpha || input.motionBlur ? "png" : "jpeg";
21+
}

‎packages/producer/src/services/render/stages/captureStage.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@
3838

3939
import { statfsSync } from "node:fs";
4040
import {
41+
frameFileExtension,
4142
type BeforeCaptureHook,
4243
type CaptureOptions,
4344
type CapturePerfSummary,
@@ -220,7 +221,7 @@ export async function runCaptureStage(input: CaptureStageInput): Promise<Capture
220221
} = input;
221222
let { probeSession } = input;
222223
let { workerCount } = plan;
223-
const { forceScreenshot, needsAlpha } = plan;
224+
const { forceScreenshot } = plan;
224225
let lastBrowserConsole: string[] = [];
225226
let captureBeyondViewport: boolean | undefined = probeSession?.options.captureBeyondViewport;
226227

@@ -271,7 +272,7 @@ export async function runCaptureStage(input: CaptureStageInput): Promise<Capture
271272
totalFrames,
272273
initialWorkerCount: workerCount,
273274
allowRetry: shouldAllowAdaptiveCaptureRetry(workerCount, job.config.workers !== undefined),
274-
frameExt: needsAlpha ? "png" : "jpg",
275+
frameExt: frameFileExtension(captureOptions.format),
275276
captureOptions,
276277
createBeforeCaptureHook: createRenderVideoFrameInjector,
277278
abortSignal,

‎packages/producer/src/services/render/stages/encodeStage.test.ts‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ mock.module("@hyperframes/engine", () => ({
3636
DEFAULT_CONFIG: { ffmpegEncodeTimeout: 600_000 },
3737
encodeFramesChunkedConcat: encodeFramesChunkedConcatMock,
3838
encodeFramesFromDir: encodeFramesFromDirMock,
39+
frameFileExtension: (format: string | undefined) => (format === "png" ? "png" : "jpg"),
3940
formatFfmpegError: (code: number | null, stderr: string) => `${String(code)} ${stderr}`,
4041
getEncoderPreset: () => ({
4142
codec: "h264",
@@ -94,6 +95,7 @@ function makeInput(overrides: Partial<EncodeStageInput> = {}): EncodeStageInput
9495
width: 2,
9596
height: 2,
9697
needsAlpha: false,
98+
captureImageFormat: "jpeg" as const,
9799
hasAudio: false,
98100
isPngSequence: false,
99101
isGif: false,
@@ -170,6 +172,38 @@ describe("gif encode args", () => {
170172
});
171173
});
172174

175+
describe("frame pattern follows the capture format, not the output's alpha need", () => {
176+
it("encodes frame_%06d.png for a motion-blur render whose output is opaque", async () => {
177+
const { runEncodeStage } = await import("./encodeStage.js");
178+
const paths = createFramesDir("png");
179+
encodeFramesFromDirMock.mockClear();
180+
181+
await runEncodeStage(
182+
makeInput({
183+
framesDir: paths.framesDir,
184+
outputPath: join(paths.root, "out.mp4"),
185+
videoOnlyPath: join(paths.root, "video-only.mp4"),
186+
// Motion blur forces PNG capture even though an mp4 output needs no alpha. Before
187+
// the capture format owned this, the encoder looked for frame_%06d.jpg against
188+
// files written as .png and the render found no frames at all.
189+
needsAlpha: false,
190+
captureImageFormat: "png",
191+
}),
192+
);
193+
194+
expect(encodeFramesFromDirMock.mock.calls[0]?.[1]).toBe("frame_%06d.png");
195+
});
196+
197+
it("still encodes frame_%06d.jpg for an ordinary opaque render", async () => {
198+
const { runEncodeStage } = await import("./encodeStage.js");
199+
encodeFramesFromDirMock.mockClear();
200+
201+
await runEncodeStage(makeInput());
202+
203+
expect(encodeFramesFromDirMock.mock.calls[0]?.[1]).toBe("frame_%06d.jpg");
204+
});
205+
});
206+
173207
describe("runEncodeStage config plumbing", () => {
174208
it("throws a typed retryable error when the GIF encoder is externally interrupted", async () => {
175209
const { EncoderInterruptedError } = await import("../encoderInterruption.js");
@@ -379,6 +413,7 @@ describe("runEncodeStage config plumbing", () => {
379413
videoOnlyPath: join(paths.root, "video-only.mp4"),
380414
isGif: true,
381415
needsAlpha: true,
416+
captureImageFormat: "png",
382417
}),
383418
);
384419

‎packages/producer/src/services/render/stages/encodeStage.ts‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ import {
5252
} from "./gifEncodeArgs.js";
5353
import { updateJobStatus } from "../shared.js";
5454
import { encoderFailureError } from "../encoderInterruption.js";
55+
import { frameFileExtension } from "@hyperframes/engine";
5556

5657
export interface EncodeStageInput {
5758
job: RenderJob;
@@ -67,6 +68,12 @@ export interface EncodeStageInput {
6768
height: number;
6869
/** True when the output format requires an alpha channel; selects frame extension. */
6970
needsAlpha: boolean;
71+
/**
72+
* Format the frames on disk were captured in. Drives the encoder's input pattern, which
73+
* must match what `writeCapturedFrame` named them. Not derivable from `needsAlpha`:
74+
* motion blur also forces PNG capture on an opaque output.
75+
*/
76+
captureImageFormat: "jpeg" | "png";
7077
/** True iff the composition has audio. Drives the sidecar copy. */
7178
hasAudio: boolean;
7279
/**
@@ -273,7 +280,7 @@ export async function runEncodeStage(input: EncodeStageInput): Promise<EncodeSta
273280
if (hasAudio) {
274281
log.warn("[Render] GIF output does not support audio; audio tracks will be ignored.");
275282
}
276-
const frameExt = needsAlpha ? "png" : "jpg";
283+
const frameExt = frameFileExtension(input.captureImageFormat);
277284
const framePattern = `frame_%06d.${frameExt}`;
278285
const loop = resolveGifLoop(job.config.gifLoop);
279286
const encodeResult = await encodeGifFromDir(framesDir, framePattern, outputPath, {
@@ -306,7 +313,7 @@ export async function runEncodeStage(input: EncodeStageInput): Promise<EncodeSta
306313
? { ...engineCfg, ffmpegEncodeTimeout: scaledEncodeTimeout }
307314
: engineCfg;
308315

309-
const frameExt = needsAlpha ? "png" : "jpg";
316+
const frameExt = frameFileExtension(input.captureImageFormat);
310317
const framePattern = `frame_%06d.${frameExt}`;
311318
const encoderOpts = {
312319
fps: job.config.fps,

‎packages/producer/src/services/render/stages/probeStage.ts‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ import {
6969
import type { RenderJob } from "../../renderOrchestrator.js";
7070
import { isActionableProbeFailure } from "./probeFailures.js";
7171
import { preflightCompositionAssetMediaTypes } from "../../assetMediaType.js";
72+
import { resolveCaptureImageFormat } from "../captureImageFormat.js";
7273

7374
export interface ProbeStageInput {
7475
projectDir: string;
@@ -328,14 +329,21 @@ export async function runProbeStage(input: ProbeStageInput): Promise<ProbeStageR
328329
});
329330
assertNotAborted();
330331

332+
// Motion blur averages the sub-frame samples pixel by pixel, so the frames have to be
333+
// captured losslessly even when the output container wants no alpha. This is the one
334+
// place a render picks its capture format; the encoder reads the result rather than
335+
// re-deriving it, so the two cannot disagree about what is on disk.
336+
const motionBlur = job.config.motionBlur;
337+
const captureFormat = resolveCaptureImageFormat({ needsAlpha, motionBlur });
331338
const captureOpts: CaptureOptions = {
332339
width,
333340
height,
334341
fps: job.config.fps,
335-
format: needsAlpha ? "png" : "jpeg",
336-
quality: needsAlpha ? undefined : 80,
342+
format: captureFormat,
343+
quality: captureFormat === "png" ? undefined : 80,
337344
variables: job.config.variables,
338345
deviceScaleFactor,
346+
motionBlur,
339347
};
340348

341349
const PROBE_MAX_ATTEMPTS = 2;

‎packages/producer/src/services/renderOrchestrator.ts‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,7 @@ import {
9393
getDrawElementVerificationDetails,
9494
augmentProtocolTimeoutError,
9595
augmentPageNavigationTimeoutError,
96+
type MotionBlurOptions,
9697
} from "@hyperframes/engine";
9798
import { join, dirname, resolve } from "path";
9899
import { totalmem } from "node:os";
@@ -185,6 +186,7 @@ import { runCaptureHdrStage } from "./render/stages/captureHdrStage.js";
185186
import { runEncodeStage } from "./render/stages/encodeStage.js";
186187
import { runAssembleStage } from "./render/stages/assembleStage.js";
187188
import { shouldUseLayeredComposite } from "./hdrCompositor.js";
189+
import { resolveCaptureImageFormat } from "./render/captureImageFormat.js";
188190

189191
function sampleDirectoryBytes(dir: string): number {
190192
let total = 0;
@@ -346,6 +348,12 @@ export interface RenderConfig {
346348
* A non-integer value throws at the start of `executeRenderJob`.
347349
*/
348350
hlsSegmentSeconds?: number;
351+
/**
352+
* Opt into sub-frame multi-sample motion blur. Absent means off and the capture path is
353+
* unchanged. Forces PNG frame capture, because the samples are averaged pixel by pixel
354+
* and JPEG samples would be averaged after a lossy quantization.
355+
*/
356+
motionBlur?: MotionBlurOptions;
349357
workers?: number;
350358
useGpu?: boolean;
351359
debug?: boolean;
@@ -2925,14 +2933,19 @@ async function executeRenderPipeline(input: {
29252933
});
29262934
const videoCaptureBeyondViewport = resolveVideoCaptureBeyondViewport(composition.videos.length);
29272935

2936+
const captureImageFormat = resolveCaptureImageFormat({
2937+
needsAlpha,
2938+
motionBlur: job.config.motionBlur,
2939+
});
29282940
const captureOptions: CaptureOptions = {
29292941
width,
29302942
height,
29312943
fps: job.config.fps,
2932-
format: needsAlpha ? "png" : "jpeg",
2933-
quality: needsAlpha ? undefined : job.config.quality === "draft" ? 80 : 95,
2944+
format: captureImageFormat,
2945+
quality: captureImageFormat === "png" ? undefined : job.config.quality === "draft" ? 80 : 95,
29342946
variables: job.config.variables,
29352947
deviceScaleFactor,
2948+
motionBlur: job.config.motionBlur,
29362949
...(videoCaptureBeyondViewport !== undefined
29372950
? { captureBeyondViewport: videoCaptureBeyondViewport }
29382951
: {}),
@@ -4193,6 +4206,7 @@ async function executeRenderPipeline(input: {
41934206
width,
41944207
height,
41954208
needsAlpha,
4209+
captureImageFormat: captureOptions.format ?? "jpeg",
41964210
hasAudio,
41974211
audioOutputPath,
41984212
isPngSequence,

0 commit comments

Comments
 (0)