Skip to content

Commit 482fb0e

Browse files
miguel-heygenmiga-heygen
authored andcommitted
feat(producer): make sub-frame motion blur reachable from a render
Second deliverable for issue #4010. The engine gained the accumulation path, but nothing set CaptureOptions.motionBlur, so the feature could not be switched on from a real render. RenderConfig now carries it and the producer threads it into the capture options. Doing that surfaced the defect that would have made it fail silently. "Does this render capture PNG frames" was spelled four times, every one of them re-derived from needsAlpha, which describes the OUTPUT container rather than the capture: probeStage picked the probe capture format renderOrchestrator picked the render capture format encodeStage built the encoder's frame_%06d input pattern, twice captureStage built the disk worker's frame extension Motion blur is a second, independent reason to capture PNG: the sub-frame samples are averaged pixel by pixel, so JPEG samples would be averaged after a lossy quantization. Add that reason to only one of those four and the encoder goes looking for frame_%06d.jpg against files written as .png, and the render finds no frames at all rather than reporting an error. So the rule now has one owner, resolveCaptureImageFormat, which names both reasons. Everyone downstream reads the resolved capture format instead of re-deriving it: the encoder and the disk worker share frameFileExtension with writeCapturedFrame, which is what actually names the files. needsAlpha keeps only the question it really answers, whether the output preserves alpha. captureStage no longer needs needsAlpha at all, which is the tell that the re-derivation was its only reason to know. No JPEG encoder is involved. The PNG capture path is already load-bearing, since every transparent-output render uses it and the streaming encoder already selects png as its input codec. Co-Authored-By: Miguel Ángel <[email protected]>
1 parent 0a1c6f5 commit 482fb0e

10 files changed

Lines changed: 137 additions & 9 deletions

File tree

‎packages/engine/src/index.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,6 +268,8 @@ export {
268268

269269
// ── Utilities ──────────────────────────────────────────────────────────────────
270270
export { quantizeTimeToFrame, MEDIA_VISUAL_STYLE_PROPERTIES } from "@hyperframes/core";
271+
export { frameFileExtension } from "./services/frameCapture.js";
272+
export type { MotionBlurOptions, MotionBlurBlendSpace } from "./services/motionBlur.js";
271273

272274
export {
273275
assertSwiftShader,

‎packages/engine/src/services/frameCapture.ts‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3881,6 +3881,16 @@ export async function captureFrame(
38813881
return { frameIndex, time: quantizedTime, path: framePath, captureTimeMs };
38823882
}
38833883

3884+
/**
3885+
* File extension for a captured frame, keyed on the format the frames were actually
3886+
* captured in. The encoder's input pattern and the writer must agree, so both read this
3887+
* rather than re-deriving the answer from whether the OUTPUT needs alpha, which is a
3888+
* different question and stops being equivalent as soon as anything else forces PNG.
3889+
*/
3890+
export function frameFileExtension(format: "jpeg" | "png" | undefined): "png" | "jpg" {
3891+
return format === "png" ? "png" : "jpg";
3892+
}
3893+
38843894
/**
38853895
* Write an already-captured frame buffer to the session's output dir using the
38863896
* canonical `frame_NNNNNN.{jpg,png}` naming. `fileIndex` is the ENCODER-facing
@@ -3894,7 +3904,7 @@ export function writeCapturedFrame(
38943904
fileIndex: number,
38953905
buffer: Buffer,
38963906
): string {
3897-
const ext = session.options.format === "png" ? "png" : "jpg";
3907+
const ext = frameFileExtension(session.options.format);
38983908
const framePath = join(session.outputDir, `frame_${String(fileIndex).padStart(6, "0")}.${ext}`);
38993909
writeFileSync(framePath, buffer);
39003910
return framePath;

‎packages/producer/src/services/distributed/renderChunk.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -957,6 +957,7 @@ export async function renderChunk(
957957
width: plan.dimensions.width * encoder.deviceScaleFactor,
958958
height: plan.dimensions.height * encoder.deviceScaleFactor,
959959
needsAlpha: plan.dimensions.format !== "mp4",
960+
captureImageFormat: captureOptions.format ?? "jpeg",
960961
// Each chunk produces video only — audio is muxed once at assemble
961962
// time. Suppressing `hasAudio` skips the png-sequence audio sidecar
962963
// AND the mp4 audio mux.
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
import { describe, expect, it } from "bun:test";
2+
import { frameFileExtension } from "@hyperframes/engine";
3+
import { resolveCaptureImageFormat } from "./captureImageFormat.js";
4+
5+
describe("resolveCaptureImageFormat", () => {
6+
it("captures JPEG for an ordinary opaque render", () => {
7+
expect(resolveCaptureImageFormat({ needsAlpha: false, motionBlur: undefined })).toBe("jpeg");
8+
});
9+
10+
it("captures PNG when the output container keeps alpha", () => {
11+
expect(resolveCaptureImageFormat({ needsAlpha: true, motionBlur: undefined })).toBe("png");
12+
});
13+
14+
it("captures PNG for motion blur even when the output needs no alpha", () => {
15+
// The sub-frame samples are averaged pixel by pixel, so they cannot be JPEG.
16+
expect(resolveCaptureImageFormat({ needsAlpha: false, motionBlur: {} })).toBe("png");
17+
});
18+
19+
it("agrees with the name the capture writer gives a frame", () => {
20+
// The encoder's input pattern is built from this extension, so a disagreement here is
21+
// a render that captures frames and then finds none.
22+
for (const needsAlpha of [false, true]) {
23+
for (const motionBlur of [undefined, {}]) {
24+
const format = resolveCaptureImageFormat({ needsAlpha, motionBlur });
25+
expect(frameFileExtension(format)).toBe(format === "png" ? "png" : "jpg");
26+
}
27+
}
28+
});
29+
});
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
import type { MotionBlurOptions } from "@hyperframes/engine";
2+
3+
/**
4+
* The format a render captures its frames in.
5+
*
6+
* Two independent reasons force PNG, and conflating them is how the encoder ends up
7+
* hunting for `frame_%06d.jpg` against files written as `.png`:
8+
*
9+
* - the output container keeps alpha, which JPEG cannot carry;
10+
* - motion blur is on, because the sub-frame samples are averaged pixel by pixel and
11+
* JPEG samples would be averaged after a lossy quantization.
12+
*
13+
* Every consumer of "is this render PNG" reads the resolved capture format rather than
14+
* re-deriving it from one of the reasons.
15+
*/
16+
export function resolveCaptureImageFormat(input: {
17+
needsAlpha: boolean;
18+
motionBlur: MotionBlurOptions | undefined;
19+
}): "jpeg" | "png" {
20+
return input.needsAlpha || input.motionBlur ? "png" : "jpeg";
21+
}

‎packages/producer/src/services/render/stages/captureStage.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@
3838

3939
import { statfsSync } from "node:fs";
4040
import {
41+
frameFileExtension,
4142
type BeforeCaptureHook,
4243
type CaptureOptions,
4344
type CapturePerfSummary,
@@ -220,7 +221,7 @@ export async function runCaptureStage(input: CaptureStageInput): Promise<Capture
220221
} = input;
221222
let { probeSession } = input;
222223
let { workerCount } = plan;
223-
const { forceScreenshot, needsAlpha } = plan;
224+
const { forceScreenshot } = plan;
224225
let lastBrowserConsole: string[] = [];
225226
let captureBeyondViewport: boolean | undefined = probeSession?.options.captureBeyondViewport;
226227

@@ -271,7 +272,7 @@ export async function runCaptureStage(input: CaptureStageInput): Promise<Capture
271272
totalFrames,
272273
initialWorkerCount: workerCount,
273274
allowRetry: shouldAllowAdaptiveCaptureRetry(workerCount, job.config.workers !== undefined),
274-
frameExt: needsAlpha ? "png" : "jpg",
275+
frameExt: frameFileExtension(captureOptions.format),
275276
captureOptions,
276277
createBeforeCaptureHook: createRenderVideoFrameInjector,
277278
abortSignal,

‎packages/producer/src/services/render/stages/encodeStage.test.ts‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ mock.module("@hyperframes/engine", () => ({
3636
DEFAULT_CONFIG: { ffmpegEncodeTimeout: 600_000 },
3737
encodeFramesChunkedConcat: encodeFramesChunkedConcatMock,
3838
encodeFramesFromDir: encodeFramesFromDirMock,
39+
frameFileExtension: (format: string | undefined) => (format === "png" ? "png" : "jpg"),
3940
formatFfmpegError: (code: number | null, stderr: string) => `${String(code)} ${stderr}`,
4041
getEncoderPreset: () => ({
4142
codec: "h264",
@@ -94,6 +95,7 @@ function makeInput(overrides: Partial<EncodeStageInput> = {}): EncodeStageInput
9495
width: 2,
9596
height: 2,
9697
needsAlpha: false,
98+
captureImageFormat: "jpeg" as const,
9799
hasAudio: false,
98100
isPngSequence: false,
99101
isGif: false,
@@ -170,6 +172,38 @@ describe("gif encode args", () => {
170172
});
171173
});
172174

175+
describe("frame pattern follows the capture format, not the output's alpha need", () => {
176+
it("encodes frame_%06d.png for a motion-blur render whose output is opaque", async () => {
177+
const { runEncodeStage } = await import("./encodeStage.js");
178+
const paths = createFramesDir("png");
179+
encodeFramesFromDirMock.mockClear();
180+
181+
await runEncodeStage(
182+
makeInput({
183+
framesDir: paths.framesDir,
184+
outputPath: join(paths.root, "out.mp4"),
185+
videoOnlyPath: join(paths.root, "video-only.mp4"),
186+
// Motion blur forces PNG capture even though an mp4 output needs no alpha. Before
187+
// the capture format owned this, the encoder looked for frame_%06d.jpg against
188+
// files written as .png and the render found no frames at all.
189+
needsAlpha: false,
190+
captureImageFormat: "png",
191+
}),
192+
);
193+
194+
expect(encodeFramesFromDirMock.mock.calls[0]?.[1]).toBe("frame_%06d.png");
195+
});
196+
197+
it("still encodes frame_%06d.jpg for an ordinary opaque render", async () => {
198+
const { runEncodeStage } = await import("./encodeStage.js");
199+
encodeFramesFromDirMock.mockClear();
200+
201+
await runEncodeStage(makeInput());
202+
203+
expect(encodeFramesFromDirMock.mock.calls[0]?.[1]).toBe("frame_%06d.jpg");
204+
});
205+
});
206+
173207
describe("runEncodeStage config plumbing", () => {
174208
it("throws a typed retryable error when the GIF encoder is externally interrupted", async () => {
175209
const { EncoderInterruptedError } = await import("../encoderInterruption.js");
@@ -379,6 +413,7 @@ describe("runEncodeStage config plumbing", () => {
379413
videoOnlyPath: join(paths.root, "video-only.mp4"),
380414
isGif: true,
381415
needsAlpha: true,
416+
captureImageFormat: "png",
382417
}),
383418
);
384419

‎packages/producer/src/services/render/stages/encodeStage.ts‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ import {
5252
} from "./gifEncodeArgs.js";
5353
import { updateJobStatus } from "../shared.js";
5454
import { encoderFailureError } from "../encoderInterruption.js";
55+
import { frameFileExtension } from "@hyperframes/engine";
5556

5657
export interface EncodeStageInput {
5758
job: RenderJob;
@@ -67,6 +68,12 @@ export interface EncodeStageInput {
6768
height: number;
6869
/** True when the output format requires an alpha channel; selects frame extension. */
6970
needsAlpha: boolean;
71+
/**
72+
* Format the frames on disk were captured in. Drives the encoder's input pattern, which
73+
* must match what `writeCapturedFrame` named them. Not derivable from `needsAlpha`:
74+
* motion blur also forces PNG capture on an opaque output.
75+
*/
76+
captureImageFormat: "jpeg" | "png";
7077
/** True iff the composition has audio. Drives the sidecar copy. */
7178
hasAudio: boolean;
7279
/**
@@ -266,7 +273,7 @@ export async function runEncodeStage(input: EncodeStageInput): Promise<EncodeSta
266273
if (hasAudio) {
267274
log.warn("[Render] GIF output does not support audio; audio tracks will be ignored.");
268275
}
269-
const frameExt = needsAlpha ? "png" : "jpg";
276+
const frameExt = frameFileExtension(input.captureImageFormat);
270277
const framePattern = `frame_%06d.${frameExt}`;
271278
const loop = resolveGifLoop(job.config.gifLoop);
272279
const encodeResult = await encodeGifFromDir(framesDir, framePattern, outputPath, {
@@ -299,7 +306,7 @@ export async function runEncodeStage(input: EncodeStageInput): Promise<EncodeSta
299306
? { ...engineCfg, ffmpegEncodeTimeout: scaledEncodeTimeout }
300307
: engineCfg;
301308

302-
const frameExt = needsAlpha ? "png" : "jpg";
309+
const frameExt = frameFileExtension(input.captureImageFormat);
303310
const framePattern = `frame_%06d.${frameExt}`;
304311
const encoderOpts = {
305312
fps: job.config.fps,

‎packages/producer/src/services/render/stages/probeStage.ts‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ import {
6969
import type { RenderJob } from "../../renderOrchestrator.js";
7070
import { isActionableProbeFailure } from "./probeFailures.js";
7171
import { preflightCompositionAssetMediaTypes } from "../../assetMediaType.js";
72+
import { resolveCaptureImageFormat } from "../captureImageFormat.js";
7273

7374
export interface ProbeStageInput {
7475
projectDir: string;
@@ -328,14 +329,21 @@ export async function runProbeStage(input: ProbeStageInput): Promise<ProbeStageR
328329
});
329330
assertNotAborted();
330331

332+
// Motion blur averages the sub-frame samples pixel by pixel, so the frames have to be
333+
// captured losslessly even when the output container wants no alpha. This is the one
334+
// place a render picks its capture format; the encoder reads the result rather than
335+
// re-deriving it, so the two cannot disagree about what is on disk.
336+
const motionBlur = job.config.motionBlur;
337+
const captureFormat = resolveCaptureImageFormat({ needsAlpha, motionBlur });
331338
const captureOpts: CaptureOptions = {
332339
width,
333340
height,
334341
fps: job.config.fps,
335-
format: needsAlpha ? "png" : "jpeg",
336-
quality: needsAlpha ? undefined : 80,
342+
format: captureFormat,
343+
quality: captureFormat === "png" ? undefined : 80,
337344
variables: job.config.variables,
338345
deviceScaleFactor,
346+
motionBlur,
339347
};
340348

341349
const PROBE_MAX_ATTEMPTS = 2;

‎packages/producer/src/services/renderOrchestrator.ts‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,7 @@ import {
9393
getDrawElementVerificationDetails,
9494
augmentProtocolTimeoutError,
9595
augmentPageNavigationTimeoutError,
96+
type MotionBlurOptions,
9697
} from "@hyperframes/engine";
9798
import { join, dirname, resolve } from "path";
9899
import { totalmem } from "node:os";
@@ -175,6 +176,7 @@ import { runCaptureHdrStage } from "./render/stages/captureHdrStage.js";
175176
import { runEncodeStage } from "./render/stages/encodeStage.js";
176177
import { runAssembleStage } from "./render/stages/assembleStage.js";
177178
import { shouldUseLayeredComposite } from "./hdrCompositor.js";
179+
import { resolveCaptureImageFormat } from "./render/captureImageFormat.js";
178180

179181
function sampleDirectoryBytes(dir: string): number {
180182
let total = 0;
@@ -317,6 +319,12 @@ export interface RenderConfig {
317319
format?: RenderOutputFormat;
318320
/** GIF Netscape loop count. 0 means infinite looping. Only used with `format: "gif"`. */
319321
gifLoop?: number;
322+
/**
323+
* Opt into sub-frame multi-sample motion blur. Absent means off and the capture path is
324+
* unchanged. Forces PNG frame capture, because the samples are averaged pixel by pixel
325+
* and JPEG samples would be averaged after a lossy quantization.
326+
*/
327+
motionBlur?: MotionBlurOptions;
320328
workers?: number;
321329
useGpu?: boolean;
322330
debug?: boolean;
@@ -2882,14 +2890,19 @@ async function executeRenderPipeline(input: {
28822890
});
28832891
const videoCaptureBeyondViewport = resolveVideoCaptureBeyondViewport(composition.videos.length);
28842892

2893+
const captureImageFormat = resolveCaptureImageFormat({
2894+
needsAlpha,
2895+
motionBlur: job.config.motionBlur,
2896+
});
28852897
const captureOptions: CaptureOptions = {
28862898
width,
28872899
height,
28882900
fps: job.config.fps,
2889-
format: needsAlpha ? "png" : "jpeg",
2890-
quality: needsAlpha ? undefined : job.config.quality === "draft" ? 80 : 95,
2901+
format: captureImageFormat,
2902+
quality: captureImageFormat === "png" ? undefined : job.config.quality === "draft" ? 80 : 95,
28912903
variables: job.config.variables,
28922904
deviceScaleFactor,
2905+
motionBlur: job.config.motionBlur,
28932906
...(videoCaptureBeyondViewport !== undefined
28942907
? { captureBeyondViewport: videoCaptureBeyondViewport }
28952908
: {}),
@@ -4140,6 +4153,7 @@ async function executeRenderPipeline(input: {
41404153
width,
41414154
height,
41424155
needsAlpha,
4156+
captureImageFormat: captureOptions.format ?? "jpeg",
41434157
hasAudio,
41444158
audioOutputPath,
41454159
isPngSequence,

0 commit comments

Comments
 (0)