Skip to content

Commit 968f97f

Browse files
vanceingallsclaude
andauthored
feat(producer): render long videos with windowed source frames behind a flag (#7242)
* feat(producer): render long videos with windowed source frames, behind HF_VIDEO_FRAME_WINDOW Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(producer): windowed calibration sees real frames, and the late full extraction reports itself Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(producer): windowed frames match a full extraction, frame for frame Renders ten compositions twice on the segmented route, with and without HF_VIDEO_FRAME_WINDOW, and compares FFmpeg framemd5 of the two outputs: a mid-clip start, NTSC 24000/1001, a loop crossing a slice edge, a playback-rate ramp, a held tail, two clips of one file, a VFR source beside a CFR one, two workers, auto workers (capture calibration runs), and motion blur, which declines the window. Each pair must also report the same capture mode. A separate test samples segment-frames during a four-segment, two-worker render and holds its peak under workers x segment frames x the largest frame file x 1.2, and checks the folder is empty once the render closes. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * test(producer): windowed frames test finds its own work dir and asserts what it claims Find the render's work dir with resolveRenderWorkDirPrefix and the owner record's pid, so the peak test works on Linux and never samples or deletes a concurrent render's dir. Assert the measured worker count, calibration, and the exact up-front frame count of the VFR video. Retry a dead Chrome only with HF_TEST_RETRY_BROWSER_STALL=1, and fail a case whose windowed render alone needed it. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * fix(producer): windowed segments honour keep-segments and cleanup never fails a render Co-Authored-By: Claude Opus 5.5 <[email protected]> * refactor(producer): trim the comments the windowed frames route added Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(producer): the VFR-beside-CFR extraction test moves to the integration lane Co-Authored-By: Claude Opus 5.5 <[email protected]> * chore(producer): raise the package size budget for windowed video frames The new segment frame modules add about 9 KB to the packed producer package. Co-Authored-By: Claude Opus 5.5 <[email protected]> * test(producer): windowed frames header says what the real runs measured Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 478ef2f commit 968f97f

20 files changed

Lines changed: 1677 additions & 52 deletions

‎docs/packages/cli.mdx‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1929,12 +1929,29 @@ mismatched encodes. The directory is deleted after a successful render unless
19291929
mp4 and mov only; png-sequence, gif, webm, HDR and shader-transition renders
19301930
keep their existing routes.
19311931

1932+
By default every frame of every source video is extracted before capture
1933+
starts. With `HF_VIDEO_FRAME_WINDOW=true`, each segment instead extracts only
1934+
the source frames it shows and deletes them when it closes. Things to know
1935+
before switching it on:
1936+
1937+
- Segment files are deleted as soon as they are joined, unless `--keep-segments`
1938+
is set. A failure after the join therefore recaptures everything on `--resume`.
1939+
- Only mp4 and mov renders that segment honour the flag. Motion blur, variable
1940+
frame rate sources and HDR-transformed sources stay fully extracted. The
1941+
render log says when the flag was declined and how many videos stayed fully
1942+
extracted.
1943+
- Source-frame disk use is bounded by roughly workers x `HF_SEGMENT_FRAMES`
1944+
frames, and only for videos that can be windowed.
1945+
- A windowed render skips the extraction cache, so a re-export gets no
1946+
warm-cache speedup.
1947+
19321948
| Environment variable | Default | Effect |
19331949
| ------------------------------ | ---------- | ------------------------------------------------------------------------------------------ |
19341950
| `HF_SEGMENTED_CAPTURE` | unset | `true` segments any eligible render; `false` is the kill switch |
19351951
| `HF_SEGMENTED_MIN_SECONDS` | off | Segment renders at least this long. Unset means duration never routes on its own |
19361952
| `HF_SEGMENT_FRAMES` | `3000` | Frames per segment (~100 s at 30 fps); minimum 30 |
19371953
| `HF_SEGMENT_BROWSER_RECYCLE` | `3` | Restart the browser every N segments; `0` keeps one session for the whole render |
1954+
| `HF_VIDEO_FRAME_WINDOW` | unset | `true` extracts each segment's source frames when it starts and deletes them when it closes |
19381955

19391956
#### Render the same composition with different content
19401957

‎packages/producer/scripts/integration-tests.json‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
"src/services/render/renderCancel.integration.test.ts",
4141
"src/services/render/segmentedColorGrading.integration.test.ts",
4242
"src/services/render/variableMountMedia.integration.test.ts",
43+
"src/services/render/windowedFrames.integration.test.ts",
4344
"src/services/render/wrappedMountTiming.integration.test.ts",
4445
"src/services/render/stages/compileStage.mediaType.test.ts",
4546
"src/services/render/stages/encodeWhileCapturing.test.ts",
@@ -49,5 +50,6 @@
4950
"src/utils/streamDurationParity.test.ts",
5051
"src/services/distributed/referenceBackground.integration.test.ts",
5152
"src/services/render/planReferenceBackground.polling.test.ts",
52-
"src/services/render/deferredVideoFrames.integration.test.ts"
53+
"src/services/render/deferredVideoFrames.integration.test.ts",
54+
"src/services/render/stages/extractVideosStage.integration.test.ts"
5355
]

‎packages/producer/src/services/render/deferredVideoFrames.integration.test.ts‎

Lines changed: 56 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,14 @@
11
import { spawnSync } from "node:child_process";
2-
import { mkdirSync, mkdtempSync, rmSync } from "node:fs";
2+
import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync } from "node:fs";
33
import { tmpdir } from "node:os";
44
import { join } from "node:path";
55
import { afterAll, beforeAll, describe, expect, it } from "bun:test";
6-
import type { ExtractedFrames, VideoElement } from "@hyperframes/engine";
7-
import { extractDeferredFrames } from "./deferredVideoFrames.js";
6+
import {
7+
extractAllVideoFrames,
8+
type ExtractedFrames,
9+
type VideoElement,
10+
} from "@hyperframes/engine";
11+
import { extractAllDeferredFrames, extractDeferredFrames } from "./deferredVideoFrames.js";
812

913
const metadata = {
1014
durationSeconds: 4,
@@ -32,6 +36,11 @@ const deferredVideo = (videoId: string, totalFrames: number): ExtractedFrames =>
3236
deferredRange: { startTime: 1.5, durationSeconds: totalFrames / 30, format: "jpg" },
3337
});
3438

39+
const fullyExtracted = (videoId: string): ExtractedFrames => {
40+
const { deferredRange: _deferred, ...rest } = deferredVideo(videoId, 90);
41+
return rest;
42+
};
43+
3544
const element = (id: string, start: number, end: number): VideoElement => ({
3645
id,
3746
src: `${id}.mp4`,
@@ -95,4 +104,48 @@ describe("extractDeferredFrames", () => {
95104
await expect(extraction).rejects.toBe(missingError);
96105
expect(reported).toEqual([{ videoId: "clip", index: 45 }]);
97106
});
107+
describe("extractAllDeferredFrames", () => {
108+
const framesOf = (dir: string) =>
109+
new Map(readdirSync(dir).map((name) => [name, readFileSync(join(dir, name))] as const));
110+
111+
it("fills a deferred video in place with the frames a full extraction writes, byte for byte", async () => {
112+
const outputDir = join(root, "all", "clip");
113+
mkdirSync(outputDir, { recursive: true });
114+
const video: ExtractedFrames = {
115+
...deferredVideo("clip", 90),
116+
srcPath: sourcePath,
117+
outputDir,
118+
deferredRange: { startTime: 0, durationSeconds: 3, format: "jpg" },
119+
};
120+
const untouched = fullyExtracted("whole");
121+
untouched.framePaths.set(0, "/kept/frame_00001.jpg");
122+
const entries = [video, untouched];
123+
124+
const progress: Array<[number, number]> = [];
125+
await extractAllDeferredFrames(entries, {
126+
fps,
127+
cfg: {},
128+
onProgress: (done, total) => progress.push([done, total]),
129+
});
130+
expect(progress).toEqual([[90, 90]]);
131+
132+
expect(entries[0]).toBe(video);
133+
expect(video.deferredRange).toBeUndefined();
134+
expect(video.outputDir).toBe(outputDir);
135+
expect(video.framePaths.size).toBe(90);
136+
expect([...video.framePaths.values()].every((path) => existsSync(path))).toBe(true);
137+
expect(untouched.framePaths.get(0)).toBe("/kept/frame_00001.jpg");
138+
139+
const reference = await extractAllVideoFrames(
140+
[{ ...element("ref", 0, 3), src: sourcePath, mediaStart: 0 }],
141+
root,
142+
{ fps, outputDir: join(root, "all-reference") },
143+
);
144+
const expected = framesOf(reference.extracted[0]!.outputDir);
145+
const actual = framesOf(outputDir);
146+
expect([...actual.keys()]).toEqual([...expected.keys()]);
147+
for (const [name, bytes] of expected)
148+
expect(actual.get(name)?.equals(bytes), name).toBe(true);
149+
}, 60_000);
150+
});
98151
});

‎packages/producer/src/services/render/deferredVideoFrames.test.ts‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
import { describe, expect, it } from "bun:test";
22
import type { ExtractedFrames, VideoElement } from "@hyperframes/engine";
3-
import { shownDeferredFrameIndexes } from "./deferredVideoFrames.js";
3+
import { extractAllDeferredFrames, shownDeferredFrameIndexes } from "./deferredVideoFrames.js";
44

55
const metadata = {
66
durationSeconds: 4,
@@ -28,6 +28,11 @@ const deferredVideo = (videoId: string, totalFrames: number): ExtractedFrames =>
2828
deferredRange: { startTime: 1.5, durationSeconds: totalFrames / 30, format: "jpg" },
2929
});
3030

31+
const fullyExtracted = (videoId: string): ExtractedFrames => {
32+
const { deferredRange: _deferred, ...rest } = deferredVideo(videoId, 90);
33+
return rest;
34+
};
35+
3136
const element = (id: string, start: number, end: number): VideoElement => ({
3237
id,
3338
src: `${id}.mp4`,
@@ -57,3 +62,17 @@ describe("shownDeferredFrameIndexes", () => {
5762
expect(shown.has("later")).toBe(false);
5863
});
5964
});
65+
66+
describe("extractAllDeferredFrames", () => {
67+
it("reads nothing when no video is deferred", async () => {
68+
const whole = fullyExtracted("whole");
69+
await extractAllDeferredFrames([whole], {
70+
fps,
71+
cfg: {},
72+
sourceOf: () => {
73+
throw new Error("a fully extracted video must not be read again");
74+
},
75+
});
76+
expect(whole.framePaths.size).toBe(0);
77+
});
78+
});

‎packages/producer/src/services/render/deferredVideoFrames.ts‎

Lines changed: 59 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,12 @@
1-
import { join } from "node:path";
1+
import { mkdirSync, rmSync } from "node:fs";
2+
import { dirname, join } from "node:path";
23
import {
34
createFrameLookupTable,
45
type EngineConfig,
56
type ExtractedFrames,
67
extractVideoFramesRange,
78
type FrameRange,
9+
runVideoExtractionWithRetry,
810
type ShownBox,
911
type VideoElement,
1012
} from "@hyperframes/engine";
@@ -96,3 +98,59 @@ export async function extractDeferredFrames(input: {
9698
}),
9799
);
98100
}
101+
102+
/**
103+
* Fills every deferred video in place with the frames a full extraction writes, into the folder it
104+
* already has. For a route that cannot read frames per segment.
105+
*/
106+
export async function extractAllDeferredFrames(
107+
extracted: ExtractedFrames[],
108+
input: {
109+
fps: { num: number; den: number };
110+
cfg: Pick<EngineConfig, "ffmpegProcessTimeout">;
111+
signal?: AbortSignal;
112+
sourceOf?: (video: ExtractedFrames) => string;
113+
maxTransientRetries?: number;
114+
onProgress?: (framesDone: number, framesTotal: number) => void;
115+
},
116+
): Promise<void> {
117+
const framesTotal = extracted.reduce(
118+
(sum, video) => sum + (video.deferredRange ? video.totalFrames : 0),
119+
0,
120+
);
121+
let framesDone = 0;
122+
for (const video of extracted) {
123+
const range = video.deferredRange;
124+
if (!range) continue;
125+
const { result: filled } = await runVideoExtractionWithRetry(
126+
() =>
127+
extractVideoFramesRange(
128+
input.sourceOf?.(video) ?? video.srcPath,
129+
video.videoId,
130+
range.startTime,
131+
range.durationSeconds,
132+
{ fps: input.fps, outputDir: dirname(video.outputDir), format: range.format },
133+
input.signal,
134+
input.cfg,
135+
),
136+
{
137+
signal: input.signal,
138+
maxTransientRetries: input.maxTransientRetries,
139+
onRetry: () => {
140+
rmSync(video.outputDir, { recursive: true, force: true });
141+
mkdirSync(video.outputDir, { recursive: true });
142+
},
143+
},
144+
);
145+
if (filled.framePaths.size < video.totalFrames) {
146+
throw new Error(
147+
`[Render] video ${JSON.stringify(video.videoId)} has ${filled.framePaths.size} frames; ` +
148+
`it was expected to have ${video.totalFrames} frames.`,
149+
);
150+
}
151+
video.framePaths = filled.framePaths;
152+
delete video.deferredRange;
153+
framesDone += video.totalFrames;
154+
input.onProgress?.(framesDone, framesTotal);
155+
}
156+
}

‎packages/producer/src/services/render/observability.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,8 @@ export interface RenderCaptureObservability {
209209
/** Segmented capture only (Phase 2): current segment and retries so far. */
210210
segmentIndex?: number;
211211
segmentRetries?: number;
212+
videoFrameWindow?: "on" | "off";
213+
videoFrameWindowPeakBytes?: number;
212214
}
213215

214216
export interface RenderExtractionObservability {

‎packages/producer/src/services/render/segmentVideoFrames.test.ts‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,38 @@ describe("createSegmentVideoFrames", () => {
142142
expect(served).toContain(join(folder, "clip", "frame_00032.jpg"));
143143
});
144144

145+
it("forFrames extracts just the given frames ahead of time and release removes them", async () => {
146+
const { dir, served, extractor, segmentVideoFrames, page } = setup();
147+
const frames = await segmentVideoFrames.forFrames([0, 40, 74], "calibration");
148+
149+
const folder = join(dir, "segment-frames", "calibration");
150+
expect(extractor.calls[0]!.outputRoot).toBe(folder);
151+
expect(extractor.calls[0]!.frames).toEqual([0, 40, 74]);
152+
expect(readdirSync(join(folder, "clip")).sort()).toEqual([
153+
"frame_00001.jpg",
154+
"frame_00041.jpg",
155+
"frame_00075.jpg",
156+
]);
157+
158+
await frames.injector()!(page, 40 / 30);
159+
expect(served).toContain(join(folder, "clip", "frame_00041.jpg"));
160+
161+
frames.release();
162+
expect(existsSync(folder)).toBe(false);
163+
expect(() => frames.release()).not.toThrow();
164+
});
165+
166+
it("forFrames leaves no folder behind when its extraction fails", async () => {
167+
const { dir, segmentVideoFrames } = setup({
168+
extractFrames: async (input) => {
169+
mkdirSync(join(input.outputRoot, "clip"), { recursive: true });
170+
throw new Error("aborted");
171+
},
172+
});
173+
await expect(segmentVideoFrames.forFrames([0], "calibration")).rejects.toThrow("aborted");
174+
expect(existsSync(join(dir, "segment-frames", "calibration"))).toBe(false);
175+
});
176+
145177
it("release removes the folder and a second release does nothing", async () => {
146178
const { dir, workerFrames } = setup();
147179
await workerFrames.prepare(slice(2, 60, 75));

‎packages/producer/src/services/render/segmentVideoFrames.ts‎

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,11 @@ export interface SegmentWorkerFrames {
2828

2929
export interface SegmentVideoFrames {
3030
forWorker(): SegmentWorkerFrames;
31+
/** Extracts the given output frames into `segment-frames/<name>/` ahead of time; `release()` is idempotent. */
32+
forFrames(
33+
frames: readonly number[],
34+
name: string,
35+
): Promise<{ injector(): BeforeCaptureHook | null; release(): void }>;
3136
}
3237

3338
type FrameExtractor = typeof extractDeferredFrames;
@@ -53,8 +58,8 @@ export function createSegmentVideoFrames(input: {
5358
const sourceOf = input.sourceOf ?? ((video: ExtractedFrames) => video.srcPath);
5459
const fullyExtracted = input.extracted.filter((video) => !video.deferredRange);
5560

56-
/** Extracts the given output frames into `dir` and returns an injector over them plus the fully extracted videos. */
57-
const injectorFor = async (frames: number[], dir: string): Promise<BeforeCaptureHook | null> => {
61+
/** Extracts the given output frames into `dir` and returns a lookup over them plus the fully extracted videos. */
62+
const tableFor = async (frames: number[], dir: string) => {
5863
const sliced = await extractFrames({
5964
videos: input.videos,
6065
extracted: input.extracted,
@@ -70,13 +75,14 @@ export function createSegmentVideoFrames(input: {
7075
`it was expected to have ${video.totalFrames} frames.`,
7176
),
7277
});
73-
const table = createFrameLookupTable(
78+
return createFrameLookupTable(
7479
input.videos,
7580
[...fullyExtracted, ...sliced],
7681
resolveRenderFpsConfig(input.fps).value,
7782
);
78-
return createVideoFrameInjector(table, input.injectorOptions);
7983
};
84+
const injectorFor = async (frames: number[], dir: string): Promise<BeforeCaptureHook | null> =>
85+
createVideoFrameInjector(await tableFor(frames, dir), input.injectorOptions);
8086

8187
/**
8288
* The drawElement self-verify runs the hook at session open and after every
@@ -106,6 +112,19 @@ export function createSegmentVideoFrames(input: {
106112
);
107113

108114
return {
115+
async forFrames(frames, name) {
116+
const dir = join(input.rootDir, "segment-frames", name);
117+
const release = () => rmSync(dir, { recursive: true, force: true });
118+
try {
119+
release();
120+
mkdirSync(dir, { recursive: true });
121+
const table = await tableFor([...frames], dir);
122+
return { injector: () => createVideoFrameInjector(table, input.injectorOptions), release };
123+
} catch (error) {
124+
release();
125+
throw error;
126+
}
127+
},
109128
forWorker() {
110129
let injector: BeforeCaptureHook | null = null;
111130
let folder: string | null = null;

‎packages/producer/src/services/render/stages/captureSegmentedStage.test.ts‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1339,6 +1339,52 @@ describe("runCaptureSegmentedStage with segmentVideoFrames", () => {
13391339
expect(removed).toEqual([]);
13401340
});
13411341

1342+
it("with keepSegments, a windowed render keeps its segment files", async () => {
1343+
const removed: string[] = [];
1344+
const input = fakeStageInput({ totalFrames: 4 });
1345+
1346+
const result = await runCaptureSegmentedStage({
1347+
...input,
1348+
job: { ...input.job, config: { ...input.job.config, keepSegments: true } },
1349+
segmentFrames: 2,
1350+
segmentDir: join(fixtureRoot, "windowed-keep-segments"),
1351+
segmentVideoFrames: fakeVideoFrames([]).segmentVideoFrames,
1352+
deps: stageDeps({
1353+
spawnEncoder: loggingEncoder([]),
1354+
removeFile: (path: string) => removed.push(path),
1355+
}),
1356+
});
1357+
1358+
expect(result.success).toBe(true);
1359+
expect(removed).toEqual([]);
1360+
});
1361+
1362+
it("a segment file that will not delete is logged and does not fail the stage", async () => {
1363+
const attempted: string[] = [];
1364+
const warnings: string[] = [];
1365+
const segmentDir = join(fixtureRoot, "windowed-remove-throws");
1366+
const input = fakeStageInput({ totalFrames: 6 });
1367+
1368+
const result = await runCaptureSegmentedStage({
1369+
...input,
1370+
log: { ...input.log, warn: (message: string) => void warnings.push(message) },
1371+
segmentFrames: 2,
1372+
segmentDir,
1373+
segmentVideoFrames: fakeVideoFrames([]).segmentVideoFrames,
1374+
deps: stageDeps({
1375+
spawnEncoder: loggingEncoder([]),
1376+
removeFile: (path: string) => {
1377+
attempted.push(path);
1378+
if (path === segmentOutputPath(segmentDir, 0)) throw new Error("EPERM");
1379+
},
1380+
}),
1381+
});
1382+
1383+
expect(result.success).toBe(true);
1384+
expect(attempted).toEqual([0, 1, 2].map((i) => segmentOutputPath(segmentDir, i)));
1385+
expect(warnings).toEqual(["[Render] could not remove a joined segment file"]);
1386+
});
1387+
13421388
it("without segmentVideoFrames, segment files are kept", async () => {
13431389
const removed: string[] = [];
13441390

0 commit comments

Comments
 (0)