Skip to content

Commit 2651398

Browse files
fix(producer): non-DE parallel-stream router silently refused on macOS multi-worker renders (#3886)
* fix(producer): read useDrawElement post-clamp for the parallel-stream router The non-DE parallel-streaming router read cfg.useDrawElement before the default-on drawElement clamp ran, so on every macOS/GPU-default host with multi-worker renders it saw a stale true and refused to stream even though the render was about to fall back to non-DE screenshot capture anyway. * refactor: trim fix-narrating comments to durable invariants Adversarial simplify pass flagged two comments that described the diff itself (what used to happen, what changed) instead of the invariant that matters going forward. Kept only the forward-looking reasoning. * fix(producer): capture probe console before closing on the DE clamp path; add regression coverage - Extract the default-on drawElement clamp condition into a named, exported, unit-tested pure predicate (shouldClampDefaultDrawElement), matching this file's existing convention for the sibling router predicate. - Preserve lastBrowserConsole when the clamp closes a drawElement probe session, matching every other probe-close site in this function (an adversarial review caught this: on the auto-worker-calibration path, a render that throws after the clamp would report the calibration browser's console instead of the probe's). - Add tests proving the router requires the clamp's post-clamp output, not its input — the exact ordering bug PRINFRA-689 was about. * test: drop internal link and fix-narrating comment from new regression test A test comment I added carried an internal Slack archive link (this is a public repo) and described the diff instead of the invariant — the user caught it in review after my own simplify pass had already run and missed it, since that pass ran before this test file existed. * refactor: collapse duplicated ordering-invariant comment into one place Two comment blocks explained the same 'why deParallelStreamForced alone is safe here' reasoning, one forward-referencing the other. Stated once.
1 parent 42a3e7a commit 2651398

2 files changed

Lines changed: 178 additions & 39 deletions

File tree

‎packages/producer/src/services/renderOrchestrator.test.ts‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ import {
4747
mergeWorkerInitObservability,
4848
resolveCompositionElementCount,
4949
resolveDeShortBand,
50+
shouldClampDefaultDrawElement,
5051
shouldPreferParallelDrawElement,
5152
shouldPreferSingleWorkerDrawElement,
5253
shouldStreamParallelCapture,
@@ -2841,6 +2842,94 @@ describe("shouldStreamParallelCapture (non-DE parallel streaming router)", () =>
28412842
});
28422843
});
28432844

2845+
describe("shouldClampDefaultDrawElement (default-on drawElement clamp)", () => {
2846+
const unverifiedParallel = {
2847+
useDrawElement: true,
2848+
fastCaptureExplicitOptIn: false,
2849+
useStreamingEncode: false,
2850+
workerCount: 2,
2851+
deParallelStreamVerified: false,
2852+
};
2853+
2854+
it("clamps default-on drawElement for unverified multi-worker capture", () => {
2855+
expect(shouldClampDefaultDrawElement(unverifiedParallel)).toBe(true);
2856+
});
2857+
2858+
it("leaves useDrawElement alone once it is already false", () => {
2859+
expect(shouldClampDefaultDrawElement({ ...unverifiedParallel, useDrawElement: false })).toBe(
2860+
false,
2861+
);
2862+
});
2863+
2864+
it("an explicit opt-in overrides the clamp", () => {
2865+
expect(
2866+
shouldClampDefaultDrawElement({ ...unverifiedParallel, fastCaptureExplicitOptIn: true }),
2867+
).toBe(false);
2868+
});
2869+
2870+
it("does not clamp a verified multi-worker streaming render", () => {
2871+
expect(
2872+
shouldClampDefaultDrawElement({ ...unverifiedParallel, deParallelStreamVerified: true }),
2873+
).toBe(false);
2874+
});
2875+
2876+
it("clamps a single-worker render with streaming off (the disk-path case)", () => {
2877+
expect(shouldClampDefaultDrawElement({ ...unverifiedParallel, workerCount: 1 })).toBe(true);
2878+
});
2879+
2880+
it("leaves a single-worker streaming render unclamped (self-verified by the drain)", () => {
2881+
expect(
2882+
shouldClampDefaultDrawElement({
2883+
...unverifiedParallel,
2884+
workerCount: 1,
2885+
useStreamingEncode: true,
2886+
}),
2887+
).toBe(false);
2888+
});
2889+
});
2890+
2891+
describe("default-on drawElement clamp feeds the non-DE parallel-stream router", () => {
2892+
// The router requires the drawElement clamp's OUTPUT, not its input. Both
2893+
// predicates below are individually correct; a caller wiring them together
2894+
// in the wrong order still reproduces the bug these two tests bound.
2895+
const macOsDefaultOnMultiWorker = {
2896+
useDrawElement: true,
2897+
fastCaptureExplicitOptIn: false,
2898+
useStreamingEncode: false,
2899+
workerCount: 2,
2900+
deParallelStreamVerified: false,
2901+
};
2902+
2903+
it("routes once the clamp's post-clamp value feeds the router", () => {
2904+
const clamped = shouldClampDefaultDrawElement(macOsDefaultOnMultiWorker);
2905+
const postClampUseDrawElement = clamped ? false : macOsDefaultOnMultiWorker.useDrawElement;
2906+
2907+
expect(
2908+
shouldStreamParallelCapture({
2909+
routerEnabled: true,
2910+
workerCount: macOsDefaultOnMultiWorker.workerCount,
2911+
useDrawElement: postClampUseDrawElement,
2912+
outputFormat: "mp4",
2913+
streamingOk: true,
2914+
layeredOrEffectRoute: false,
2915+
}),
2916+
).toBe(true);
2917+
});
2918+
2919+
it("never routes if the router instead reads the PRE-clamp value (the bug)", () => {
2920+
expect(
2921+
shouldStreamParallelCapture({
2922+
routerEnabled: true,
2923+
workerCount: macOsDefaultOnMultiWorker.workerCount,
2924+
useDrawElement: macOsDefaultOnMultiWorker.useDrawElement,
2925+
outputFormat: "mp4",
2926+
streamingOk: true,
2927+
layeredOrEffectRoute: false,
2928+
}),
2929+
).toBe(false);
2930+
});
2931+
});
2932+
28442933
describe("closeOrphanedProbeForRetry (probe cleanup before verify-triggered retry)", () => {
28452934
// Enough of a CaptureSession stand-in to exercise the closer path — the
28462935
// helper never inspects the object; it just hands it to the injected closer.

‎packages/producer/src/services/renderOrchestrator.ts‎

Lines changed: 89 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -2008,6 +2008,33 @@ export function shouldStreamParallelCapture(args: {
20082008
);
20092009
}
20102010

2011+
/**
2012+
* Default-on drawElement clamp: is this render's runtime self-verification
2013+
* net absent, so `useDrawElement` must fall back to the screenshot/beginframe
2014+
* baseline? The disk path and unverified parallel capture ship frames no
2015+
* drain verifies — see the call site's comment on why this clamp's own
2016+
* outcome is independent of running before or after
2017+
* {@link shouldStreamParallelCapture} in the caller. Pure; exported for tests.
2018+
*/
2019+
export function shouldClampDefaultDrawElement(args: {
2020+
useDrawElement: boolean;
2021+
/** PRODUCER_EXPERIMENTAL_FAST_CAPTURE === "true" — explicit opt-in always wins. */
2022+
fastCaptureExplicitOptIn: boolean;
2023+
useStreamingEncode: boolean;
2024+
workerCount: number;
2025+
/** (deParallelStreamForced || HF_DE_PARALLEL_STREAM === "true") &&
2026+
* useStreamingEncode && workerCount > 1 — the multi-worker streaming
2027+
* self-verification net is present. */
2028+
deParallelStreamVerified: boolean;
2029+
}): boolean {
2030+
return (
2031+
args.useDrawElement &&
2032+
!args.fastCaptureExplicitOptIn &&
2033+
(!args.useStreamingEncode || args.workerCount > 1) &&
2034+
!args.deParallelStreamVerified
2035+
);
2036+
}
2037+
20112038
export function resolveCaptureForceScreenshotForPageSideCompositing(args: {
20122039
forceScreenshot: boolean;
20132040
usePageSideCompositing: boolean;
@@ -3231,9 +3258,66 @@ async function executeRenderPipeline(input: {
32313258
deParallelRouter: deParallelRouter ?? "none",
32323259
});
32333260

3261+
// Streaming encode pipes captured frames through ffmpeg's stdin to produce
3262+
// a single video file. Keep the default enabled for sequential capture, but
3263+
// let auto-parallel renders use disk frames: the current ordered streaming
3264+
// writer would otherwise stall later workers behind earlier frame ranges.
3265+
// png-sequence has no encoded video output, so streaming is always bypassed.
3266+
let useStreamingEncode = shouldUseStreamingEncode(
3267+
cfg,
3268+
outputFormat,
3269+
workerCount,
3270+
job.duration,
3271+
deParallelStreamForced,
3272+
);
3273+
// Default-on drawElement is only safe where the runtime self-verification
3274+
// net actually runs: the single-worker streaming worker-encode drain, or
3275+
// (HF_DE_PARALLEL_STREAM) a verified multi-worker streaming render. The
3276+
// disk path and unverified parallel capture ship frames no drain
3277+
// verifies, so they clamp to the screenshot baseline unless drawElement
3278+
// was explicitly opted into.
3279+
//
3280+
// This must run before the non-DE parallel-streaming router below, which
3281+
// needs `cfg.useDrawElement` post-clamp to know whether this render will
3282+
// actually use non-DE capture. The router's own force flag
3283+
// (`captureParallelStreamForced`) requires `!useDrawElement`, so it can
3284+
// only be true once this clamp has already fired — meaning the router's
3285+
// flag is always false while this clamp's guard is still evaluating,
3286+
// and `deParallelStreamForced` above is the only flag that can affect it.
3287+
const deParallelStreamVerified =
3288+
(deParallelStreamForced || process.env.HF_DE_PARALLEL_STREAM === "true") &&
3289+
useStreamingEncode &&
3290+
workerCount > 1;
3291+
if (
3292+
shouldClampDefaultDrawElement({
3293+
useDrawElement: cfg.useDrawElement,
3294+
fastCaptureExplicitOptIn: process.env.PRODUCER_EXPERIMENTAL_FAST_CAPTURE === "true",
3295+
useStreamingEncode,
3296+
workerCount,
3297+
deParallelStreamVerified,
3298+
})
3299+
) {
3300+
cfg.useDrawElement = false;
3301+
deClampReason = workerCount > 1 ? "parallel" : "disk_path";
3302+
log.info(
3303+
"[Render] Fast capture: default-on drawElement disabled for this render — " +
3304+
(workerCount > 1 ? "parallel capture" : "the disk capture path") +
3305+
" has no runtime self-verification. Set PRODUCER_EXPERIMENTAL_FAST_CAPTURE=true to override.",
3306+
);
3307+
// The probe session already initialized in drawElement mode (canvas
3308+
// injected); it must not be reused by the unverified path.
3309+
if (probeSession && probeSession.captureMode === "drawelement") {
3310+
lastBrowserConsole = probeSession.browserConsoleBuffer;
3311+
await closeCaptureSession(probeSession);
3312+
probeSession = null;
3313+
}
3314+
}
3315+
32343316
// Non-DE parallel-streaming router — see shouldStreamParallelCapture.
32353317
// Mutually exclusive with the DE inversion/router above by construction
32363318
// (both DE predicates require useDrawElement; this requires its negation).
3319+
// Reads `cfg.useDrawElement` after the clamp above, so it sees the
3320+
// capture mode this render will actually use.
32373321
const captureParallelStreamRouterEnabled = process.env.HF_CAPTURE_PARALLEL_STREAM === "true";
32383322
const captureParallelStreamArgs = {
32393323
workerCount,
@@ -3282,12 +3366,11 @@ async function executeRenderPipeline(input: {
32823366
probeSession = null;
32833367
}
32843368

3285-
// Streaming encode pipes captured frames through ffmpeg's stdin to produce
3286-
// a single video file. Keep the default enabled for sequential capture, but
3287-
// let auto-parallel renders use disk frames: the current ordered streaming
3288-
// writer would otherwise stall later workers behind earlier frame ranges.
3289-
// png-sequence has no encoded video output, so streaming is always bypassed.
3290-
let useStreamingEncode = shouldUseStreamingEncode(
3369+
// Re-resolve now that the non-DE router above may have forced streaming
3370+
// on for this multi-worker render (same formula as the early value above,
3371+
// now including `captureParallelStreamForced`). This is the value the
3372+
// rest of the pipeline (encode/writer selection, logging) uses.
3373+
useStreamingEncode = shouldUseStreamingEncode(
32913374
cfg,
32923375
outputFormat,
32933376
workerCount,
@@ -3302,39 +3385,6 @@ async function executeRenderPipeline(input: {
33023385
durationSeconds: job.duration,
33033386
maxDurationSeconds: cfg.streamingEncodeMaxDurationSeconds,
33043387
});
3305-
// Default-on drawElement is only safe where the runtime self-verification
3306-
// net actually runs: the single-worker streaming worker-encode drain. The
3307-
// disk path (png-sequence / over the streaming duration cap) and parallel
3308-
// capture ship frames no drain verifies — route those renders to the
3309-
// screenshot baseline unless drawElement was explicitly opted into.
3310-
// HF_DE_PARALLEL_STREAM: multi-worker STREAMING renders now carry the
3311-
// full drain-time self-verification (per-worker ground truth + the shared
3312-
// drain guard), so the confinement rule is satisfied and the parallel
3313-
// clamp does not apply. The disk path stays clamped.
3314-
const deParallelStreamVerified =
3315-
(deParallelStreamForced || process.env.HF_DE_PARALLEL_STREAM === "true") &&
3316-
useStreamingEncode &&
3317-
workerCount > 1;
3318-
if (
3319-
cfg.useDrawElement &&
3320-
process.env.PRODUCER_EXPERIMENTAL_FAST_CAPTURE !== "true" &&
3321-
(!useStreamingEncode || workerCount > 1) &&
3322-
!deParallelStreamVerified
3323-
) {
3324-
cfg.useDrawElement = false;
3325-
deClampReason = workerCount > 1 ? "parallel" : "disk_path";
3326-
log.info(
3327-
"[Render] Fast capture: default-on drawElement disabled for this render — " +
3328-
(workerCount > 1 ? "parallel capture" : "the disk capture path") +
3329-
" has no runtime self-verification. Set PRODUCER_EXPERIMENTAL_FAST_CAPTURE=true to override.",
3330-
);
3331-
// The probe session already initialized in drawElement mode (canvas
3332-
// injected); it must not be reused by the unverified path.
3333-
if (probeSession && probeSession.captureMode === "drawelement") {
3334-
await closeCaptureSession(probeSession);
3335-
probeSession = null;
3336-
}
3337-
}
33383388

33393389
// png-sequence is "no container" — outputPath is treated as a directory and
33403390
// the encode/mux/faststart stages are skipped entirely. The empty extension

0 commit comments

Comments
 (0)