Repository navigation
fix(producer): graded video renders split into chunks that finish inside the worker limit - #5883
Conversation
…ide the worker limit The distributed planner priced every frame at a fixed cost per pixel, so a color-graded video, which costs about 10 s a frame on software GL, was planned into chunks that could never finish before the worker time limit. When the composition has color grading and the plan sizes chunks from the worker limit, the planner now captures the renderer's calibration frames in a session set up like a chunk worker's and prices frames at the measured p95 when it is higher than the estimate. Compositions without grading never pay for the measurement and keep their current plan.
Edit accuracy: accurate 271 (base branch 271), smooth 192 of thoseThe gate passes. Quarantined, measured but not gated (0) Unstable (3)
|
vanceingalls
left a comment
There was a problem hiding this comment.
Review at cf4e772a.
Strengths
plan.ts:427-429: frames are priced atmax(per-pixel estimate, measured), so the measured cost can only add chunks. It never lowers the price below the existing estimate, and an ungraded plan is unchanged by construction.chunkSizingUsesFrameCost(plan.ts:448) lifts the oldpinsChunkingcondition into one predicate.resolveMaxParallelChunksand the measure-or-not decision both use it, so they can't disagree.- The measuring session mirrors a chunk worker. It uses the same
buildChunkCaptureOptions,browserGpuMode: "software"(plan.ts:1086), and the injected video frames. It extracts the deferred frames for exactly the indicesselectCaptureCalibrationFrames(totalFrames)returns, which is the functionmeasureCaptureCostFromSessioncalls too, so no sampled frame is drawn without its video.
Review focus
- Sample size and outliers: 5 frames (0, 25%, 50%, 75%, last). The p95 of 5 is the slowest sample (
captureCost.ts:222), so a cold or contended outlier biases the price up. That's the safe direction: more chunks, never a chunk that runs too long. Browser start and shader compile happen ininitializeSession, before the timed samples, and the body's 10.54 s next to the ~17 s setup is consistent with that. Static-frame dedup is bypassed during calibration (captureCost.ts:272-273), so a sample can't read near 0 ms. - Chunk bounds: no zero-frame or runaway chunks.
wideis clamped to ≥1 (:515). If even one frame doesn't fit the margin,maxFramesPerChunk < 1makesneededInfinity, which raises the typedChunkExceedsWorkerTimeoutError(:522-529). The 256-chunk ceiling also raises that error rather than producing an oversized chunk, andresolveChunkPlanstill asserts positive integers. At 10.5 s/frame with a 900 s limit, a chunk holds 58 frames, so the refusal starts at about 14.8k frames (about 10 min at 24 fps). - Timeout margin: unchanged at 0.75 (
:424), and the measured chunks (160-178 s against a 218 s prediction) sit comfortably inside it. Chunk workers run several capture workers (calculateOptimalWorkers) while the planner measures one session, so the prediction is conservative. - Non-graded renders: with
measureFrameCostfalse, the thirdallSettledslot isundefinedand the chunk plan is the sameresolveChunkPlan(..., maxParallel, ...)as before, just computed later. Also,chunkFrames.tsis a faithful move. I diffed the removedrenderChunk.tslines against the new file with comments stripped. The only changes areslice→frames: Iterable<number>(the caller expands the slice) and the capture-options object turned intobuildChunkCaptureOptions. That readsencoder.quality, which is whatjob.config.qualitywas (buildSyntheticRenderJob({ quality: encoder.quality })).
Important (non-blocking)
- When measurement fails (
plan.ts:1431), a rejected measurement now fails the whole plan. That covers Chrome launch, ffmpeg extraction of the sampled frames, or a capture timeout under the shortened calibrationprotocolTimeout. The measurement is sizing advice, and the per-pixel price is still available. Falling back to it, or to a conservative floor, with alog.warnwould keep a transient plan-side flake from failing a render the chunks might have finished. Failing fast is defensible, since an under-sized graded plan times out anyway, so I'm not blocking. If it stays, a sentence in the code saying it's deliberate would help.
Nits
2. planNeedsFrameCostMeasurement (:444) is a static regex on the compiled HTML. Grading added at runtime by script isn't seen, and that render keeps the old per-pixel plan. The attribute appearing in text or comments only costs an unnecessary measurement. Both are fine for now.
3. The measurement runs alongside the audio mix and prepublish on the same box. On a small plan instance that contention inflates the p95. That's the safe direction, but it's worth knowing when reading a surprisingly high [plan] measured frame cost.
CI at cf4e772a: all 8 required checks passed. The first CI run on this head (run 39264487869, 13:56Z) ran Build, Test, Test: runtime contract and Typecheck, plus Producer unit and integration tests, and all succeeded. Windows render verification run 39264487857 passed Render on windows-latest and Tests on windows-latest. A later re-trigger at 14:18Z on the same SHA saw no changed paths and skipped them. regression and Semantic PR title are green.
Verdict: APPROVE
Reasoning: The measured price can only make chunks smaller, chunk-count bounds and the timeout refusal are intact, non-graded plans are unchanged, and the extracted module is a faithful move. Required CI ran and passed at this exact head.
— Via
What changes for users
A distributed render of a composition with a color-graded
<video>now gets chunks small enough to finish inside the worker time limit with default flags. Before, the planner priced every frame at a fixed cost per pixel (about 0.23 s for a 1080x1920 frame). A graded frame costs about 10 s on software GL, so a 45 s graded render was planned as 4 chunks of 270 frames, each needing about 2,800 s against a 900 s limit. Every chunk timed out and was retried until the execution gave up.Compositions without color grading are planned exactly as before and pay nothing extra.
How
resolveMaxParallelChunkstakes an optional measured seconds per frame and prices frames at the larger of it and the per-pixel estimate. Everything downstream (the one-wave width, the fit-the-limit split, theCHUNK_EXCEEDS_WORKER_TIMEOUTrefusal) uses that price.data-color-gradingand the plan sizes chunks from the worker limit or concurrency (no explicitchunkSize,targetChunkFramesormaxParallelChunks).measureCaptureCostFromSession: 5 frames spread over the timeline, p95), run in a session set up like a chunk worker's: same capture options (buildChunkCaptureOptions, now shared withrenderChunk), same software-GL engine config, and the same video frames injected. planV2 defers video extraction to chunks, so the planner extracts just the sampled frames first; without them a graded video has no frame to draw and would measure cheap.chunkFrames.tsis a move, not new logic: the chunk's frame and capture-option helpers leftrenderChunk.tsso the planner can use them without an import cycle (renderChunkimportsplanV2, which importsplan).renderChunk.tsre-exports what it exported before.Interaction with the chunk cap
With default flags the measured price flows into the fit-the-limit split, which may exceed the one-wave width (it warns and runs in waves, as today). An explicit
--chunk-sizestill pins chunking and is still capped at one wave (16 when the worker concurrency is unknown); this PR does not change that.Measured
Devbox (Linux, software GL, same flags as a chunk worker, a shared and loaded box), planV2 with a 900 s worker limit and 8 workers, 1080x1920 at 24 fps. The graded and ungraded variants differ only by the
data-color-gradingattribute on the video. Chunks rendered one after another withrenderChunkV2.Tests
plan.test.ts: a measured 10.5 s frame turns a 1,080-frame 1080x1920 plan whose chunks would each need far over 900 s into chunks that fit 75% of the limit, and a cheap measured frame keeps the per-pixel plan;plan()measures a graded fixture (mocked measurement) and splits it, and never measures the ungraded fixture. Both fail without the change (checked by removing the measured price from the sizing and by skipping the re-size in the planner).renderChunk.test.ts: the deferred-frame extraction tests now pass frame lists.Not in this PR
The grading shader itself is the cost: it compiles every effect into one program and software GL pays for the disabled ones. Compiling only the enabled effects is a separate change.