feat(producer): a cloud plan can leave its audio to be mixed beside its chunks - #5988

Reviewed at 4c142905. Required CI is green and there are no other reviews yet.

What holds

• The stitch is sound. The audio track gets a new id, mvhd next_track_ID moves past it (TRACK_ID_AT offsets match v0 and v1 boxes), and the audio's samples sit ahead of every chunk's. withId only ever writes into the copy withDuration makes, so the template moov is never mutated.
• Nothing a render does today changes. No caller passes deferAudio. s3Stitch.ts:141 still calls buildStitchedHeader(layouts) with no audio, and cfr or a non-mp4 format keeps the plan's own mix.
• Byte-identity holds for <audio> and static audible video. Locally (ffmpeg 6.0), the PR's buildPlanAudio test passes. For a <video src="clip.mp4" data-has-audio="true"> composition, buildPlanAudio also writes the same bytes as the plan's audio.m4a.

Blocker: buildPlanAudio drops the sound of a video the static parse can't see
The plan's mix includes audio that runExtractVideosStage (plan.ts:1428) appends to composition.audios through appendAutoDetectedVideoAudio (extractVideosStage.ts:544). That append is the only way a <video data-has-audio="true"> gets into the mix when the static parse misses it. The common case is a src assigned from the scene script, a pattern the compiler explicitly supports (core/src/compiler/mediaRenderIds.ts:48). The browser probe then adds the video (probeStage.ts:610), but parseAudioElements never saw a src. buildPlanAudio (plan.ts:1741) runs compile and probe, then mixes at :1756 without that step, so those tracks are missing.

Repro at this head. The composition is <video id="v" data-has-audio="true" data-start="0" data-duration="1"> plus <script>document.getElementById("v").src = "clip.mp4"</script>, where clip.mp4 has an AAC tone:

• buildLocalExecutionPlan(…, { chunksCarryAudio: true }) writes audio.m4a.
• buildPlanAudio(…) returns { hasAudio: false } and writes nothing.

Once the next PR wires deferAudio, that render ships silent where today's plan has sound. The fix belongs here, because the function's contract is "the mix … that plan would have made". A probe-only pass would do it: ffprobe the audible, non-hidden composition.videos sources and feed appendAutoDetectedVideoAudio, without extracting frames. Alternatively, share the plan's post-extraction audio list. A buildPlanAudio-equals-plan test on a script-assigned src would pin the fix.

Nit: the "carries its own audio" refusal misses chunk 0
sourceTracks (mp4Stitch.ts:359) checks tracks.at(-1), the match for the separate audio's plan slot. When chunk 0 carries audio, its soun joins the plan, and every chunk's audio matches that slot instead, so nothing throws. With a separate file, buildStitchedHeader([chunk0(vide+soun), chunk1(vide+soun)], mix) returns a moov with tracks vide, soun, soun. This is unreachable from the plan, since separateAudio and chunksCarryAudio are exclusive, but the guard and its test only cover chunk ≥1. Checking chunkPlan.some(t => t.handler === "soun") up front closes it.

Verdict: REQUEST CHANGES
Reasoning: buildPlanAudio doesn't make the mix the plan makes for audible videos with a script-assigned src, which would silently drop their sound once it is wired. The rest is sound.

— Via

Re-reviewed at 73801e70, covering the delta from 4c142905 (plan.ts, mp4Stitch.ts and their tests).

The blocker is closed

• buildPlanAudio now runs extractPlanVideos before mixing, the same stage the plan runs. So appendAutoDetectedVideoAudio adds the sound of a <video data-has-audio> whose src a script sets. The plan's own call is now the shared helper, with the same arguments and the same failureToEnforce and assertVideoExtractionSucceeded checks.
• deferRangeExtraction: true is right for an audio-only pass. A deferred long CFR window still carries the probed metadata (videoFrameExtractor.ts:2259-2305), so its hasAudio reaches the append without extracting frames.
• Locally at this head (ffmpeg 6.0), the two buildPlanAudio tests pass and the script-src one is byte-identical to the plan's audio.m4a. Removing the extractPlanVideos call turns the script-src test red.

The nit is closed. The refusal now checks each chunk's own tracks for soun before matching, so chunk 0 is covered too. mp4Stitch.test.ts passes 13/13. Exempting chunk 0 from the check turns the new chunk 0 case red.

Nit (cost, not correctness): the windows extraction won't defer (VFR, HDR transforms, final-frame-only, or a clip of about a second or less) are fully extracted again in buildPlanAudio, only to be discarded with workDir. This doesn't touch the 4K takes this targets (long CFR). A VFR-heavy film would pay the frame extraction twice, once on the plan's critical path and once beside the chunks. A metadata-only pass would avoid that if it ever shows up.

CI: every required check is green at 73801e70.

Verdict: APPROVE
Reasoning: buildPlanAudio now goes through the same extraction the plan does, so a deferred plan's audio matches today's for script-set video sources. The stitch refuses audio-carrying chunks at every index.

— Via