Repository navigation
perf(producer): assemble in one pass and keep its disk use under one output copy - #5768
Conversation
…output copy The distributed assembler joined the chunks into a full-size file, muxed the audio into a second one and rewrote that again for faststart, so local disk peaked near four copies of the video and a 30-minute 1080p render came close to Lambda's 10 GB /tmp. The join, audio mux and faststart now run as one ffmpeg pass over the concat demuxer; the audio pad/trim takes its frame count from the chunks. Video and audio packets are unchanged. Assemble and the Lambda assemble handler log per-stage timings.
… final write Keeps assemble under the complexity gate and drops an orphaned doc comment in chunkEncoder. No behaviour change.
Edit accuracy: accurate 271 (base branch 271), smooth 187 of thoseThe gate passes. Quarantined, measured but not gated (0) Unstable (2)
|
jrusso1020
left a comment
jrusso1020
left a comment
There was a problem hiding this comment.
Approving at be87872b. The second commit splits assemble into readAssemblePlan, reencodeCfr and writeJoined with the same arguments in the same order (plan check, then the cfr and h265 guards, before any ffmpeg runs).
Same join, one pass. main already joined the chunks with -r fps -f concat -safe 0 -i list -c copy and '\'' quote escaping. concatDemuxerInput produces the same arguments, and a single chunk is still read directly. The mux pass sets the output -r, provenance tags and +faststart for mp4. The no-audio join does the same, and the old applyFaststart pass was just a copy for mov/webm, so dropping it changes nothing there. "Every write sets the output -r" holds on all three write paths: the cfr re-encode, the join and the mux. I also checked by hand with ffmpeg: 3-chunk VP9 yuva420p webm and ProRes 4444 mov chunks, in a directory whose name contains a space and a single quote. The old two-step path and this single pass produced identical per-stream framemd5 for video and audio. The webm kept ALPHA_MODE=1, and the frame rates stayed 30/1.
The frame-count sum equals the old probe. The old code probed the joined file, whose rate the input -r pins to the plan fps, and a -c copy concat neither drops nor duplicates packets. For example, 7 + 13 + 11 frames of webm at 30000/1001 join into 31 packets.
The rename can't cross filesystems. workDir is mkdtempSync(\${outputPath}.assemble-work-`), right beside the output, and the finallyremoves the temp file when ffmpeg fails. That's an improvement onmain, where faststart wrote straight to outputPath. The only other caller of muxVideoWithAudio, assembleStage.ts, doesn't pass videoInputArgs, so it keeps -i videoPath`.
Non-blocking
padOrTrimAudioToVideoFrameCountis givenvideoPath: videoSources[0]but probes every chunk.sanitizeProbeFailureonly hides the paths it was given, so a corrupt chunk 3 shows its full path in the error.- The per-chunk probes run through an uncapped
Promise.all. On webm,nb_framesis N/A, so each probe falls back to counting packets across the whole chunk. With a largemaxParallelChunks, that's N full reads at once on a memory-limited Lambda. A small concurrency cap would avoid it. - The packet-identity test covers mp4 with audio only. There's no equivalent test for webm, mov or single chunk plus audio.
CI is green. No live run, and no Lambda run.
— Rames
What changed
The distributed assembler used to write the video three times on local disk: concat the chunks into a full-size file, mux the audio into a second one, then rewrite that again for faststart (a no-op rewrite, since the mux already wrote
+faststart). While the last pass ran, disk held the chunks plus three full copies of the output. On Lambda's 10 GB/tmpthat caps a 1080p render at roughly 35-40 minutes at typical bitrates.Now the join, the audio mux and faststart run as one ffmpeg pass that reads the chunks through the concat demuxer, writes into the private work dir and renames into place. Disk peaks at the chunks plus one output copy.
assemble.ts: one pass for mp4/mov/webm, with and without audio. A single chunk is still read directly (not through a one-entry concat list), and every write still sets the output-r.muxVideoWithAudiogains avideoInputArgsoption so the mux can take a concat-demuxer input. It is the only engine change.cfrre-encode reads the chunks directly instead of a joined intermediate. Its output is the same.[assemble] stage timings(cfrMs,audioMs,joinMs,totalMs). The v2 Lambda assemble handler logs anassemble_stagesevent (planMs,chunksMs,assembleMs,uploadMs).What I measured
Fixture: a 30-minute 1080p30 H.264 video cut into 16 chunks (0.90 GB) plus 30 minutes of AAC audio. Real
assemble()frommainand from this branch, alternated, page cache dropped before each run. Ran on a Linux x86 machine with the same ffmpeg 7.0.2 static build the Lambda image ships. A wrapper ffmpeg logged each pass's wall time and the disk total at its exit.ffmpeg -c copy -f framemd5per stream).moov/udta/meta, which is 160 bytes smaller.main's file also carriedmajor_brand,minor_versionandcompatible_brandsasmdtakeys, copied from the joined intermediate. This branch writes onlyhf_renderer,hf_versionandencoder. Atom layout is otherwise the same, and ffprobe format and stream fields match apart from size and bit rate.keeps local disk under the chunks plus one copy of the output. It pointsHYPERFRAMES_FFMPEG_PATHat a wrapper that records disk use after every ffmpeg exit. Onmainit fails: 284 KB above the inputs for an 88 KB output. With this change it uses 100 KB.writes the same video and audio packets as separate concat, mux and faststart passes. It rebuilds the old path from the engine's ownmuxVideoWithAudio+applyFaststart+padOrTrimAudioToVideoFrameCountand compares per-packetframemd5. Counting one frame short in the chunk sum makes it fail on the audio packets.assemble.test.ts: 18/18, three runs in a row. Also green:audioPadTrim(38 + 2 integration),planV2(62),aws-lambdahandler.test.ts(16), enginechunkEncoder.test.ts, andtscfor producer, engine and aws-lambda. oxlint clean.What I did NOT exercise
cfr: truestill writes the re-encode and then the muxed output (two copies). It is opt-in and off by default.