Skip to content

perf(producer): overlap the plan step's audio work and hash files once - #5777

Merged
miguel-heygen merged 2 commits into
mainfrom
cloudrender/plan-overlap
Oct 17, 2026
Merged

miguel-heygen merged 2 commits into
mainfrom
cloudrender/plan-overlap

Conversation

@miguel-heygen

Copy link
Copy Markdown
Collaborator

On a 30-minute 1080p30 project the distributed plan step on AWS Lambda drops from about 112 s to about 88 s. The audio mix now runs alongside hashing and uploading the project files, instead of in front of them, and each file is hashed once instead of up to four times. Plan artifacts and final videos are byte-identical to main.

What changed

  • The audio mix overlaps the rest of the plan. Only the assembler reads the mixed audio. The plan used to run the full-length AAC encode first, then hash every file, then upload. Now buildLocalExecutionPlan moves compiled/ and the video trees into their final places first. It then starts the audio stage and, in parallel, streams a hash of those trees. planV2WithPublisher adds a whileAudioMixes callback that uploads the compiled/ blobs in that same window, using the existing 32-slot pool. artifactTargets ships every compiled/ file to every chunk, so each pre-uploaded blob is in the manifest; the new test checks this.
    • The audio stage resolves sources from the final compiled/ dir. Its contents are the same copy of the project, so the audio input bytes and ffmpeg args don't change.
  • Hash once. A PlanFileDigests memo (sha256 by absolute path) is filled by that streamed prehash. It is then read by freezePlan, by the publication's plan-hash recompute, and by the per-artifact digests.
    • Freeze used to readFileSync each whole file, including the full 902 MB video. It now streams.
    • The S3 upload no longer re-hashes the local file before the PUT: the PUT already sends ChecksumSHA256, and S3 rejects bytes that don't match.
    • S3PlanV2ArtifactPublisher.putBlob returns early for a digest it already published, so blobs uploaded during the audio window are not checked again with a HEAD afterwards.
    • Chunks still verify every artifact they download.
  • Stage timers. The plan logs one line per stage with its wall time ([plan] stage {"stage","ms"}): copy, compile, probe, extract, audio, prehash, prepublish, freeze, publication, upload, manifest. The Lambda handler adds plan_stage lines for download and untar.
    • Before this, the plan's 86 s after compile had no log lines at all.

What I measured

All runs used the same 30-min fixture: a 902 MB H.264 background, 29 MB of AAC, and a GSAP title, 16 chunks. Everything ran on one 10 GB x86 Lambda stack, swapping only the function code. Each plan was a direct invoke of the plan action.

  • Main was measured with bench-only stage timers added and no other change.
  • This PR's first commit was measured three times, and the final head twice more after the simplify pass.
stage (ms) main run 1 main run 2 PR runs (3 + 2)
download + untar 17,181 19,294 14,933-18,370
copy + compile + probe + extract 4,745 4,221 4,613-5,868
audio (prehash and prepublish run inside it) 65,599 65,382 62,719-66,006
prehash, overlapped - - 2,732-3,195
prepublish (compiled/ upload), overlapped - - 9,790-9,936
freeze 3,512 3,453 120-125
publication 6,667 6,648 140-231
upload 13,623 13,048 446-471
Lambda Duration 111.8 s 112.5 s 85.8-90.5 s
  • The tail after audio went from about 23.5 s to about 0.8 s. Peak memory went from about 5.1 GB to about 4.2 GB.
  • The audio encode itself (about 64 s on Lambda) is now the critical path of the plan.

Byte identity:

  • Full renders. The same 2-min project rendered end to end (plan, 16 chunks, assemble) on main, on the first commit and on the head. All three final MP4s have the same sha256 (138,129,659 bytes).
  • 30-min plan on a Linux box.
    • Every published artifact is byte-identical between main and this PR, including the mixed audio.m4a, except meta/videos.json and plan.json.
    • meta/videos.json embeds the planner's random staging directory, so it also differs between two runs of main. With that one path normalized it is identical. plan.json differs only because it carries the resulting hashes.

Tests (bun, each file 3 runs in a row):

  • New, failing on main: the compiled tree is published while the audio mixes, and that publish produces the same manifest as a serial build.
  • New, failing on main: a digest taken earlier in the plan is reused, not recomputed.
  • New, failing on main: a local file that no longer matches its digest is rejected by the S3 checksum.
  • New, failing on main: the handler logs download and untar timings.
  • New: the S3 publisher sends nothing for a digest it already published. Deleting the early return fails it.
  • Each new test was also checked with one deliberate mutation of the code it covers, and failed.

What I did NOT exercise

  • 30-min renders end to end. I did not run one; output identity at 30 minutes rests on the plan artifacts being identical, plus the 2-min full renders.
  • The GCS planner. It shares the overlap and the memo but was not run against real GCS. gcsTransport still hashes locally before its upload, since GCS has no SHA-256 check on its side.
  • The non-deferred plan path. Here the prehash also covers extracted frame trees, and it was only exercised by the existing unit tests.
  • Downloaded or out-of-project videos. These ship under video-sources/ and are not pre-uploaded during the audio window, so they still upload after it. This fixture's video lives in the project, so it ships under compiled/.

@github-actions

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 183 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (1)

  • crop-none-px-r0-root-z100: tracking 0.04, pressJump 0, drop 40.03, reload 40.03, render 39.98, renderKey -, undo true, teleport true / tracking 0.04, pressJump 0, drop 0.06, reload 0.06, render 0.03, renderKey -, undo true, teleport true / tracking 0.04, pressJump 0, drop 0.06, reload 0.06, render 0.03, renderKey -, undo true, teleport true

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at a3fa10e2. I checked the byte-equivalence claim by reading every path that feeds an artifact digest or the plan hash. I didn't re-run the bench.

Why the outputs can't change

  • Audio inputs are the same. runAudioStage writes only under workDir (MIXED_AUDIO_FILENAME and audio-work/), which is .plan-work, outside the three trees being hashed and uploaded beside it. It only reads compiledDir, now finalCompiledDir. That's the same tree after the rename (staged compiled/ no longer exists at that point), and resolveProjectRelativeSrc still resolves against projectDir first. So the overlap can't race a file being hashed or uploaded.
  • The hashes are the same function. sha256File moved verbatim from planV2.ts to planHash.ts, and sha256FileAsync hashes the same byte stream. computePlanHash gets the same {path, sha256} list. Only the source of each sha changed: before, readFileSync and sha256Hex; now the memo or a streamed read.
  • Nothing in the prehashed trees changes after the prehash. Everything written into planDir after the audio block lands in meta/, plan.json and the audio path. None of that is prehashed: the memo only misses on them, and freeze fills it after writing them. The meta/videos.json write happens before the audio block. The memo is per plan (new Map() in planV2WithPublisher), so a reused planDir can't hit a stale entry.
  • Upload integrity holds without the local re-hash. uploadContentAddressedFileToS3 is a single PutObjectCommand (not multipart) with ChecksumSHA256 set to the expected digest, so S3 checks the full object and rejects a local file that drifted. The new test covers that. The IfNoneMatch: "*" conditional create and the HEAD reuse check are unchanged.
  • Pre-uploaded blobs end up in the manifest. The #publishedDigests early return in putBlob happens after assertSha256, so only already-published content-addressed blobs skip the HEAD. artifactTargets ships every compiled/ file to every chunk, so nothing uploaded in the audio window is orphaned, and the new test checks the manifest matches a serial build.

Non-blocking

  • Promise.allSettled means a fast audio failure still waits for the prepublish uploads to finish before the plan throws and the publisher aborts. That's correct, just slower on the failure path. Racing the abort is an option if it ever matters.
  • sha256File in s3Transport.ts may now have no callers in the package. Worth a quick check whether anything else imports it.
  • As the body says, the GCS planner and the non-deferred path (prehash over extracted frame trees) are covered only by unit tests.

Required CI is green at this head. No live run.

— Rames

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 17, 2026
Merged via the queue into main with commit fb17f41 Oct 17, 2026
75 checks passed
@miguel-heygen
miguel-heygen deleted the cloudrender/plan-overlap branch October 17, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants