Skip to content

fix(producer): extract HDR video frames a window at a time to bound scratch disk - #6304

Merged
miguel-heygen merged 5 commits into
mainfrom
fix/hdr-extract-per-window
Oct 25, 2026
Merged

miguel-heygen merged 5 commits into
mainfrom
fix/hdr-extract-per-window

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

A long HDR clip no longer needs the whole clip's raw frames on disk at once, so a 5 minute HLG video renders instead of stopping at the disk-space check.

Cause: extractHdrVideoFrames wrote every frame of each HDR video as raw 16-bit RGB before the first frame was captured: 6 bytes per pixel per frame. 5 minutes at 30 fps and 1080p is 9000 frames x 12.4 MB = 112 GB. The headroom check then refuses it (draft quality changes nothing, extraction does not depend on it).

Fix: each video is cut into windows of about 512 MB (43 frames at 1080p). A window is extracted with one FFmpeg pass when the capture first reaches a frame in it, the next one starts half way through, and least-recently-used windows are deleted once more than workers + 1 exist. Scratch disk is that bound, not the clip length. Frames are read through a lease that pins a window while a frame is read, so parallel capture workers cannot evict each other's window.

Frame accuracy: every window runs on the clip's own tick grid. The seek lands on a whole second before the window, fps is pinned to the clip's first frame offset (probed once), and the lead-in frames are trimmed. Checked against the old whole-file pass byte by byte at every frame, window boundaries included, on a 5 minute 10-bit HLG clip (mediaStart 3.37 and 0, 43-frame windows) and a 24 fps clip (three start offsets): 0 mismatches of about 1200 frames each. The naive per-window seek is wrong on 24 to 30 fps (478 of 1199 frames), which is why the grid is pinned.

One behaviour change: a variable frame rate source now maps each output frame to the nearest source frame. The old pass (-r) carried timing drift from earlier frames that a window cannot replay (171 of 1200 frames in the VFR fixture differed, always the neighbouring source frame). The windowed output is identical between windows and a single whole-file pass.

Measured on a 1080p 10-bit HLG clip, draft render, peak extra disk during the render:

  • 30 s: 11.3 GB before, 2.2 GB after; output MP4 byte-identical. Wall time 3m 13.8s on main vs 3m 12.6s windowed on the final build (same devbox, one run each; an earlier windowed run with prefetch took 2m 36.4s on a quieter box), so the window path is not slower. The 60 s old path was not run: it needs 22 GB of scratch and the shared devbox cannot spare it.
  • 60 s: 22.3 GB before, 2.6 GB total after (1.8 GB in scratch dirs); windowed render 5m 37.6s.
  • 5 min (the reported case, 112 GB before): renders to completion, 9000 frames, 300.0 s, 10-bit HLG, peak 3.3 GB total (2.9 GB in scratch dirs).

Tests: window store (extract on demand, bounded resident windows, a leased window is not evicted, short source and end-at-boundary hold the last frame, failed extraction retries, close removes files), the FFmpeg arguments, and a long clip held in at most two windows. The two bound tests fail with eviction disabled. 50 tests pass three runs in a row.

Package size: producer packed budget raised by 2389 bytes (768601 to 770990), exactly this change's measured delta on a linux pack against current main; the bytes buy HDR scratch of 112 GB down to about 3 GB for a 5 minute clip. The code lands in both producer bundles; prefetch of the next window was left out because the 5 min render was 29m 17s with it and 30m 01s without.

Stands alone.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES · 1 blocker · 0 important · 0 nits · head cd8b174d

The windowed store and lease/eviction look right to me, and the shared scratch is safe because nothing awaits after the readSync. One blocker, inline: a looped source's cycle length is now an estimate, and it gets corrected late.

Scope

I read all non-test changes: hdrFrameWindows.ts, captureHdrResources.ts, and the hdrCompositor.ts blit/close. CI was still running when I posted. I did not run the 5 min render.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 113 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (1)

  • seqplainkeys-none-px-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true

@miguel-heygen
miguel-heygen force-pushed the fix/hdr-extract-per-window branch from cd8b174 to a255b65 Compare October 25, 2026 10:38

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES — 0 resolved, 1 open, 0 new · cd8b174d → a255b65e. The looped-source frame count is still ceil(duration*fps - firstTick) (captureHdrResources.ts:478). Setup still leases only window 0, so the loop modulus is corrected lazily, as in the open thread.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES — 0 resolved, 1 open, 0 new · a255b65e → a21d677a (removes one comment only). The looped-source frame-count blocker in the open thread is unchanged.

@miguel-heygen
miguel-heygen force-pushed the fix/hdr-extract-per-window branch from 6cb995f to e6cee60 Compare October 25, 2026 11:55

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES — 0 resolved, 1 open, 0 new · a21d677a → e6cee608. The blit split and the propagation of extraction failures look fine. The looped-source frame count (captureHdrResources.ts:479) is still an estimate corrected lazily after only lease(0) (L523), per the open thread.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE — 1 resolved, 0 open, 0 new · e6cee608 → e0760d69.

@miguel-heygen
miguel-heygen force-pushed the fix/hdr-extract-per-window branch from e0760d6 to 1fbdacd Compare October 25, 2026 13:35

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE — 0 resolved, 0 open, 0 new · e0760d69 → 1fbdacd3. Rebase onto 89af2908. The code patch is identical, and the only change is the producer packed budget (+2389 bytes).

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 25, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 25, 2026
@miguel-heygen
miguel-heygen force-pushed the fix/hdr-extract-per-window branch from 1fbdacd to 1a9281b Compare October 25, 2026 14:33

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE — 0 resolved, 0 open, 0 new · 1fbdacd3 → 1a9281bd. Rebase onto 9b80c2d0. The code patch is identical, and the producer packed budget is main plus 2389 bytes, as before.

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 25, 2026
Merged via the queue into main with commit 7569ec1 Oct 25, 2026
129 of 141 checks passed
@miguel-heygen
miguel-heygen deleted the fix/hdr-extract-per-window branch October 25, 2026 15:33
jrusso1020 added a commit that referenced this pull request Oct 27, 2026
Window slots counted from the first decoded frame, so a video stream that
starts after its audio played early by its lead. Slots now count from the
extraction start and show the first frame until the stream begins, as they
did before windowed extraction (#6304) and as Chrome does.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
ym256-stanford pushed a commit to ym256-stanford/hyperframes that referenced this pull request Oct 27, 2026
…one frame (heygen-com#6575)

* fix(producer): stop HDR video windows after the first from repeating one frame

A raw HDR window seeks to a whole second, then trims its lead-in frames.
trim keeps their timestamps, and the raw muxer writes none, so when the
input has an audio track FFmpeg's default CFR sync pads from pts 0 with
copies of the window's first frame and -frames:v cuts the real ones off.
Phone footage always has audio, so HLG clips rendered at ~half their frames.
Pass frames through as the filter graph emits them.

Fixes heygen-com#6556

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* fix(producer): hold the first frame until a delayed HDR video starts

Window slots counted from the first decoded frame, so a video stream that
starts after its audio played early by its lead. Slots now count from the
extraction start and show the first frame until the stream begins, as they
did before windowed extraction (heygen-com#6304) and as Chrome does.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* test(producer): add an HDR fixture whose clips carry audio and span extraction windows

Neither HDR fixture had an audio track, which is how heygen-com#6556 shipped. Window A
checks frames after the first extraction window are not repeats; window B
checks a video stream that starts 0.5 s after its audio holds its first frame
until the picture starts. The baseline was generated in Docker and the fixture
fails against main.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant