fix(engine): late-starting MP4 and WebM videos render every frame the preview shows by miguel-heygen · Pull Request #6664 · heygen-com/hyperframes

miguel-heygen · 2026-10-28T11:52:45Z

What

A render could stop at the video coverage gate with "captured 94 of expected 101 frames" for a clip that plays fine in the preview. The clip was an MP4 fragment cut from the middle of a longer stream, so its first frame is at 0.25 s, not 0.

The browser counts an MP4 or WebM from timestamp 0: it shows that clip's first frame at 0.25 s and plays until the duration the file reports. FFmpeg counts from the file's earliest stream start instead. Extraction therefore read the clip 0.25 s early, ran out of frames 0.25 s before the end the browser plays to, and came back 7 frames short. Every frame it did return was also shown 0.25 s too early. The audio mixer seeked the same way, so in a file whose streams all start late the audio was early by the same amount.

Rule

Render matches preview: media time is the browser's timeline.

The probe records mediaTimeOffsetSeconds: how far the browser's timeline runs ahead of FFmpeg's. For an MP4 or WebM that is the earliest stream start when it is positive. Transport streams (MPEG-TS starts its clock at 1.4 s) keep their earliest stream as media time 0, as before.
ffmpegSeekSeconds converts a media time into FFmpeg's -ss. It is now used by all three seeks in frame extraction (single pass, frame ranges, final frame), all three in HDR extraction, and the audio mixer's input seek and timestamp trim. Audio metadata carries the same offset. The audio sidecar is a stream copy on FFmpeg's clock, and the chunk audio cuts read the assembled mix, not the source, so neither changes.
videoStartOnMediaTimeline is where the video begins on that timeline. It anchors the source frame grid, the rule that hides a video until its first frame is under a second away, and the held-tail frame count.
The playable duration is unchanged. It already matched the browser's duration for these files. The post-extraction duration correction still subtracts only the stream lead: it computes a stream duration, which the playable duration adds back to that same lead.
When a file runs past the duration it reports, the held final frame is the one at that end, which is what the browser holds.
Distributed plans carry the new field to chunks. Plans written before it read 0, which keeps today's behavior.

Behavior changes only for MP4 and WebM files whose earliest stream starts after 0:

An edit-list offset of a second or more: the video is now hidden until about a second before its first frame, as in the preview. Before, it showed its first frame from 0.
Small positive starts are common, for example a screen recording whose first frame is at 0.008 s. Frames now change where the browser changes them, up to that offset later than before.

Regression baseline

vfr-screen-recording uses such a clip: a variable-frame-rate screen recording whose first frame is at 0.008008 s. The browser reports that frame at media time 0.008008, its raw timestamp, so at time t it shows the last frame stamped at or before t. The old extraction picked the last frame stamped at or before t + 0.008. Over the fixture's 90 output frames the two rules pick different source pictures at 28 frames, in eight runs: 0-0.13 s, 0.70-0.83 s, 0.97 s, 1.27-1.33 s, 1.93 s, 2.17-2.23 s, 2.43-2.57 s and 2.83-2.97 s. CI's 26 failed checkpoints all fall in these runs. The baseline is regenerated in the test image.

Tests

fails without the fix: streamStartOffset.test.ts (94 of 99 frames, 90 of 106 frames, frame order from slot 2; audio 2 s early in three cases; picture off the audio's clock)

streamStartOffset.test.ts generates its clips with FFmpeg: a fragmented MP4 cut to start at 0.25 s (24 fps, B-frames), a WebM starting at 0.52 s (off the source frame grid), an MP4 with a 1.5 s edit-list offset, and an MP4 whose audio and video both start at 2 s. In the video clips every frame is a different grey, so two output frames are equal only when they show the same source frame.

The frame count equals the playable duration times the output rate. This is what the gate checks.
Every pair of neighbouring frames is equal exactly when the browser shows the same source frame at both times.
The video is visible, or hidden, where the browser shows it.
Frame-range extraction writes the same bytes as one pass.
In the file whose streams start at 2 s, the audio's tone heard each second, for <audio> and <video> and through both the timestamp-trim and input-seek paths, is the source second the browser plays then.
In that file, the picture at media time 2 + k is the source picture at k: the picture is on the same clock as the audio.

Each of these mutants fails at least one case:

frame-range seeks unshifted
single-pass seek unshifted
grid anchored at the old lead
hide rule on the old lead
WebM not counted from 0
the mixer's trim unshifted
the mixer's seek unshifted

The HDR seeks use the same helper but have no HDR-specific test.

… preview shows The browser counts an MP4 or WebM from timestamp 0, but FFmpeg seeks from the file's earliest stream start. A file whose first frame comes after 0 (a fragment cut from a longer stream, an edit-list offset, a WebM with a start offset) was extracted early and short, so renders tripped the video coverage gate. Every video seek now maps media time to FFmpeg's timeline, and the video begins where the browser shows it.

github-actions · 2026-10-28T12:13:13Z

Edit accuracy: accurate 271 (base branch 271), smooth 137 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

heygengenesis

Verdict: REQUEST CHANGES · 4 blockers · 0 important · 0 nits · head 99a5bfb4

No fails without the fix: line. "On main the three cases fail…" names the failures but not on the required line. Add e.g. fails without the fix: streamStartOffset.test.ts (94/99, 90/106, slot 2).
Required regression is red. vfr-screen-recording failed visual (26 frames, threshold 2). Its src/clip.mp4 starts at 0.008008 s, so the new offset applies to it. The merge base b847622 is green. Either regenerate the golden and show the new frames match the browser, or fix the shift. Either way the PR body has to account for it.
Audio isn't moved with the video ↓ inline (ffprobe.ts:525).
Comments is red: comment share went up in videoFrameExtractor.ts (+1) and ffprobe.ts (+4) ↓ inline.

Scope

Read the whole non-test diff and grepped every videoStreamLeadSeconds reader and every -ss in packages/*/src. videoFrameExtractor.ts:1354 (post-extraction duration correction) still uses only the lead; its output is a stream duration, so I think that is consistent, but please confirm. CI at review: CI and Windows render verification still running.

heygengenesis · 2026-10-28T12:14:12Z

+}
+
+/** FFmpeg's input `-ss` for a media time: FFmpeg counts from the file's earliest stream start. */
+export function ffmpegSeekSeconds(


blocker (A/V sync): the audio extraction for the same <video> still seeks on FFmpeg's timeline. audioMixer.ts:797 does -ss mediaStart -i src with no offset. After this PR, video in a file whose a/v streams both start late is shifted by mediaTimeOffsetSeconds and its audio isn't, so renders now have audio ahead of picture by the offset (0.25 s for the fragment case, 1.5 s for the edit-list case). Before, both were early by the same amount and stayed in sync.

Repro (ffmpeg 5.1): -output_ts_offset 0.25 MP4 with a click and a white flash both at content 0.5 s (browser media time 0.75):

audio as the mixer reads it (-ss 0): click at 0.521 s

video before this PR (-ss 0): white at 0.542 s

video after this PR (-ss -0.25): white at 0.792 s

Route the mixer's seek (and chunkAudio.ts cuts if they read the source) through ffmpegSeekSeconds, and add a test that checks a/v alignment.

Fixed in 2a7659e. The mixer's input seek and timestamp trim now go through ffmpegSeekSeconds, and audio metadata carries the same mediaTimeOffsetSeconds. The chunk audio cuts read the assembled mix, not the source, so they are unchanged; the audio sidecar is a stream copy on FFmpeg's clock. Tests: a generated MP4 whose audio and video both start at 2 s. The tone heard each second is the source second the browser plays (<audio> and <video>, trim and seek paths; 2 s early without the fix), and the picture at media time 2 + k is the source picture at k (63b68f7; fails with the video seek unshifted).

✅ Fixed in 3498b3c.

heygengenesis · 2026-10-28T12:14:12Z

  /** Seconds after the file's media time 0 (its earliest audio/video stream start) at which video begins. */
  videoStreamLeadSeconds?: number;
+  /** How far the browser's media time runs ahead of the file's: it counts an MP4 or WebM from timestamp 0. */
+  mediaTimeOffsetSeconds?: number;


blocker (Comments ratchet): RAISED comment share: ffprobe.ts 13.8% (208/1506), base (204/1481). videoFrameExtractor.ts too (+1, the line-1624 comment). Cut these, or move the meaning into names. mediaTimeOffsetSeconds / videoStartOnMediaTimeline already say most of it.

Fixed in 63b68f7: the added comments are gone and their meaning is in names (BROWSER_PLAYS_FROM_TIMESTAMP_ZERO, lastInstantTheBrowserPlays). scripts/comment-ratchet.mjs passes locally against the merge base.

✅ Fixed in 3498b3c.

…view plays it The mixer seeked and trimmed audio by FFmpeg's clock, which starts at the file's earliest stream. In an MP4 or WebM whose streams all start after 0, the browser plays that audio later, so the render heard it early by the same amount. The mixer now maps media time onto FFmpeg's clock with the same offset frame extraction uses.

…dio plays on

… timestamp

miguel-heygen · 2026-10-28T12:30:00Z

Head 3498b3c answers the four blockers:

fails without the fix: line is in the body under Tests.
Audio: moved with the video (2a7659e), with an a/v clock test (63b68f7). Replied inline.
Comments: ratchet passes (63b68f7). Replied inline.
regression / vfr-screen-recording: its clip's first frame is at 0.008008 s, and the browser reports that frame at media time 0.008008, so frames now change up to 8 ms later than before. Over the fixture's 90 frames the old and new rules pick different source pictures at 28 frames in eight runs, and CI's 26 failed checkpoints all fall in those runs. The baseline is regenerated in the test image (3498b3c, LFS pointer), and the fixture passes there with 0 failed frames. The body's Regression baseline section has the runs.
Scope question (line 1354): confirmed. The correction computes a stream duration, and the playable duration adds it back to the same stream lead, so it keeps using the lead alone.

heygengenesis

Verdict: APPROVE — 4 resolved, 0 open, 0 new · 99a5bfb4 → 3498b3c. The audio seek now uses ffmpegSeekSeconds, and I re-checked with ffmpeg: on the timestamp-trim path the click lands at 0.749 s against the browser's 0.75 s. All CI is green, including regression with the updated vfr-screen-recording golden.

heygengenesis Bot requested changes Oct 28, 2026

View reviewed changes

miguel-heygen added 3 commits October 28, 2026 08:15

fix(engine): a late-starting file's picture shows on the clock its au…

63b68f7

…dio plays on

test(producer): vfr-screen-recording baseline shows each frame at its…

3498b3c

… timestamp

heygengenesis Bot approved these changes Oct 28, 2026

View reviewed changes

heygengenesis Bot mentioned this pull request Oct 28, 2026

fix(core): bundled fonts in a linked stylesheet or a font shorthand are seen by the bundler #6665

Merged

2 of 4 tasks

miguel-heygen added this pull request to the merge queue Oct 28, 2026

Merged via the queue into main with commit 4377840 Oct 28, 2026
77 of 78 checks passed

miguel-heygen deleted the fix/video-start-offset branch October 28, 2026 13:31

miguel-heygen mentioned this pull request Oct 28, 2026

chore: release v0.8.213 #6673

Merged

brandonbryant12 mentioned this pull request Oct 28, 2026

Track motion-design trends, standout X videos and rendering resources brandonbryant12/clearframe#9

Open

mrgoonie mentioned this pull request Oct 31, 2026

Nâng @hyperframes/* từ 0.8.78 lên 0.8.223 (fix video shot, FFmpeg 8 ở width 1080, GSAP seek) digitopvn/studix#90

Closed

AksharP5 mentioned this pull request Oct 31, 2026

fix(engine): preserve boosted audio through volume automation #6612

Open

5 tasks done

Navigation Menu

Repository navigation

fix(engine): late-starting MP4 and WebM videos render every frame the preview shows - #6664

fix(engine): late-starting MP4 and WebM videos render every frame the preview shows#6664
miguel-heygen merged 4 commits into
mainfrom
fix/video-start-offset

miguel-heygen commented Oct 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 28, 2026 •

edited

Loading

Uh oh!

heygengenesis Bot left a comment

Uh oh!

heygengenesis Bot Oct 28, 2026

Uh oh!

miguel-heygen Oct 28, 2026

Uh oh!

heygengenesis Bot Oct 28, 2026

Uh oh!

heygengenesis Bot Oct 28, 2026

Uh oh!

miguel-heygen Oct 28, 2026

Uh oh!

heygengenesis Bot Oct 28, 2026

Uh oh!

miguel-heygen commented Oct 28, 2026

Uh oh!

heygengenesis Bot left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

miguel-heygen commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Rule

Regression baseline

Tests

Uh oh!

github-actions Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Edit accuracy: accurate 271 (base branch 271), smooth 137 of those

Uh oh!

heygengenesis Bot left a comment

Choose a reason for hiding this comment

Uh oh!

heygengenesis Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

miguel-heygen Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

heygengenesis Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

heygengenesis Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

miguel-heygen Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

heygengenesis Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

miguel-heygen commented Oct 28, 2026

Uh oh!

heygengenesis Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

miguel-heygen commented Oct 28, 2026 •

edited

Loading

github-actions Bot commented Oct 28, 2026 •

edited

Loading