fix(engine): MP4, MOV and TS exports keep sound and picture in sync when streams start apart - #5455

What

Exports of a source whose audio and video streams start at different times (MP4, MOV, MPEG-TS) now place picture and sound where the browser preview shows them. That covers the whole clip, every render path (in-process, distributed, HDR layers) and loops.

Scope: MP4, MOV and MPEG-TS sources. WebM and MKV are a known gap that a follow-up fixes: the Matroska demuxer reports start_time 0 for every stream even when the first video packet is 7 s in, so this change cannot see the offset there and those files export exactly as on main.

A user reported an export that lost its first ~7 s of picture and whose sound was wrong. One source property reproduces both symptoms: per-stream start_time values that differ, e.g. a video stream that starts 7 s after its audio. Keyframe-cut screen recordings have this shape, and so do recorders that start audio before the first video frame.

Why

On main, for a 12 s clip with the source played from 0:

The picture of the video-late source has two problems on main:

• It shows the first frame frozen for 7 s, where the browser shows nothing.
• It freezes on the last frame for the final 7 s of the clip, where the browser keeps playing.

Root cause, in packages/engine:

• Audio (audioMixer.ts, both the <video> and the <audio> path):
  • -ss <mediaStart> -t <dur> went before -i. Even -ss 0 makes the mov demuxer seek the default stream (video) and move every other stream to the keyframe it found, so audio before the video's first keyframe was cut.
  • The extracted WAV has no timestamps and is placed sample 0 at the clip start, so a late audio stream played early by its offset.
• Picture:
  • Frames are sampled with fps=...:start_time=0, which fills the slots before the first frame with that frame.
  • resolvePlayableVideoDuration returned the video stream's own length. Its callers treat it as an end time on the media timeline (the extraction window, the held tail, the loop cycle), so a late stream was cut, or wrapped, lead seconds early.

Related work

None open. The sibling fix for a split scene's in-point lives in the producer's media collector and does not touch these files.

How

• ffprobe.ts: one helper measures how far after the file's media time 0 a stream starts. Media time 0 is the earliest audio/video stream start, which is the browser's origin. It feeds AudioMetadata.latestStreamLeadSeconds and VideoMetadata.videoStreamLeadSeconds, and both come from the probe each path already makes.
• audioMixer.ts: one extractAudioSegment serves both audio paths.
  • When mediaStart is at or past every stream's start, it keeps today's fast input seek, with the same argv and output.
  • Otherwise it reads without an input seek: atrim=start=S:end=S+D,asetpts=PTS-S/TB,aresample=async=1:first_pts=0. This trims by timestamp first and only then pads the leading gap, so memory stays bounded by the clip, not by the stream offset. The trim also goes in front of the rate-lane graph.
• videoFrameExtractor.ts:
  • resolvePlayableVideoDuration now returns the video's end on the media timeline (lead + stream length). The extraction window, the held tail, the loop cycle and frame coverage all read it from there.
  • The final-frame probe converts its stream-relative timestamp by the lead.
  • isVideoHiddenBeforeStreamStart is the one rule for "nothing on screen yet". Both getFrameIndexAtTime (the SDR and distributed lookup) and the HDR layer blit use it.
• Distributed plans (distributed/shared.ts) carry videoStreamLeadSeconds the same way they carry videoStreamStartSeconds. Older plans read it as 0.

Decisions (reversible):

• Before a late video stream starts, export shows nothing, not a frozen first frame.
  • Measured by seeking hyperframes snapshot on sources with a 0.1, 0.5, 1, 2, 3 and 7 s video lead. Chrome paints the upcoming first frame when it is under 1 s away and nothing when it is 1 s or more away (3 s lead: black at 2.00 s, first frame at 2.03 s).
  • The rule copies that, so the common small offsets (a video starting a frame or two late) render exactly as before.
• Audio track selection is unchanged. FFmpeg still picks the track it picks today.

Test plan

New packages/engine/src/services/streamStartOffset.test.ts (real FFmpeg, skipped without it). It builds the sources from lavfi; a tone of 200 + 100·s Hz names each source second.

Sound, checked as a per-second source map:

• <video> and <audio> elements, from media start 0 and 3;
• an MPEG-TS remux (clock starts at 1.4 s, so no stream starts at 0);
• a clip with a rate lane;
• a clip that ends before the audio starts.

Picture:

• no frame while a late video's first frame is ≥ 1 s away, frames after that;
• the extraction window covers the whole media length;
• a loop wraps on the full media length.

Other tests:

• hdrCompositor.test.ts: the HDR blit paints nothing for a late stream before it starts.
• videoMetadata.test.ts: plans round-trip the lead.

Red without each fix (each mutant applied alone, then restored):

Runs:

• New tests pass 3 runs in a row.
• audioMixer*, videoFrameExtractor*, audioFxRender, audioVolumeEnvelope, videoFrameInjector: green. ffprobe.test.ts: the only failures are 4 tests that need LFS fixtures my checkout skipped, the same as on main.
• Producer: hdrCompositor, captureHdr* and videoFrameCoverage green (93); distributed/ green (232, bun).

Memory, reading a 12 s window from a file whose audio starts 1 h late:

End to end, hyperframes render --format mp4 --fps 30 --quality standard --workers auto on Linux:

• Unit tests added/updated
• Manual testing performed
• Documentation updated (if applicable)
• Comments follow CONTRIBUTING.md "Comments"

Not verified:

• Chrome's audio timing for these files is inferred from the container timestamps, not measured.
• The browser's picture rule was measured on paused seeks (snapshot), not during continuous playback.
• HDR: there is a unit test for the blit, and I checked that FFmpeg's HDR raw extraction fills the leading slots the same way as SDR (360 frames for 12 s of a 7 s-late source). I did not render a skewed HDR source end to end.
• Distributed: the plan round trip is tested, but no distributed render was run end to end.
• Not checked on Windows or on the macOS GPU path.
• WebM/MKV: not fixed here (see Scope). FLV-like files that report a stream start but no stream duration are handled in that follow-up too.
• Out of scope, and wrong on main too: a ramped (rate-lane) clip on a source whose audio starts late plays the ramp offset. That is a follow-up.