Skip to content

fix(engine): a chunk's first frame shows its own time, not the composition's opening frame - #5807

Closed
miguel-heygen wants to merge 1 commit into
mainfrom
cloudrender/first-frame-present
Closed

miguel-heygen wants to merge 1 commit into
mainfrom
cloudrender/first-frame-present

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

What the user sees

A distributed render (plan, then chunks, then assemble) could start any chunk after the first on the composition's opening frame: the frame at time 0, with the background video on its first frame and no overlays yet. It is one wrong frame per chunk boundary, so it shows as a flash at every seam. Whether a given chunk got it was down to timing, so two renders of the same plan gave different videos.

Cause

On a BeginFrame capture session, the first display-producing HeadlessExperimental.beginFrame can return the frame Chrome composited during session init, not the seek the page has already applied. Logged just before that capture, the DOM was already correct (title opacity 1, native <video> hidden, injected frame visible and decoded), and Chrome answered hasDamage: true with a fresh screenshot. In some runs that screenshot was still the init frame. Every later frame of the session was on time (checked against the source video, frame by frame).

What triggers it is the chunk's BeginFrame liveness probe (renderChunk → beginFrameSessionNeedsScreenshotFallback → probeBeginFrameLiveness). It sends one more BeginFrame on the capture session after init and before the first seek. On main, on the host where it was stale 9 times in 10:

chunk session (chunks 1 to 4, idle) first frame stale
as shipped (probe on) 10 of 10
probe skipped 0 of 12
probe sent with noDisplayUpdates: true 11 of 12
locked warmup turned off (probe on) 10 of 10 (ruled out)

So any extra BeginFrame between init and the first seek is enough, whether or not it displays. The probe itself is needed: it is how a chunk detects a SwiftShader compositor stall. Local renders never probe their capture sessions this way, which is why they are not affected (next section).

Local renders (hyperframes render): not affected

Do local multi-worker renders flash too? No. 0 of 240 worker-start frames were stale on main, and 0 of 240 with this PR. I rendered the same pinned fixture locally with --workers 1, with --workers 4 (the default interleaved split: worker i draws frames i, i+4, ..., so the worker starts are frames 0 to 3), and with --workers 4 plus HF_CAPTURE_PARALLEL_STREAM=false (the contiguous disk split: worker starts 0, 75, 150, 225). That was 5 runs of each idle and 5 under 16 CPU hogs, on two Linux hosts, plus 15 extra loaded --workers 4 rounds alternating main and this PR. Every log reported captureMode: beginframe and the requested worker count. Worker-start frames were compared with the --workers 1 render by per-frame PSNR (a stale one is about 21 to 25 dB).

host tree --workers 1 frame 0 --workers 4 interleaved starts --workers 4 contiguous starts renders byte-identical to --workers 1
A (8 cores) main 0 of 10 stale 0 of 100 0 of 40 45 of 45
A this PR 0 of 10 0 of 100 0 of 40 44 of 45
B (32 cores) main 0 of 10 0 of 40 0 of 40 30 of 30
B this PR 0 of 10 0 of 40 0 of 40 30 of 30

Local output is byte-identical between main and this PR on both hosts, so the extra BeginFrame changes nothing there. One loaded --workers 4 render with this PR differed at frame 5 only. That frame is worker 1's second frame, not a worker start, and showed the title one step behind the video. It did not recur in 15 alternating rounds (main 0 of 20 loaded --workers 4 renders, this PR 1 of 20), so I cannot tell whether it is related.

Fix

In captureFrameSurface, the first BeginFrame capture of a session first displays the seek once. It sends one visual BeginFrame with no screenshot at the capture tick minus half an interval. Ticks stay monotonic: the producer's liveness probe sits at base − 5 intervals and the init commit at base − 6. This covers chunk workers, parallel workers and retry pages, because each one is a fresh session. It adds one BeginFrame per session.

Measured

A 10 s 1080p30 fixture with an H.264 background video, a GSAP title and AAC audio, split into 5 chunks. It was generated once and pinned by sha256, and every chunk of the same plan was rendered 10 times (5 idle, 5 under CPU hogs).

distinct outputs per chunk (0 to 4) chunk first frames showing the opening frame
main, Linux host A (8 cores, 8 hogs) 2, 2, 2, 2, 2 23 of 40 (18 of 20 idle, 5 of 20 loaded)
main, Linux host B (32 cores, 32 hogs) 2, 2, 2, 1, 3 4 of 40 (0 idle, 4 of 20 loaded)
this PR, host A (8 hogs) 1, 1, 1, 1, 1 0 of 40
this PR, host B (16 hogs) 1, 1, 1, 1, 1 0 of 40

Which way the race goes depends on the host: host A was usually stale when idle, host B only when loaded. On both hosts chunk 0 also flipped between Chrome's own paint of the native video and the injected extracted frame. With the fix it always shows the injected frame. A stale first frame measures 21 dB against the correct one; the rest of the chunk differs only by the encoder carrying that frame forward (about 50 dB).

Tests: the new frameCapture-firstBeginFrame.test.ts fails on main (both cases) and passes with the fix, 3 runs in a row. The engine's frameCapture and screenshotService suites pass (31 files, 333 tests). The full engine unit suite passes except 4 ffprobe.test.ts cases that read an LFS fixture which was not fetched in this checkout; they fail identically on main.

Not exercised

  • The producer Docker regression baselines. A render starting at frame 0 now shows the injected video frame on frame 0, not Chrome's native paint of the same frame. In this fixture that is a 42 dB difference, which should sit inside checkpoint tolerances, but I have not run them.
  • macOS (screenshot or drawElement capture). Only the beginframe branch changed.
  • Mid-session hard cuts. Only a session's first capture was seen to go stale; later frames were on time in every run.
  • Why an extra BeginFrame before the first seek makes Chrome present the pre-seek frame. The table above pins the trigger, not Chrome's internal reason.

The plan step was deterministic apart from the plan folder path baked into planHash, which #5782 already fixes.

@github-actions

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 168 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant