Long 4K renders on Linux/NVIDIA: NVDEC decodes 2× faster, yet the full render gains only ~7% — where the time actually goes #6881

TL;DR

In isolation, NVDEC decodes one of my 4K H.264 sources (3840×2160, 25 fps, 9 300 frames) about 2× faster (117.5 s → 57.8 s, plain ffmpeg -f null, median of 3 runs). Plugged into a full HyperFrames render (120 s excerpt, 4K sources, 1080p output), the same change makes the whole render only ~7% faster (204.6 s → 190.1 s, median of 3). The bottleneck is elsewhere, and this issue is my attempt to show where — with numbers, the code paths involved, and two questions about which direction would fit the project.

Follow-up to #6096.

A word on method

I'm a lead tech / technical consultant and a long-time video geek, not a TypeScript or FFmpeg expert. I directed this investigation and kept the overall architecture in view; most of the code was written with Claude Code (Opus), and this text was drafted with an AI assistant. I've checked what I'm claiming against the code and the logs, and every measurement below comes from my machine and can be replayed. Anything I haven't verified is marked as such.

Context and hypothesis

My guess is that the current defaults are tuned for shorter compositions and for modern machines (Macs especially, with VideoToolbox in the loop). My setup is the other end of the spectrum:

• Linux, headless, NVIDIA RTX 3060, 8 vCPU
• long compositions (a 16-minute master), 25 fps camera footage in 1080p and 4K, with HTML overlays (titles, subtitles, illustrations)

Here is what happens there.

Finding 1 — the fixed 15 s static-frame verification budget

packages/engine/src/services/frameCapture.ts caps static-frame dedup verification at STATIC_VERIFY_MAX_MS = 15_000. When the budget runs out, ranges already verified stay deduplicated and the rest are captured normally (outcome=time_budget).

On long or 4K compositions, 15 s covers only a small fraction of the ranges:

Proposal: make the budget scale with the work (e.g. with frame count × pixel count, or with the measured cost of the first verifications), keeping 15 s as the floor so short renders behave exactly as today. A first step is in the fork: an HF_STATIC_DEDUP_VERIFY_MAX_MS override (default unchanged, still 15 s, exhaustion still fails closed). It applies cleanly on today's main, and the engine's frameCapture tests pass with it (35 files, 347 tests, plus tsc --noEmit). It's an override, not yet the adaptive budget; I'd rather agree on the shape with you first.

Note: on one 4K excerpt with a 600 s budget, verification correctly found a mismatch (predicted-static frame 2633) and disabled dedup, so a larger budget did not help there. To me that's a good sign that verification works and should keep running, just with a budget proportional to the job.

Finding 2 — JPEG round-trips on the CPU

For each frame that contains video, as far as I can read the code:

1. ffmpeg extracts the video frame as JPEG (videoFrameExtractor.ts)
2. it is base64-encoded and served to the page (videoFrameInjector.ts, videoFrameTransport.ts)
3. Chrome decodes that JPEG
4. Chrome re-encodes the composited page as JPEG, quality 80 (screenshotService.ts)
5. ffmpeg decodes it again (image2pipe / mjpeg in streamingEncoder.ts)
6. and only then does the final encoder (NVENC here) run

That's two JPEG encodes, two JPEG decodes and a base64 hop per frame, all on the CPU. In my 1080p master, capture accounts for ~875 s of a ~1 205 s render. It's a bit like making a photocopy of a photocopy of a photocopy of a photocopy: each step is reasonable on its own, but the result costs time and a little quality at every hop.

I understand why it's built this way (Chrome is the compositor, and JPEG keeps the transport cheap). I'm not suggesting it's wrong in general, only that on long 4K renders it dominates everything else, which is why NVDEC alone barely moves the total.

Finding 3 — Chrome and ffmpeg interact badly around transparency (out-of-tree prototype)

This one showed up in my out-of-tree prototype (Finding 4), not in the stock pipeline, but I think it matters for any design that composites a transparent Chrome layer over video:

• The overlay was rendered with a transparent background (--format png-sequence).
• On frames that are fully opaque (a full-frame illustration, for example), the PNG comes out as RGB without an alpha channel; other frames are RGBA. HyperFrames writes the buffer returned by Page.captureScreenshot to disk unchanged (writeCapturedFrame in frameCapture.ts), so the RGB-vs-RGBA choice is made by Chrome's PNG encoder, not by HyperFrames. I checked the HyperFrames side in the code; I haven't looked at the Chromium side.
• When ffmpeg reads such a sequence through concat, each RGB↔RGBA switch rebuilds the filter graph, and fps drops a frame each time: 1 995 of 9 180 frames differed from the dense reference. With -reinit_filter 0, 875 frames were still wrong (the RGB frames get misread).
• The first workaround, re-encoding the RGB PNGs as RGBA files, was correct (45.1 dB) but slow: 859 s / 913 CPU-s.
• The retained fix: read the overlay in a separate ffmpeg process that normalises to RGBA and pipes rawvideo through NUT into the compositing ffmpeg. 0 frames differ, same fidelity, 207 s / 323 CPU-s.

So it's a chain reaction: a transparent page changes what comes out of the capture, and that change pushes ffmpeg onto a much slower or incorrect path. If a native "video under transparent overlay" path is ever built, the alpha format needs to be pinned at the boundary.

Finding 4 — out-of-tree GPU prototype (not a PR)

To check where the ceiling is, I built a prototype outside HyperFrames: Chrome renders only the overlay (videos removed, transparent background); the source video is decoded, scaled and composited on the GPU (Vulkan + libplacebo), then encoded with NVENC.

Caveats, honestly: one run, one machine; output is PSNR-Y ≈ 45 dB, not bit-exact; it relies on a strict composition shape (full-frame video, overlay on top); it's NVIDIA/Vulkan-only; it hands PNGs over through disk (a 4K master needed ~7.5 GB and once filled the disk); and it rewrites the HTML from outside the engine. It's a measurement tool, not something I'd propose to merge.

Questions

1. Would an opt-in, internal "video layer passthrough" fit the project: for intervals where a <video> is full-frame under the overlay, capture the page without the video on a transparent background, and composite the source video inside the streamingEncoder ffmpeg (software overlay by default, GPU behind the hwaccel probes introduced in perf(engine): on a Mac, render frame extraction decodes video on the media engine #5878)?
2. Or would you rather start with something small: a PR making the verification budget adaptive (15 s floor), then discuss the rest?

I'm happy to go with whichever you prefer, or with a different approach you have in mind; you know the code far better than I do.

Reproducing

• Documentation fork with clean branches off main, the scripts, the measurement commands and the reasoning behind each choice: https://github.com/Junod972/hyperframes (not maintained, for reference only):
  • budget patch + unit test: https://github.com/Junod972/hyperframes/tree/doc/dedup-verify-budget
  • GPU passthrough prototype, scripts only, not wired into the engine: https://github.com/Junod972/hyperframes/tree/doc/gpu-video-passthrough-poc
• Synthetic test composition built on Tears of Steel 4K — (CC) Blender Foundation | mango.blender.org, CC BY 3.0 — so no private footage is involved. Note: the film is 24 fps; my footage is 25 fps, and the test will state which rate it uses.

Thanks for HyperFrames: the tool is great, and that's exactly why I'd like to see it go further on this kind of workload.