Skip to content

Commit 8c1b6c5

Browse files
docs(media): document automatic proxying for hostile codecs (#2596)
* docs(media): document automatic proxying for hostile codecs Describes the shipped behavior: which input codecs render, that live preview auto-proxies what the browser cannot decode, where the cache lives, and how to turn it off. Carries the skills notes and the hardening design documents. * docs(media): align proxy guidance with runtime
1 parent 35eff50 commit 8c1b6c5

8 files changed

Lines changed: 149 additions & 20 deletions

File tree

‎docs/guides/rendering.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -166,9 +166,9 @@ Video assets referenced by a composition (a `<video src="...">` clip) are decode
166166
- VP8 / VP9 and ProRes 4444 (with alpha; see [Transparent Video](#transparent-video))
167167
- HDR sources (HLG / PQ) are tone-mapped for SDR renders; see the [HDR guide](/guides/hdr)
168168

169-
The one caveat is **live preview**: `preview`, `play`, Studio, and published player pages play the file in a real browser, so playback there depends on that browser's codec support. Chrome (107+), Edge, and Safari hardware-decode HEVC on most modern machines; Firefox does not. If an HEVC asset shows a black frame in preview while rendering fine, generate an H.264 proxy for authoring (for example with `ffmpeg -i asset.mp4 -c:v libx264 -crf 18 proxy.mp4`, or via the media-use skill) and swap the original back in for the final render, or just keep the HEVC source, since the rendered output is unaffected.
169+
The one caveat is **live preview**: `preview`, `play`, Studio, and published player pages play the file in a real browser, so playback there depends on that browser's codec support. Chrome (107+), Edge, and Safari hardware-decode HEVC on most modern machines; Firefox does not. Preview handles this automatically: on first use, it transcodes the asset into a cached H.264 proxy (stored under `.transcode-cache/`, safe to delete) and serves that for playback, while render always uses the original file. `check` and `snapshot` prepare the same proxies before opening their browser phase. Publishing bakes the proxies into the published archive so static player pages work too. Opt out per command with `--no-proxy`, or project-wide by setting `media.autoProxy` to `false` in `hyperframes.json`.
170170

171-
`hyperframes lint` emits an info-level `hevc_preview_codec` note when a composition references an HEVC video, as a reminder of this preview-only limitation.
171+
`hyperframes lint` emits an info-level `hevc_preview_codec` note when a composition references an HEVC video, naming the affected asset(s).
172172

173173
## Animated GIF
174174

‎docs/guides/troubleshooting.mdx‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -137,15 +137,15 @@ If your issue is about a specific coding mistake (animations not working, video
137137
</Accordion>
138138

139139
<Accordion title="Video asset plays black in preview but renders fine (HEVC/H.265)">
140-
Rendering decodes video assets with FFmpeg, so HEVC (H.265) inputs render correctly on every platform. Live preview is different: `preview`, `play`, Studio, and published player pages play the file in your browser, and not every browser decodes HEVC (Chrome 107+, Edge, and Safari do on most modern hardware; Firefox does not).
140+
Rendering decodes video assets with FFmpeg, so HEVC (H.265) inputs render correctly on every platform. Live preview is different: `preview`, `play`, Studio, and published player pages play the file in your browser, and not every browser decodes HEVC (Chrome 107+, Edge, and Safari do on most modern hardware; Firefox does not). This is now self-healing: preview detects an undecodable asset and automatically swaps in a cached H.264 proxy, so this should no longer show as a persistent black frame.
141141

142-
If an HEVC asset shows a black or frozen frame in preview:
142+
If a hostile asset is still stuck black in preview:
143143

144-
1. Confirm the render itself is fine: `npx hyperframes render` output will contain the video.
145-
2. For authoring, generate an H.264 proxy (`ffmpeg -i asset.mp4 -c:v libx264 -crf 18 proxy.mp4`) and point the composition at it while you iterate.
146-
3. Swap the HEVC original back before the final render, or keep the proxy; both render correctly.
144+
1. Confirm auto-proxying is not disabled: no `--no-proxy` flag, and `media.autoProxy` is not `false` in `hyperframes.json`.
145+
2. Run `npx hyperframes doctor` to confirm ffmpeg/ffprobe are installed; the proxy transcode needs both.
146+
3. Run `npx hyperframes lint --verbose`: the info-level `hevc_preview_codec` finding names every affected asset.
147147

148-
`npx hyperframes lint` flags HEVC assets with an info-level `hevc_preview_codec` note. See [Rendering: Input Video Codecs](/guides/rendering#input-video-codecs).
148+
See [Rendering: Input Video Codecs](/guides/rendering#input-video-codecs).
149149
</Accordion>
150150
</AccordionGroup>
151151

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# Transparent Proxy Hardening Design
2+
3+
## Context
4+
5+
PR #2462 adds transparent authoring proxies for browser-hostile codecs. Its server-side
6+
FFmpeg architecture is the correct foundation: Chrome Headless Shell cannot decode HEVC
7+
through WebCodecs, and MediaBunny server support ultimately wraps FFmpeg through NodeAV.
8+
The review found lifecycle, policy, and coverage gaps that must be resolved before merge.
9+
10+
## Goals
11+
12+
- Preserve transparent preview recovery for hostile HEVC and ProRes media.
13+
- Never destroy alpha as an accidental fallback.
14+
- Bound CPU, memory, queue, cache, and caller wait time.
15+
- Apply one proxy policy consistently across scan, runtime rescue, check, and publish.
16+
- Make path lookup safe across Unicode/case differences without allowing project escapes.
17+
- Make partial publish behavior explicit and test render/publish separation with real media.
18+
- Preserve the existing system-FFmpeg subprocess boundary and zero new media runtime dependency.
19+
20+
## Non-goals
21+
22+
- Replacing FFmpeg with MediaBunny or NodeAV in this PR.
23+
- Proxying alpha-bearing media to opaque H.264.
24+
- Changing final render inputs or silently publishing incomplete hostile-media coverage.
25+
26+
## Architecture
27+
28+
### 1. Canonical project media identity
29+
30+
Create one resolver used by codec-map lookup and proxy resolution:
31+
32+
1. Resolve the authored path exactly first.
33+
2. Build a project-relative index normalized to POSIX separators, Unicode NFC, and lowercase.
34+
3. Use the normalized fallback only when it maps to exactly one file; ambiguous case/Unicode
35+
collisions fail explicitly.
36+
4. Resolve symlinks with real paths and reject any source outside the real project root.
37+
38+
The codec map stores canonical relative identities so scan and runtime rescue cannot disagree.
39+
40+
### 2. One alpha-aware proxy policy
41+
42+
Centralize the decision into a single policy function consumed by:
43+
44+
- initial scan and prewarm,
45+
- mapped runtime recovery,
46+
- unlisted/runtime rescue,
47+
- check-browser recovery,
48+
- publish bake.
49+
50+
Unknown/unlisted media is probed before transcoding. Alpha-bearing media is returned as an
51+
explicit unsupported result and is never sent through the opaque H.264 proxy path. Browser-safe
52+
mapped media follows the same policy at every runtime fallback tier.
53+
54+
### 3. Bounded transcode lifecycle
55+
56+
- Keep the two-process transcode semaphore.
57+
- Add a bounded wait queue and a typed overload error mapped to HTTP 503 plus Retry-After.
58+
- Deduplicate in-flight work by canonical cache key.
59+
- Cache transient failures only for a short TTL; keep FFmpeg-unavailable recoverable immediately.
60+
- Give publish/check callers explicit bounded waits. Caller timeout does not cancel shared work;
61+
it returns a typed timeout while the deduplicated transcode may finish and warm the cache.
62+
- Preserve subprocess timeout/SIGKILL and atomic temp-then-rename behavior.
63+
64+
### 4. Bounded cache
65+
66+
Maintain cache metadata from file stat data and sweep opportunistically at a rate-limited cadence.
67+
Defaults:
68+
69+
- 30-day maximum idle age,
70+
- 10 GiB maximum total size,
71+
- oldest-accessed entries removed first,
72+
- active/in-flight entries excluded,
73+
- stale temp files removed separately.
74+
75+
Environment/config overrides allow constrained machines to lower the cap. Cache cleanup failures
76+
are warnings, never preview failures.
77+
78+
### 5. Complete CLI/config plumbing
79+
80+
Carry the resolved proxy setting through regular preview, dev, local-studio, and static server
81+
spawn paths. Both positive and negative CLI flags override config consistently. Add contract tests
82+
that exercise the final staticProjectServer arguments instead of only parser-level values.
83+
84+
### 6. Explicit publish outcome
85+
86+
Publish bake returns a structured manifest containing proxied, intentionally skipped, and failed
87+
assets. Hostile non-alpha failures stop publish with the asset list; alpha skips remain explicit.
88+
Callers surface the manifest in diagnostics so an incomplete publish cannot become a silent black
89+
frame. Final render/cloud-render archive creation remains on the original-media path.
90+
91+
### 7. Regression coverage
92+
93+
Tests must cover:
94+
95+
- unlisted alpha rescue never invoking H.264 transcode,
96+
- transient failure expiry,
97+
- queue overload and HTTP 503 mapping,
98+
- case/Unicode fallback plus ambiguity rejection,
99+
- symlink/project containment,
100+
- publish/check caller timeout,
101+
- complete proxy flag forwarding,
102+
- cache size/age eviction while preserving in-flight entries,
103+
- publish manifest behavior,
104+
- reactive/tertiary policy parity,
105+
- bounded probe caches and global probe concurrency,
106+
- preview-script escaping,
107+
- a real hostile-codec fixture proving publish bake changes authoring archives while
108+
render/cloud-render archives remain byte-identical and proxy-free.
109+
110+
## Alternatives
111+
112+
### Explicit pre-bake only
113+
114+
This removes on-demand queueing but loses transparent cold-preview recovery. Rejected because it
115+
regresses the core authoring experience #2462 is meant to provide.
116+
117+
### MediaBunny replacement or hybrid
118+
119+
Browser MediaBunny inherits WebCodecs HEVC limits; its ProRes path adds a WASM extension and its
120+
server path adds NodeAV native bindings over FFmpeg. It would remove little application lifecycle
121+
code while weakening subprocess isolation. Rejected for this PR; a metadata-only spike can be
122+
evaluated independently later.
123+
124+
## Rollout and merge gate
125+
126+
The material head requires fresh adversarial review from Via and Rames, all required CI green,
127+
mergeable/dependency-clean state, and no unresolved human feedback. Skill-facing files already in
128+
the PR require Wenbo and Miao visibility before merge.

‎packages/lint/src/hevcPreviewLint.ts‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -145,12 +145,12 @@ export async function lintHevcPreviewCodec(
145145
message:
146146
`Video file(s) use the HEVC/H.265 codec: ${unique.join(", ")}. ` +
147147
"The render pipeline pre-decodes video with FFmpeg and never uses the browser's video decoder, so these render correctly. " +
148-
"Live preview/player playback requires a browser with HEVC support. " +
149-
"If preview playback fails, generate an H.264 proxy (e.g. via the media-use skill) and reference that instead.",
148+
"Live preview/player playback automatically uses a cached H.264 proxy when the browser cannot decode HEVC. " +
149+
"If playback still fails, verify ffmpeg/ffprobe are installed and auto-proxying is enabled.",
150150
fixHint:
151151
unique.length === 1
152-
? `Generate an H.264 proxy for "${unique[0]}" (e.g. via the media-use skill) if it fails to play in preview.`
153-
: "Generate H.264 proxies for these files (e.g. via the media-use skill) if they fail to play in preview.",
152+
? `If "${unique[0]}" fails to play in preview, run hyperframes doctor and confirm media.autoProxy is not false.`
153+
: "If these files fail to play in preview, run hyperframes doctor and confirm media.autoProxy is not false.",
154154
},
155155
];
156156
}

‎packages/lint/src/project.test.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -323,7 +323,7 @@ describe("hevc_preview_codec", () => {
323323
expect(findings[0]?.severity).toBe("info");
324324
expect(findings[0]?.code).toBe("hevc_preview_codec");
325325
expect(findings[0]?.message).toContain("clip.mp4");
326-
expect(findings[0]?.message).toContain("requires a browser with HEVC support");
326+
expect(findings[0]?.message).toContain("automatically uses a cached H.264 proxy");
327327
expect(result.totalErrors).toBe(0);
328328
expect(result.results[0]?.result.ok).toBe(true);
329329
});

‎skills-manifest.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@
3030
"files": 11
3131
},
3232
"hyperframes-core": {
33-
"hash": "546d4b1a0f0a8440",
33+
"hash": "fa201c126c87fd89",
3434
"files": 17
3535
},
3636
"hyperframes-creative": {
@@ -46,7 +46,7 @@
4646
"files": 10
4747
},
4848
"media-use": {
49-
"hash": "36630c2bb9d281da",
49+
"hash": "b036210167a3ca39",
5050
"files": 133
5151
},
5252
"motion-graphics": {

‎skills/hyperframes-core/references/variables-and-media.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,4 +106,4 @@ Video elements must be muted and inline. Audio must be a separate `<audio>` elem
106106

107107
For media duration: `<video>` and `<audio>` can omit `data-duration` if the media's intrinsic length is known and you want the full clip. Otherwise provide `data-duration` explicitly.
108108

109-
Input codecs: render decodes video via FFmpeg (frames are pre-extracted and injected), so HEVC/H.265 assets (8/10-bit) render correctly everywhere; only live preview depends on the browser's codec support (`lint` emits an info-level `hevc_preview_codec` note; use an H.264 proxy for authoring if preview shows black).
109+
Input codecs: render decodes video via FFmpeg (frames are pre-extracted and injected), so HEVC/H.265 assets (8/10-bit) render correctly everywhere; live preview auto-proxies any browser-hostile asset (transcodes and caches an H.264 copy on first use, opt out with `--no-proxy` or `media.autoProxy: false`), and `lint` emits an info-level `hevc_preview_codec` note naming affected assets.

‎skills/media-use/SKILL.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -393,10 +393,11 @@ with `resolve --from <output> --type <type>` so it joins the ledger + global
393393
cache.
394394

395395
HEVC/H.265 sources need no conversion for **render** (FFmpeg pre-decodes all
396-
input video); only live preview depends on the browser's codec support. If an
397-
HEVC asset previews black, make an H.264 authoring proxy (`ffmpeg -i in.mp4
398-
-c:v libx264 -crf 18 proxy.mp4`), register it with `resolve --from`, and keep
399-
either file for the final render.
396+
input video) or for **preview** (auto-proxy transcodes and caches an H.264
397+
copy on first use, disable with `--no-proxy` or `media.autoProxy: false` in
398+
hyperframes.json). A manual H.264 proxy via `ffmpeg -i in.mp4 -c:v libx264
399+
-crf 18 proxy.mp4`, registered with `resolve --from`, remains available for
400+
edge cases (e.g. auto-proxy disabled, or ffmpeg unavailable at preview time).
400401

401402
## CLI tools used (what to run, and how to enable each)
402403

0 commit comments

Comments
 (0)