Skip to content

fix(producer): convert HEIC images to JPEG at compile time - #6455

Merged
miguel-heygen merged 4 commits into
mainfrom
fix/heic-compile-time
Oct 26, 2026
Merged

miguel-heygen merged 4 commits into
mainfrom
fix/heic-compile-time

Conversation

@miguel-heygen

Copy link
Copy Markdown
Collaborator

What

HEIC and HEIF images now work in every render path. The compile step converts each one to a JPEG before the page loads and rewrites the <img src>, so the CLI, Studio preview, Lambda and cloud renders all get it from one place.

No visible change

Compositions without HEIC images compile exactly as before. A HEIC image that used to fail the render now shows as a normal picture.

Why

Chrome cannot decode HEIC, and a tiled iPhone photo also reads as several video streams. The media-type check then rejects the <img> and the render fails at the plan step. On main, with a real tiled HEIC:

THROW ASSET_MEDIA_TYPE_MISMATCH 1 composition asset(s) do not match their authored media element type (expected: image) [{"expected":"image","detected":"video"}]

The media probe reports the file as visualKind: "moving". A real CLI render on main fails with the same message (exit 1).

Related work

Follows the same shape as the animated-GIF preparation (animatedGifPrep.ts), which is the existing owner of "turn an unsupported <img> into something Chrome can show".

How

  • New heifImagePrep.ts sits next to the GIF prep and is wired in the same two places: htmlCompiler.ts (renders) and Studio's preview transform.
  • A source counts as HEIC by its content (ftyp brands heic, heix, hevc, hevx, heim, heis, mif1, msf1; AVIF is excluded), not its extension. Covered: a .heic file, a mislabeled .jpg, a data:image/heic;base64 source, and a remote source already downloaded by the existing localisation pass.
  • ffmpeg alone does not do it: through ffmpeg 8.0 a tiled HEIC decodes as separate tile streams and the file's rotation is not applied (one tile of 512x512 comes out). So the grid is stacked with xstack from the tile layout ffprobe -show_stream_groups reports, cropped to the declared size, and rotated by the primary item's irot box (heifRotation.ts). The result must equal the declared size (swapped for a quarter turn) or the conversion fails, so a lone tile can never pass.
  • Each unique source converts once (content hash, with the same cache directory the GIF prep uses), through the shared media-probe slots. A composition with no <img> never reaches the parse; one with only non-HEIC images costs a 4 KB header read per local image.
  • A file that cannot be converted fails with a typed error that names the file and says to re-export it; it is not retried (HEIF_IMAGE_CONVERSION_FAILED).
  • runFfprobe is now exported from the engine and the source resolver in animatedGifPrep.ts is shared, so there is one resolver.

Design calls: (1) a sibling of the GIF prep, not a new stage: it is the existing owner of this kind of rewrite and its two callers; (2) JPEG output only, since there is no alpha HEIC fixture to prove a PNG path; (3) tiles joined by us, because ffmpeg does not do it.

Test plan

  • Unit tests added: 25 new tests pass 3 runs in a row with real ffmpeg (tiled + rotated fixture with pixel checks derived from decoding each tile separately, data URL, mislabeled .jpg, remote-style source, dedupe, corrupt file names the file, no-HEIC spawns nothing, compile no longer fails the media-type check); breaking rotation or the tile join fails 3 tests each
  • Manual testing performed: a 1100x700 composition showing the fixture rendered with the CLI (exit 0); the extracted frame is portrait, upright, whole, continuous across tile seams
  • Documentation updated (not applicable)
  • Comments follow CONTRIBUTING.md

Not covered: HEIC with an alpha plane (always JPEG), a single non-tiled HEIC that carries a rotation (relies on ffmpeg's own handling), a mirror (imir) property, CSS url() and srcset. The tile join needs ffprobe 7.0 or newer for -show_stream_groups; the arm64 Lambda image builds ffmpeg 7.0.2 and the GCP and CLI Docker images ship a BtbN 8.0.1 build, all with the hevc decoder, while the x86 Lambda zip uses the npm ffmpeg-static binary and is worth checking.

Tiled HEIC photos probe as several video streams, so the media-type
preflight rejected them as image:video, and Chrome cannot decode HEIC.
Sniff the ftyp brands of every <img>, join and rotate the tile grid with
ffmpeg, verify the declared size, and swap in a cached JPEG.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES · 1 blocker · 2 important · 1 nit · head cb913b37

  • blocker: the required Producer: unit tests job fails (CI run 40368001169). heifImagePrep.test.ts and htmlCompiler.heif.test.ts spawn ffmpeg/ffprobe, but they are not listed in INTEGRATION_TEST_FILES (packages/producer/scripts/test-classification.mjs). The unit lane does not install FFmpeg, so every conversion case fails with [FFmpeg] ffprobe not found. Add both files to the list. The integration lane installs BtbN 8.0.1, which has -show_stream_groups. Editing that file also means updating its sha256 in scripts/test-reachability.json:50.
  • important: environment failures blame the photo (heifImagePrep.ts:38-44). A missing ffprobe, or one older than 7.0, fails here with owner: "user" and the advice "Re-export the photo as JPEG or PNG". Older versions reject -show_stream_groups: ffprobe 5.1 fails with Failed to set value '-show_streams' for option 'show_stream_groups': Option not found. Those are exactly the x86 Lambda zip (ffprobe-static ^3.1.0) and stock Debian/Ubuntu hosts. Re-exporting cannot fix that. Report a tool/version error as a platform failure instead.
  • important: one failed HEIC drops the whole Studio preview transform (studioServer.ts:579). The throw rejects transformPreviewHtml, and studio-server/src/routes/preview.ts:400 then falls back to the untransformed HTML. The preview therefore also loses GIF prep and the deterministic font faces, where today only that <img> is broken. Catch HeifImageConversionError in the preview path and leave that src as it is.
  • nit: heifImagePrep.ts:145 takes stream_groups[0]. FFmpeg marks the primary item's grid with disposition.default (mov_parse_tiles). If a file lists an auxiliary grid (for example an HDR gain map) first, the wrong image is converted and still passes the size check, because that check uses the same grid's declared size. Select the group whose disposition is default.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 167 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

… image as authored

Also: HEIC tests run in the integration lane; the primary grid is picked by default disposition.
@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

Fixed in 345220b:

  • Blocker: heifImagePrep.test.ts and htmlCompiler.heif.test.ts are now in INTEGRATION_TEST_FILES, and the sha256 in scripts/test-reachability.json is updated. The reachability check and the classification tests pass.
  • Host tools: a missing ffprobe, or one without -show_stream_groups, is now HeifToolUnavailableError (HEIF_TOOL_UNAVAILABLE, owner system, added to the safe error codes). Its message names the file and says HEIC needs ffmpeg and ffprobe 7.0 or newer; it no longer says to re-export the photo. A test covers the missing and the old-ffprobe messages.
  • Studio preview: prepareHeifImageInputs takes an onFailure callback. Studio passes one that logs a warning, so a failed image stays as authored and the GIF prep and font faces still run. Renders pass none and still fail with the named error. A test covers one bad and one good image in the same HTML.
  • Nit: the grid is the group whose disposition is default, with the first group as fallback. A test covers an auxiliary grid listed first.

28 tests pass 3 runs in a row; producer typecheck is clean.

@miguel-heygen
miguel-heygen merged commit ad4d3fa into main Oct 26, 2026
73 checks passed
@miguel-heygen
miguel-heygen deleted the fix/heic-compile-time branch October 26, 2026 17:56
SanHsien added a commit to SanHsien/hyperframes that referenced this pull request Oct 26, 2026
Merge 20 more upstream commits (acd0bc3..ad4d3fa, v0.8.205: heygen-com#6403,
heygen-com#6363, heygen-com#6209, heygen-com#6395, heygen-com#6424, heygen-com#6426, heygen-com#6442, heygen-com#6430, heygen-com#6402, heygen-com#6435, heygen-com#6439,
heygen-com#6449, heygen-com#5436, heygen-com#6455 and docs). No conflicts. Record the 11 open PRs
heygen-com#6440-heygen-com#6458 and issue heygen-com#6443 in docs/DECISIONS.md (none adopted), re-check
the earlier open PRs, and advance the commit watermark to ad4d3fa, the
PR watermark to heygen-com#6458 and the issue watermark to heygen-com#6443.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant