Skip to content

feat(cli): info --media lists every clip the render hears, on the film's clock - #7235

Merged
miguel-heygen merged 3 commits into
mainfrom
feat/cli-info-media
Nov 5, 2026
Merged

miguel-heygen merged 3 commits into
mainfrom
feat/cli-info-media

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

What

hyperframes info --media --json adds heard: every clip the render mixes into the film, scenes included, on the film's clock. Each entry has src, start, end, mediaStart, playbackRate, volume and type. Text mode prints one "Heard" line per clip. Without --media, info is unchanged.

Why

A tool that needs to know which clips the viewer hears (captions, for one) had to walk the HTML itself, and a second walker drifts from the render. Two shapes it got wrong:

  • a speaking clip inside a timed wrapper in a scene;
  • a speaking clip with no data-duration, which runs until its media ends.

The render already knows the answer. This exposes it from the CLI, so callers read the same list the mix uses instead of rebuilding it.

How

heardClips(projectDir) in utils/projectMedia.ts (the file check already uses to load the producer) bundles the project with core's bundleToSingleHtml (scenes inlined, nested src rebased) and runs the producer's collectRenderMedia over it: the same parser the render uses, so the same rule for what is audible (<audio>, and a <video data-has-audio="true"> that is not muted) and the same clock.

The collector ends an unbounded clip at its scene's end. The mix stops earlier if the media runs out, so each clip's end becomes min(end, start + (mediaSeconds - mediaStart) / rate), with the length probed by the engine's extractAudioMetadata. A clip whose media start is past its file's end is left out. Each file is probed once. A looping clip, a ramped rate or a file that cannot be probed keeps the collector's end.

Test plan

  • Unit tests added/updated: projectMedia.test.ts, four tests on a real bundled project with a fixed probe:
    • "hears a speaking video inside a timed wrapper in a scene, where the render plays it": a wrapper at 1 s in a scene at 10 s, the clip is heard 10 s to 15 s.
    • fails without the fix: projectMedia.test.ts "ends a speaking clip with no set length where its media runs out, not where its scene ends": 12 s to 17 s, not to the scene's end at 30 s. It fails with the probe cap removed.
    • "hears a video only when it says data-has-audio and is not muted, as the render does".
    • fails without the fix: projectMedia.test.ts "leaves out a clip that starts past its media's end, and reads each file's length once": a clip whose media start is past its file's end adds nothing to the mix, so it is not listed, and two clips of one file probe it once. It fails with either change taken out.
    • info.test.ts passes media: false in its args. The touched files' tests (17) pass three runs in a row.
  • Manual testing performed: the built CLI on a project with a real mp4 (sine audio) in both shapes. info --media --json prints only JSON, with heard holding the clip at 10 s to 15 s and the one at 22 s to 27 s; info --json has no heard key; text mode prints both "Heard" lines.
  • Documentation updated (if applicable): the info section of docs/packages/cli.mdx.
  • Comments follow CONTRIBUTING.md "Comments": one doc line on HeardClip and one on heardClips.

Size

About 55 lines of source in two files, about 85 of tests and three doc lines. One change: info can print the render's audible clips.

@mintlify

mintlify Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
hyperframes 🟢 Ready View Preview Nov 5, 2026, 11:53 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at db86286c. No blockers.

What I checked:

  • Fails without the fix: I removed the probe cap in endedClip (returned heard before the probe). projectMedia.test.ts > ends a speaking clip with no set length where its media runs out, not where its scene ends fails with end: 30 instead of 17. It passes at this head.
  • Suites: projectMedia.test.ts + info.test.ts 16/16. tsc reports nothing in projectMedia.ts or info.ts; the only errors locally come from the unbuilt inspector package.
  • Real CLI run (tsx src/cli.ts, pinned ffmpeg 8): a project with a real 5 s mp4 (sine audio) used as a root <audio>, a scene at 10 s holding a wrapped clip, an unbounded clip with data-media-start="1", and a muted clip. info --media --json gives bed 0–3 (volume 0.5), a-audio 10–15 (capped by the media from data-duration=9), b-audio 22–26 (5 s − 1 s media start). The muted clip is correctly absent. Text mode prints three Heard lines, and info --json without --media has no heard key.

Non-blocking:

  1. heardClips repeats collectProjectMedia. It reimplements the same bundle + loadProducer() sequence in the same file, but without that function's collectRenderMedia guard. If the loaded producer lacks the collector, you get is not a function instead of Render media collector unavailable. (await collectProjectMedia(projectDir)).audios would give one path and the clearer error.
  2. A clip that starts past its media's end is still listed. When seconds <= mediaStart, the clip keeps the collector's end and shows up as heard, even though the render plays nothing from it. For a captions consumer, dropping it (or ending it at start) would be more accurate.
  3. The probes aren't deduplicated. Promise.all probes every clip at once, once per clip even for the same src (three probes of one file in my run). Caching by path would help projects with many cuts of one recording.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 267 (base branch 267), smooth 100 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (8)

Unstable (1)

  • seqrotatenudge-none-px-r0-root-z100: tracking 0.03, pressJump 0, drop 0, reload 3.07, render 0.04, renderKey -, undo false, teleport true / tracking 0.03, pressJump 0, drop 0, reload 0, render 0.08, renderKey -, undo true, teleport true / tracking 0.03, pressJump 0, drop 0, reload 0, render 0.08, renderKey -, undo true, teleport true

@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

All three notes taken at c8da1fb:

  1. heardClips now reads (await collectProjectMedia(projectDir)).audios: one path, and its Render media collector unavailable error.
  2. A clip whose media start is past its file's end is left out: the mixer extracts from that start, so it adds nothing. The new test fails with it kept.
  3. Each file is probed once per call (a map of probe promises by path). The same test counts one probe for two clips of one file and fails with the cache removed.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 30d0eb60. One correction to the handoff first: this head is not "otherwise unchanged". Compared with db86286c, it changes behaviour in heardClips and adds a test, answering all three of my earlier notes. I reviewed it as a code change:

  1. Duplicate media collection: fixed. heardClips now goes through collectProjectMedia, so it keeps that function's collectRenderMedia guard and error.
  2. Clip starting past its media: fixed. A clip whose mediaStart is at or beyond the probed length is now dropped (return null, then filtered out). A failed probe still keeps the clip at the collector's end, which is the right fallback.
  3. Repeated probes: fixed. Lengths are memoised per resolved path as a shared promise, so concurrent clips of one file trigger a single probe.

Checked:

  • projectMedia.test.ts + info.test.ts: 17/17, including the new leaves out a clip that starts past its media's end, and reads each file's length once.
  • tsc is clean for the changed files, and oxfmt --check passes.
  • Real CLI run on my earlier project (real 5 s mp4): the same three Heard lines as before (0–3, 10–15, 22–26).

@miguel-heygen
miguel-heygen added this pull request to the merge queue Nov 5, 2026
Merged via the queue into main with commit df54f38 Nov 5, 2026
85 checks passed
@miguel-heygen
miguel-heygen deleted the feat/cli-info-media branch November 5, 2026 14:28
SanHsien added a commit to SanHsien/hyperframes that referenced this pull request Nov 6, 2026
…eview

Adopts or partially adopts 11 groups: multi-worker tween replay behind
the timeline stand-in (heygen-com#7231), template entries in the shared compile
stage (heygen-com#7217), HYPERFRAMES_CACHE_DIR and HYPERFRAMES_CONFIG_DIR for
every per-user cache and settings file (heygen-com#7224, heygen-com#7229; HeyGen
credentials stay under HEYGEN_CONFIG_DIR), WebKit audio clock (heygen-com#7225),
packet-based video length (heygen-com#7233), HLG tone mapping (heygen-com#7234), fetch
network cause (heygen-com#7236), .hyperframesignore in lint (heygen-com#7239), info
--media (heygen-com#7235), lint priority (heygen-com#7244), check --budget (heygen-com#7245) and the
unknown-command answer (heygen-com#7247). bun.lock gains only the lint `ignore`
edge from heygen-com#7239, already resolved. Watermark advances to 0a6746e.

Co-authored-by: Jerrai <[email protected]>
Co-authored-by: Miguel Ángel <[email protected]>
Co-authored-by: WaterrrForever <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant