Skip to content

Commit f235831

Browse files
feat(skills): re-land reverse any video on the HyperFrames plan, with clearer credit wording (#6833)
* fix(cli,skills): say a step's credits are its total, and keep cost out of retries An out-of-credits refusal named what the step needs, and an agent read that as the shortfall. It now says the number is the step's total. After its last try, an agent told the person what was spent and what another try would cost. The HeyGen reference now says not to. * fix(cli): plan avatar credits from the HyperFrames price rows An avatar's credits came from its old API price, about five times what the plan charges, so --max-credits refused calls that fit and a top-up asked for too much. They now use the Avatar IV photo row plus the general avatar video row, 950 credits a minute. * fix(cli): pin the avatar price test and keep pricing under the comment share * fix(cli): plan avatar seconds on the Avatar IV row alone The general avatar row is only checked at admission and billed on seconds left after Avatar IV, so adding it to every second made a 15 s script plan at 238 instead of 200 and a 220 limit refuse a call that fits. * fix(cli): the credits link opens the web plan page, which is live * test(cli): pin the production credits link too * chore(skills): regenerate the skills manifest after merging main * docs(cli): keep the person measure's doc comments to four lines * docs(cli): keep the measure doc comments to four lines * docs(cli): keep the measure comment lines to 120 characters --------- Co-authored-by: Miguel Angel Simon Sierra <[email protected]>
1 parent 5169422 commit f235831

95 files changed

Lines changed: 10051 additions & 528 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.prettierignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,3 +56,4 @@ skills/media-use/SKILL.md
5656
skills/figma/SKILL.md
5757
skills/slideshow/SKILL.md
5858
skills/remotion-to-hyperframes/SKILL.md
59+
skills/reverse-any-video/SKILL.md

‎docs/packages/cli.mdx‎

Lines changed: 73 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,11 +28,12 @@ npm install -g hyperframes
2828
| Bring in source material | [`capture`](#capture), [`media-use`](#media-use), [`transcribe`](#transcribe), [`tts`](#tts), [`remove-background`](#remove-background), [`media-treatment`](#media-treatment), [`beats`](#beats), [`audio`](#audio), [`normalize-audio`](#normalize-audio), [`models`](#models) |
2929
| Look at it, or share it | [`preview`](#preview), [`present`](#present-and-play), [`play`](#present-and-play), [`publish`](#publish), [`open`](#open-and-catch-up), [`catch-up`](#open-and-catch-up) |
3030
| Find problems | [`lint`](#lint), [`check`](#check), [`snapshot`](#snapshot), [`keyframes`](#keyframes), [`compare`](#compare-and-grade-compare), [`grade-compare`](#compare-and-grade-compare) |
31+
| Rebuild a video | [`measure`](#measure), [`diff`](#diff) |
3132
| Edit from the terminal | [`timeline`](#timeline), [`history`](#history) |
3233
| Make a file | [`render`](#render), [`benchmark`](#benchmark) |
3334
| Render somewhere else | [`cloud`](#hyperframes-cloud), [`lambda`](#hyperframes-lambda), [`cloudrun`](#hyperframes-cloudrun) |
3435
| Fix your setup | [`doctor`](#doctor), [`browser`](#browser), [`upgrade`](#upgrade), [`docs`](#docs), [`info`](#info-and-compositions), [`compositions`](#info-and-compositions), [`clean`](#clean) |
35-
| Connect other tools | [`auth`](#hyperframes-auth), [`skills`](#skills), [`usage`](#usage), [`figma`](#figma-and-events), [`events`](#figma-and-events), [`telemetry`](#telemetry), [`feedback`](#feedback) |
36+
| Connect other tools | [`auth`](#hyperframes-auth), [`billing`](#billing), [`skills`](#skills), [`usage`](#usage), [`figma`](#figma-and-events), [`events`](#figma-and-events), [`telemetry`](#telemetry), [`feedback`](#feedback) |
3637

3738
`validate`, `inspect`, and `layout` still work but are
3839
[deprecated](#deprecated-validate-inspect-and-layout). Use `check` instead.
@@ -1504,6 +1505,59 @@ session, and it is where new work should go. Their flags are unchanged — run
15041505
`npx hyperframes inspect --help` if you are maintaining automation that still
15051506
calls them.
15061507

1508+
## Rebuild an existing video
1509+
1510+
These commands serve the `reverse-any-video` skill: measure a video, have HeyGen
1511+
draft its timeline, then score your rebuild against it.
1512+
1513+
### `measure`
1514+
1515+
Measures a video file and prints JSON. Each measurement is a subcommand; run
1516+
`npx hyperframes measure <subcommand> --help` for its flags.
1517+
1518+
```bash
1519+
npx hyperframes measure probe ref.mp4
1520+
npx hyperframes measure cuts ref.mp4
1521+
npx hyperframes measure frames ref.mp4 --events
1522+
npx hyperframes measure face ref.mp4 --start 1 --end 2.5
1523+
```
1524+
1525+
| Subcommand | What it reports |
1526+
| ------------ | --------------------------------------------------------------------------------------- |
1527+
| `probe` | Displayed size, frame rate, duration, sound, and what to fix before rebuilding |
1528+
| `cuts` | Hard cuts from the frame-to-frame difference, and the shots between them |
1529+
| `events` | Cuts, flashes, appears, disappears and holds; `--map` adds the sound's hits |
1530+
| `frames` | Frames as pictures: chosen moments, an overview sheet, a range, or strips around events |
1531+
| `extract` | A still or a clip cut from the video, cropped to a region, as project media |
1532+
| `curve` | Each move of one element: start, duration, overshoot and the closest GSAP ease |
1533+
| `energy` | How much two clips move, and whether at the same moments |
1534+
| `look` | Brightness, contrast, saturation and warmth, alone or against another video |
1535+
| `face` | The face's size and position over a window |
1536+
| `sync` | The lag between mouth motion and the voice |
1537+
1538+
### `diff`
1539+
1540+
Scores a render, or a project folder without rendering it, against the original
1541+
video frame by frame: SSIM, the worst moments, missed and extra cuts, duration
1542+
and the sound's offset. It exits non-zero when the scores fall under the bar.
1543+
1544+
```bash
1545+
npx hyperframes diff original.mp4 renders/final.mp4
1546+
npx hyperframes diff original.mp4 . --at 0.5,2.4,3.1
1547+
npx hyperframes diff original.mp4 renders/final.mp4 --segments segments.md --sbs
1548+
```
1549+
1550+
| Flag | Description |
1551+
| ----------------------------------------- | -------------------------------------------------------------------- |
1552+
| `--at` | Compare only these seconds |
1553+
| `--range` | Compare every frame of `start,end` seconds |
1554+
| `--segments` | A verdict per `## S01 · a–b s` segment of a file |
1555+
| `--drawn` | Score one drawn element's box `x,y,w,h[,a-b]` by itself (repeatable) |
1556+
| `--min-mean` / `--min-p05` / `--min-cell` | The SSIM bars to pass |
1557+
| `--sbs` | Also write an original-beside-yours video |
1558+
| `--out` | Folder for the sheet and log (default `.hyperframes/diff/<yours>`) |
1559+
| `--json` | Print one JSON result |
1560+
15071561
## Edit from the command line
15081562

15091563
### `timeline`
@@ -2178,6 +2232,24 @@ goes only to that provider's own usage endpoint. The command sends no telemetry
21782232
and skips the update check. The numbers are a snapshot: they do not reserve
21792233
allowance or estimate what the next video will cost.
21802234

2235+
### `billing`
2236+
2237+
Opens the HyperFrames plans or credits page in the browser, then waits until the
2238+
account shows the change, so an agent's run carries on by itself. The page is on
2239+
the web app for `HYPERFRAMES_ENV`: hyperframes.studio, or dev.hyperframes.dev
2240+
with `HYPERFRAMES_ENV=dev`.
2241+
2242+
```bash
2243+
npx hyperframes billing subscribe
2244+
npx hyperframes billing top-up --needed 180
2245+
```
2246+
2247+
| Flag | Description |
2248+
| ----------- | -------------------------------------------------------------------------- |
2249+
| `--needed` | Credits the waiting step needs; `top-up` waits until the balance covers it |
2250+
| `--tool` | The step that is waiting, shown on the page |
2251+
| `--timeout` | Seconds to wait (default 600) |
2252+
21812253
### `figma` and `events`
21822254

21832255
`npx hyperframes figma` imports through the Figma REST API, with three

‎packages/cli/src/api/transcribe.ts‎

Lines changed: 74 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import { existsSync, realpathSync, rmSync, statSync, writeFileSync } from "node:fs";
22
import { basename, dirname, extname, join, resolve } from "node:path";
3+
import { transcribeWithHeygen } from "../heygen/transcribe.js";
34
import { normalizeErrorMessage } from "../utils/errorMessage.js";
45
import {
56
createRenderCancellationScope,
@@ -38,7 +39,7 @@ import {
3839
// finishing even on the fastest hardware for the shortest clip.
3940
export const MIN_TIMEOUT_MS = 5000;
4041

41-
export type TranscribeEngine = "auto" | "parakeet" | "whisper";
42+
export type TranscribeEngine = "auto" | "parakeet" | "whisper" | "heygen";
4243
export type CaptionFormat = "srt" | "vtt";
4344
/** The program that does the work: Parakeet runs on sherpa-onnx or parakeet-mlx. */
4445
export type TranscribeRunner = ParakeetRunner | "whisper";
@@ -69,8 +70,10 @@ export interface TranscribeOptions {
6970
input: string;
7071
/** Where transcript.json goes; caption scripts in this folder's .html files are patched. Default: the input's folder. */
7172
dir?: string;
72-
/** Default "auto": Parakeet when installed and it covers `language`, else whisper. */
73+
/** Default "auto": Parakeet when installed and it covers `language`, else whisper; "heygen" runs on HeyGen. */
7374
engine?: TranscribeEngine;
75+
/** HeyGen only: label each word with its speaker. */
76+
speakers?: boolean;
7477
/** Whisper model. Parakeet has its own fixed model. */
7578
model?: string;
7679
language?: string;
@@ -97,7 +100,7 @@ export interface TranscribeOptions {
97100
export type TranscribeResult =
98101
| {
99102
kind: "transcribed";
100-
engine: "parakeet" | "whisper";
103+
engine: "parakeet" | "whisper" | "heygen";
101104
model: string;
102105
detectedLanguage: string | null;
103106
wordCount: number;
@@ -367,6 +370,71 @@ function failureMessage(err: unknown): string {
367370
return stderr ? `${base}\n${stderr}` : base;
368371
}
369372

373+
interface EngineTranscript {
374+
transcriptPath: string;
375+
model: string;
376+
detectedLanguage: string | null;
377+
durationSeconds: number;
378+
speechOnsetSeconds: number | null;
379+
}
380+
381+
/** Every engine's words end the same way: normalized, patched into caption scripts, and as a sidecar if asked. */
382+
function finishTranscript(
383+
result: EngineTranscript,
384+
engine: "parakeet" | "whisper" | "heygen",
385+
dir: string,
386+
sidecar: CaptionSidecar | undefined,
387+
): TranscribeResult {
388+
const { words } = loadTranscript(result.transcriptPath);
389+
390+
writeFileSync(result.transcriptPath, JSON.stringify(words, null, 2));
391+
patchCaptionHtml(dir, words);
392+
if (sidecar) {
393+
try {
394+
writeCaptionSidecar(words, sidecar, false);
395+
} catch (err) {
396+
throw new TranscribeError(
397+
"caption_write_failed",
398+
`Transcript saved to ${result.transcriptPath}, but the caption file ${sidecar.outPath} could not be written: ${normalizeErrorMessage(err)}`,
399+
{ cause: err, transcriptPath: result.transcriptPath },
400+
);
401+
}
402+
}
403+
404+
return {
405+
kind: "transcribed",
406+
engine,
407+
model: result.model,
408+
detectedLanguage: result.detectedLanguage,
409+
wordCount: words.length,
410+
durationSeconds: result.durationSeconds,
411+
speechOnsetSeconds: result.speechOnsetSeconds,
412+
transcriptPath: result.transcriptPath,
413+
...(sidecar && { captions: { format: sidecar.to, outputPath: sidecar.outPath } }),
414+
};
415+
}
416+
417+
async function transcribeOnHeygen(
418+
inputPath: string,
419+
dir: string,
420+
sidecar: CaptionSidecar | undefined,
421+
options: TranscribeRequest,
422+
): Promise<TranscribeResult> {
423+
try {
424+
const result = await transcribeWithHeygen(inputPath, join(dir, TRANSCRIPT_FILE), {
425+
diarize: options.speakers,
426+
projectDir: dir,
427+
onStatus: options.onStatus,
428+
});
429+
return finishTranscript(result, "heygen", dir, sidecar);
430+
} catch (err) {
431+
if (err instanceof TranscribeError) throw err;
432+
const fix =
433+
err instanceof Error && "fix" in err && typeof err.fix === "string" ? ` Fix: ${err.fix}` : "";
434+
throw new TranscribeError("failed", `${normalizeErrorMessage(err)}.${fix}`, { cause: err });
435+
}
436+
}
437+
370438
// fallow-ignore-next-line complexity
371439
async function transcribeAudio(
372440
inputPath: string,
@@ -375,10 +443,11 @@ async function transcribeAudio(
375443
options: TranscribeRequest,
376444
): Promise<TranscribeResult> {
377445
const engine = (options.engine ?? "auto").toLowerCase();
446+
if (engine === "heygen") return transcribeOnHeygen(inputPath, dir, sidecar, options);
378447
if (engine !== "auto" && engine !== "parakeet" && engine !== "whisper") {
379448
throw new TranscribeError(
380449
"invalid_option",
381-
`Unknown --engine: ${options.engine}. Use auto, parakeet, or whisper.`,
450+
`Unknown --engine: ${options.engine}. Use auto, parakeet, whisper, or heygen.`,
382451
);
383452
}
384453
const unsupported = sherpaUnsupportedReason();
@@ -488,33 +557,7 @@ async function transcribeAudio(
488557
}
489558
}
490559

491-
const { words } = loadTranscript(result.transcriptPath);
492-
493-
writeFileSync(result.transcriptPath, JSON.stringify(words, null, 2));
494-
patchCaptionHtml(dir, words);
495-
if (sidecar) {
496-
try {
497-
writeCaptionSidecar(words, sidecar, false);
498-
} catch (err) {
499-
throw new TranscribeError(
500-
"caption_write_failed",
501-
`Transcript saved to ${result.transcriptPath}, but the caption file ${sidecar.outPath} could not be written: ${normalizeErrorMessage(err)}`,
502-
{ cause: err, transcriptPath: result.transcriptPath },
503-
);
504-
}
505-
}
506-
507-
return {
508-
kind: "transcribed",
509-
engine: engineOf(runner),
510-
model: result.model,
511-
detectedLanguage: result.detectedLanguage,
512-
wordCount: words.length,
513-
durationSeconds: result.durationSeconds,
514-
speechOnsetSeconds: result.speechOnsetSeconds,
515-
transcriptPath: result.transcriptPath,
516-
...(sidecar && { captions: { format: sidecar.to, outputPath: sidecar.outPath } }),
517-
};
560+
return finishTranscript(result, engineOf(runner), dir, sidecar);
518561
} catch (err) {
519562
if (err instanceof TranscribeError) throw err;
520563
if (err instanceof DecodeCancelled || cancellation?.signal.aborted) {

‎packages/cli/src/auth/browser.ts‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
/**
2-
* Open a URL in the user's default browser. Falls back to printing the
3-
* URL when no browser is openable (SSH session, CI, `BROWSER=none`,
4-
* or `open` rejects).
2+
* Open a URL in the user's default browser. Falls back to printing the URL when no browser is openable
3+
* (SSH session, CI, `BROWSER=none`, or `open` rejects). ponytail: `HF_BROWSER_LOG` is a test seam for
4+
* scripted journeys: the URL goes to that file, and a harness plays the person in its own browser.
55
*/
66

7+
import { appendFileSync } from "node:fs";
78
import { c } from "../ui/colors.js";
89
import { normalizeErrorMessage } from "../utils/errorMessage.js";
910

@@ -13,6 +14,11 @@ export interface OpenBrowserResult {
1314
}
1415

1516
export async function openBrowser(url: string): Promise<OpenBrowserResult> {
17+
const log = process.env["HF_BROWSER_LOG"];
18+
if (log) {
19+
appendFileSync(log, `${url}\n`);
20+
return { opened: true };
21+
}
1622
if (process.env["BROWSER"] === "none" || process.env["HF_NO_BROWSER"] === "1") {
1723
printManualInstructions(url);
1824
return { opened: false };

‎packages/cli/src/auth/loopback.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ export interface LoopbackHandle {
4545
}
4646

4747
const CALLBACK_PATH = "/oauth/callback";
48-
const DEFAULT_TIMEOUT_MS = 120_000;
48+
const DEFAULT_TIMEOUT_MS = 300_000;
4949

5050
export async function startLoopback(opts: LoopbackOptions): Promise<LoopbackHandle> {
5151
const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS;
@@ -191,7 +191,7 @@ function respond(res: ServerResponse, status: number, body: string): void {
191191
}
192192

193193
function successPage(): string {
194-
return `<!doctype html><html><head><meta charset="utf-8"><title>Signed in to HeyGen</title>
194+
return `<!doctype html><html><head><meta charset="utf-8"><title>Signed in to HyperFrames</title>
195195
<style>body{font-family:system-ui,sans-serif;display:flex;align-items:center;justify-content:center;height:100vh;margin:0;background:#0b0f14;color:#e6e8eb}main{max-width:480px;text-align:center;padding:32px;border-radius:12px;background:#11161d;border:1px solid #1f2630}h1{font-weight:600;margin:0 0 8px;color:#3CE6AC}p{margin:0;color:#9aa3ad}</style>
196196
</head><body><main><h1>You're signed in.</h1><p>You can close this tab and return to your terminal.</p></main></body></html>`;
197197
}

‎packages/cli/src/auth/oauth.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -474,7 +474,9 @@ function buildAuthorizeUrl(args: {
474474
code_challenge: args.challenge,
475475
code_challenge_method: "S256",
476476
});
477-
return `${authorizeEndpoint()}?${params.toString()}`;
477+
const urlWithEndpointQuery = new URL(authorizeEndpoint());
478+
for (const [key, value] of params) urlWithEndpointQuery.searchParams.set(key, value);
479+
return urlWithEndpointQuery.toString();
478480
}
479481

480482
async function exchangeCodeForTokens(args: {

‎packages/cli/src/cli.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -175,6 +175,9 @@ const commandLoaders = {
175175
import("./commands/media-treatment.js").then((m) => m.mediaTreatmentCommand),
176176
"grade-compare": () => import("./commands/grade-compare.js").then((m) => m.default),
177177
compare: () => import("./commands/compare.js").then((m) => m.default),
178+
diff: () => import("./commands/diff.js").then((m) => m.default),
179+
measure: () => import("./commands/measure.js").then((m) => m.default),
180+
billing: () => import("./commands/billing.js").then((m) => m.default),
178181
capture: () => import("./commands/capture.js").then((m) => m.default),
179182
lambda: () => import("./commands/lambda.js").then((m) => m.default),
180183
cloudrun: () => import("./commands/cloudrun.js").then((m) => m.default),

‎packages/cli/src/commands/auth.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ ${c.bold("ENV VARS:")}
4242
${c.accent("HYPERFRAMES_API_KEY")} Alias for HEYGEN_API_KEY.
4343
${c.accent("HEYGEN_ACCESS_TOKEN")} OAuth token from a host app; used after the API keys, never refreshed or saved.
4444
${c.accent("HEYGEN_API_URL")} Override the API base URL (default https://api.heygen.com).
45-
${c.accent("HYPERFRAMES_ENV")} prod (default) or dev; selects OAuth/API hosts together.
45+
${c.accent("HYPERFRAMES_ENV")} prod (default) or dev; selects OAuth, API and plans-page hosts together.
4646
${c.accent("HYPERFRAMES_CONFIG_DIR")} Override the credentials directory (default ~/.hyperframes).
4747
${c.accent("HYPERFRAMES_OAUTH_CLIENT_ID")} Override the OAuth client_id (for dev/test).
4848
${c.accent("HYPERFRAMES_OAUTH_DEVICE_URL")} Override the RFC 8628 device endpoint (for dev/test).

‎packages/cli/src/commands/auth/login.test.ts‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,8 @@ describe("auth login", () => {
109109
"REMOTE_CONTAINERS",
110110
"GITPOD_WORKSPACE_ID",
111111
"container",
112+
"CLAUDECODE",
113+
"CLAUDE_CODE_ENTRYPOINT",
112114
].map((key) => [key, process.env[key]]),
113115
);
114116
for (const key of Object.keys(runtimeEnv)) delete process.env[key];
@@ -130,6 +132,7 @@ describe("auth login", () => {
130132
for (const token of Object.keys(usersByToken)) delete usersByToken[token];
131133
for (const fn of Object.values(telemetry)) fn.mockClear();
132134
for (const fn of Object.values(deviceAuth)) fn.mockClear();
135+
for (const fn of Object.values(browserAuth)) fn.mockClear();
133136
vi.spyOn(console, "log").mockImplementation(() => {});
134137
vi.spyOn(console, "error").mockImplementation(() => {});
135138
});
@@ -317,6 +320,21 @@ describe("auth login", () => {
317320
expect(deviceAuth.start).not.toHaveBeenCalled();
318321
});
319322

323+
it("lets a coding agent on this machine open the browser sign-in without a terminal", async () => {
324+
process.env["CLAUDECODE"] = "1";
325+
Object.defineProperty(process.stdout, "isTTY", { configurable: true, value: undefined });
326+
await runCommand({});
327+
expect(browserAuth.start).toHaveBeenCalledOnce();
328+
});
329+
330+
it("keeps a coding agent in CI off the browser sign-in", async () => {
331+
process.env["CLAUDECODE"] = "1";
332+
process.env["CI"] = "true";
333+
Object.defineProperty(process.stdout, "isTTY", { configurable: true, value: undefined });
334+
await expect(runCommand({})).rejects.toThrow(/Invalid command usage/);
335+
expect(browserAuth.start).not.toHaveBeenCalled();
336+
});
337+
320338
it("opens verification_uri_complete without asking the user to re-enter the code", async () => {
321339
deviceChallenge.verificationUriComplete =
322340
"https://app.heygen.com/oauth/device?user_code=ABCD-2345";

0 commit comments

Comments
 (0)