Skip to content

Commit 8089f00

Browse files
feat(skills): soft-launch reverse-any-video, and fix what Wanbi's Pro test found (#7006)
* feat(skills): publish reverse-any-video, render it at high quality, and stop on a stuck shot From Wanbi's Pro test of reverse-any-video (11-01, checked against 0.9.2): - A standard install never found the skill: it was internal, and /hyperframes did not route to it. It is now an on-demand skill on every catalog surface, and the routing line names it. - The cloud render in SKILL.md had no --quality high, so the final rendered at standard. - A re-shoot that stayed pending had no rule; the run waited four hours. people.md now runs the command once more after generate's 30-minute stop, then treats the shot as failed. * feat(cli): generate takes a language for --voice, so a Pro sign-in gets a HeyGen voice Wanbi's Pro run fell back to Kokoro: --voice needed an id, and the only way to list ids was the separate heygen CLI and its own login. /v3/voices answers on the HyperFrames sign-in, so: - --voice <language> picks HeyGen's first public voice in it and names it in the output. - A missing --voice lists six English voices to pick from instead of a bare error. - A voice id still goes straight through, with no extra call. * fix(cli): a voice picked by language stays with its request across re-runs Rames: `--voice Chinese` re-picked HeyGen's first voice on every run, and the voice is part of the body and the idempotency key. If the catalog reordered while a job was pending, the same command could start a second paid job instead of resuming the first. The pick is now bound to the request (every field, plus the account scope a job bills) in the project's heygen-jobs.jsonl, or the CLI config dir outside a project, and a re-run reuses it. Regression: a pending timeout, a reordered catalog, the same command, for voice and avatar; the second POST keeps the voice, body and key. * fix(cli): a voice pick waits for a known account instead of binding to none Rames: when /v3/users/me failed, senderScope() was null and the pick was bound to an empty scope; once the account came back, the same command missed that binding and could pick another voice and paid key. pickVoice now stops before picking or posting while the account is unknown, and never reads or writes a blank-account binding. Regressions, both directions: unknown then known refuses with no POST, then picks once; known then unknown refuses the re-run with no second POST, then resumes with the same body and key. Also README: a non-interactive install gets all 8 skills, not 7.
1 parent d17b958 commit 8089f00

19 files changed

Lines changed: 298 additions & 39 deletions

File tree

‎AGENTS.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ This repo ships AI agent skills via [vercel-labs/skills](https://github.com/verc
88

99
```bash
1010
npx hyperframes skills update # default: installs/refreshes the core set plus anything already installed
11-
npx hyperframes skills # all 7 published skills at once
11+
npx hyperframes skills # all 8 published skills at once
1212
npx skills add heygen-com/hyperframes # interactive picker (terminal only; repo-internal skills are excluded by default)
1313
```
1414

@@ -21,6 +21,7 @@ Make any video by asking for it; the `/hyperframes` skill carries the contract a
2121
- `/figma` — bring Figma into a video: rendered assets, brand tokens, components, storyboard frames as reconstructed motion.
2222
- `/slideshow` — a presentation, pitch deck or interactive deck. Output is a navigable deck, not a rendered MP4.
2323
- `/remotion-to-hyperframes` — bring existing video code or footage into HyperFrames (Remotion first; also Motion Canvas, Lottie, Three.js, canvas, or renders from other tools) and verify it frame by frame against the original.
24+
- `/reverse-any-video` — rebuild any video file as an editable HyperFrames project (motion graphics, talking heads, live action or AI-generated footage) and check the render against the original frame by frame. A HyperFrames Pro feature.
2425

2526
## Issue and PR triage
2627

‎CLAUDE.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,11 @@ Open-source video rendering framework: write HTML, render video.
44

55
## Skills
66

7-
This repo ships 7 AI agent skills via [vercel-labs/skills](https://github.com/vercel-labs/skills). Install them before writing compositions — they encode framework-specific patterns that generic docs don't cover. **Default to the core set** (`/hyperframes`, `/hyperframes-cli`, `/hyperframes-registry`, `/media-use`); `/slideshow`, `/remotion-to-hyperframes` and `/figma` install on demand. Install all 7 only when the user explicitly asks for the full set.
7+
This repo ships 8 AI agent skills via [vercel-labs/skills](https://github.com/vercel-labs/skills). Install them before writing compositions — they encode framework-specific patterns that generic docs don't cover. **Default to the core set** (`/hyperframes`, `/hyperframes-cli`, `/hyperframes-registry`, `/media-use`); `/slideshow`, `/remotion-to-hyperframes`, `/reverse-any-video` and `/figma` install on demand. Install all 8 only when the user explicitly asks for the full set.
88

99
```bash
1010
npx hyperframes skills update # default: installs/refreshes the core set plus anything already installed
11-
npx hyperframes skills # all 7 published skills at once — only on explicit request
11+
npx hyperframes skills # all 8 published skills at once — only on explicit request
1212
npx skills add heygen-com/hyperframes # interactive picker; internal skills require INSTALL_INTERNAL_SKILLS=1 for discovery
1313
npx skills add heygen-com/hyperframes --all # excludes internal skills unless INSTALL_INTERNAL_SKILLS=1
1414
npx skills add heygen-com/hyperframes --skill <name> # a named match installs regardless of metadata.internal (bare name, no leading slash)
@@ -27,6 +27,7 @@ Make any video by asking for it; the `/hyperframes` skill carries the render con
2727
- `/figma` — bring Figma into a video: rendered assets, brand tokens, components, storyboard frames as reconstructed motion, shaders and connector-assisted motion. Use when the person pastes a figma.com link or asks to use a Figma design, frame, logo or brand.
2828
- `/slideshow` — a presentation, pitch deck or interactive deck with discrete slides, fragment reveals, branching, hotspot navigation and presenter mode with speaker notes; also converts an existing page into a deck. Output is a navigable deck, not a rendered MP4.
2929
- `/remotion-to-hyperframes` — bring existing video code or footage into HyperFrames (Remotion first; also Motion Canvas, Lottie, Three.js, canvas, or renders from other tools) and verify it frame by frame against the original.
30+
- `/reverse-any-video` — rebuild any video file as an editable HyperFrames project (motion graphics, talking heads, live action or AI-generated footage) and check the render against the original frame by frame. A HyperFrames Pro feature.
3031

3132
## Skill catalog maintenance
3233

@@ -35,8 +36,8 @@ When adding a new skill, or substantially renaming / repurposing an existing one
3536
1. The skill list above (CLAUDE.md) AND the skill list in the root `AGENTS.md` AND the `## Skills` section in `README.md` AND `docs/guides/skills.mdx` (rendered at [hyperframes.heygen.com/guides/skills](https://hyperframes.heygen.com/guides/skills)) AND the two setup pages that compress the same descriptions — `docs/prompting/overview.mdx` ("One-time setup") and `docs/quickstart.mdx`. Out-of-date entries silently kill discovery. This list is also the sync set for a **changed contract**, not just an added or renamed skill: a reworded `description:` has to be pushed to every surface or the compressed copies start asserting the opposite of the skill.
3637
2. The scaffolded project template `packages/cli/src/templates/_shared/CLAUDE.md` + `AGENTS.md` — written into every `hyperframes init` project, so a stale entry there ships to users. The two template files must stay byte-identical.
3738
3. The routing surface for "make a video" requests is the References table and closing line of `skills/hyperframes/SKILL.md`. A new reference or skill that a request should reach gets a row or a mention there.
38-
4. Keep the same grouping on every surface: the core set (`/hyperframes`, `/hyperframes-cli`, `/hyperframes-registry`, `/media-use`) first, then the on-demand skills (`/figma`, `/slideshow`, `/remotion-to-hyperframes`).
39-
5. Skill count appears in the README and CLAUDE.md intro lines ("7 AI agent skills…") — update on add/remove. The `docs/guides/skills.mdx` page and the CLI templates deliberately omit a count to avoid drift; keep them count-free.
39+
4. Keep the same grouping on every surface: the core set (`/hyperframes`, `/hyperframes-cli`, `/hyperframes-registry`, `/media-use`) first, then the on-demand skills (`/figma`, `/slideshow`, `/remotion-to-hyperframes`, `/reverse-any-video`).
40+
5. Skill count appears in the README and CLAUDE.md intro lines ("8 AI agent skills…") — update on add/remove. The `docs/guides/skills.mdx` page and the CLI templates deliberately omit a count to avoid drift; keep them count-free.
4041

4142
The skill's own `SKILL.md` frontmatter `description:` is the source of truth for the one-line "use when" blurb; copy from there into the catalog rather than paraphrasing.
4243

‎README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ For standalone skills (including OpenCode), use:
6464
npx skills add heygen-com/hyperframes
6565
```
6666

67-
> The picker opens with nothing pre-selected — the **Core Skills** group is all you need; `/slideshow`, `/remotion-to-hyperframes` and `/figma` install on demand. Agents and non-interactive runs should use `npx hyperframes skills update` instead — it installs exactly the core set, whereas `skills add --all` installs all 7 published skills. The six repo-internal skills under `.claude/skills` / `.agents/skills` are excluded by default. For the full published set use `npx hyperframes skills`.
67+
> The picker opens with nothing pre-selected — the **Core Skills** group is all you need; `/slideshow`, `/remotion-to-hyperframes`, `/reverse-any-video` and `/figma` install on demand. Agents and non-interactive runs should use `npx hyperframes skills update` instead — it installs exactly the core set, whereas `skills add --all` installs all 8 published skills. The six repo-internal skills under `.claude/skills` / `.agents/skills` are excluded by default. For the full published set use `npx hyperframes skills`.
6868
>
6969
> `skills add` resolves the skills.sh registry blob, which can lag `main` by hours. `npx hyperframes skills update` installs from the current `main`, so reach for it when you need the newest copy of a skill.
7070
@@ -81,11 +81,11 @@ The skills teach agents the HyperFrames production loop: plan the video, write v
8181

8282
## Skills
8383

84-
HyperFrames ships 7 skills. Make any video by asking for it; the `/hyperframes` skill carries the contract and the references (film, captions, overlays, motion, look). Read it first.
84+
HyperFrames ships 8 skills. Make any video by asking for it; the `/hyperframes` skill carries the contract and the references (film, captions, overlays, motion, look). Read it first.
8585

86-
For standalone installation, default to the **core set** (`/hyperframes`, `/hyperframes-cli`, `/hyperframes-registry`, `/media-use`). `npx hyperframes skills update` installs exactly that from anywhere; the interactive picker (`npx skills add heygen-com/hyperframes`) lists it as the "Core Skills" group, nothing pre-selected. The picker is interactive-only — a non-interactive or agent run without `--skill` installs all 7. Use `npx skills add heygen-com/hyperframes --all` to install the complete published set deliberately (skips the picker), or `npx skills add heygen-com/hyperframes --skill <name>` for just one (bare name, no leading `/`).
86+
For standalone installation, default to the **core set** (`/hyperframes`, `/hyperframes-cli`, `/hyperframes-registry`, `/media-use`). `npx hyperframes skills update` installs exactly that from anywhere; the interactive picker (`npx skills add heygen-com/hyperframes`) lists it as the "Core Skills" group, nothing pre-selected. The picker is interactive-only — a non-interactive or agent run without `--skill` installs all 8. Use `npx skills add heygen-com/hyperframes --all` to install the complete published set deliberately (skips the picker), or `npx skills add heygen-com/hyperframes --skill <name>` for just one (bare name, no leading `/`).
8787

88-
Standalone installs stay lean after that: `npx hyperframes init` installs the **core set** only when it is missing and never touches a skill you already have (`npx hyperframes skills update` refreshes them, and never removes one); `/figma`, `/slideshow` and `/remotion-to-hyperframes` install **on demand** with `npx hyperframes skills update <name>`. Nothing re-pulls the full set behind your back.
88+
Standalone installs stay lean after that: `npx hyperframes init` installs the **core set** only when it is missing and never touches a skill you already have (`npx hyperframes skills update` refreshes them, and never removes one); `/figma`, `/slideshow`, `/remotion-to-hyperframes` and `/reverse-any-video` install **on demand** with `npx hyperframes skills update <name>`. Nothing re-pulls the full set behind your back.
8989

9090
### Plugin packages
9191

@@ -119,6 +119,7 @@ This writes `dist/hyperframes-plugin.zip` with a `hyperframes/` root folder and
119119
| `/figma` | Bring Figma into a video — rendered assets, brand tokens, components, storyboard frames as reconstructed motion, shaders and connector-assisted motion. Use when you paste a figma.com link. |
120120
| `/slideshow` | A presentation, pitch deck or interactive deck — discrete slides, fragment reveals, branching, hotspot navigation, presenter mode with speaker notes. Output is a navigable deck, not a rendered MP4. |
121121
| `/remotion-to-hyperframes` | Bringing existing video code or footage into HyperFrames (Remotion first; also Motion Canvas, Lottie, Three.js, canvas, other tools' renders), verified frame by frame. |
122+
| `/reverse-any-video` | Rebuilding any video file as an editable HyperFrames project (motion graphics, talking heads, live action, AI-generated footage), checked frame by frame against the original. HyperFrames Pro. |
122123

123124
For visual design handoff workflows, see the [Claude Design guide](https://hyperframes.heygen.com/guides/claude-design) and [Open Design guide](https://hyperframes.heygen.com/guides/open-design).
124125

‎docs/guides/skills.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,7 @@ Make any video by asking for it; the `/hyperframes` skill carries the contract a
125125
| `/figma` | A Figma file or frame: rendered assets, brand tokens, components, and storyboard frames as reconstructed motion. |
126126
| `/slideshow` | A presentation, pitch deck or interactive deck. Output is a navigable deck, not a rendered MP4. |
127127
| `/remotion-to-hyperframes` | Bringing existing video code or footage into HyperFrames (Remotion first; also Motion Canvas, Lottie, Three.js, canvas, other tools' renders), verified frame by frame. |
128+
| `/reverse-any-video` | Rebuilding any video file as an editable HyperFrames project (motion graphics, talking heads, live action, AI-generated footage), checked frame by frame against the original. HyperFrames Pro. |
128129

129130
You can name a skill directly, but you do not need to memorize the names to start.
130131

‎packages/cli/src/commands/media-use/generate.ts‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import {
77
checkRequest,
88
generate,
99
parseMask,
10+
pickVoice,
1011
type GenerateRequest,
1112
type GenerateType,
1213
} from "../../heygen/generate.js";
@@ -88,7 +89,10 @@ export default defineCommand({
8889
"first-frame": { type: "string", description: "video: the picture the shot starts from" },
8990
photo: { type: "string", description: "avatar: the person's photo" },
9091
script: { type: "string", description: "avatar, voice: the words spoken" },
91-
voice: { type: "string", description: "avatar, voice: a HeyGen voice id" },
92+
voice: {
93+
type: "string",
94+
description: "avatar, voice: a HeyGen voice id, or a language for its first voice in it",
95+
},
9296
seconds: {
9397
type: "string",
9498
description: "video 5–15 (default 5); music: 30, 60, 120 or 180 (default 30)",
@@ -143,6 +147,8 @@ export default defineCommand({
143147
bpm: num(args.bpm, "--bpm"),
144148
instrumental: !args.vocals,
145149
};
150+
const voice = type === "voice" || type === "avatar" ? await pickVoice(request) : undefined;
151+
if (voice) request.voice = voice.voice_id;
146152
const price = priceOf(request);
147153
const credits = creditsOf(price);
148154
if (args.plan) {
@@ -153,6 +159,7 @@ export default defineCommand({
153159
out: request.out,
154160
price,
155161
credits,
162+
...(voice ? { voice } : {}),
156163
next: "run the same command without --plan to make it",
157164
};
158165
}
@@ -162,7 +169,7 @@ export default defineCommand({
162169
(status) => console.error(`generate: ${status}…`),
163170
credits ?? undefined,
164171
);
165-
return { ...made, price, credits };
172+
return { ...made, price, credits, ...(voice ? { voice } : {}) };
166173
});
167174
},
168175
});

‎packages/cli/src/heygen/generate.test.ts‎

Lines changed: 149 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,9 +14,24 @@ vi.mock("../cloud/auth.js", () => ({
1414

1515
vi.stubEnv("HEYGEN_API_KEY", "k");
1616

17-
const { checkRequest, foldShort, generate, parseMask, ringChange, structureRatio, urlOf } =
18-
await import("./generate.js");
17+
const account = vi.hoisted(() => ({ scope: "acct" as string | null }));
18+
vi.mock("./v3.js", async (load) => ({
19+
...(await load<typeof import("./v3.js")>()),
20+
senderScope: async () => account.scope,
21+
}));
22+
23+
const {
24+
checkRequest,
25+
foldShort,
26+
generate,
27+
parseMask,
28+
pickVoice,
29+
ringChange,
30+
structureRatio,
31+
urlOf,
32+
} = await import("./generate.js");
1933
const { checkLimit, creditsOf, priceOf } = await import("./pricing.js");
34+
type GenerateRequest = import("./generate.js").GenerateRequest;
2035
const ffmpeg = findFFmpeg();
2136
const project = mkdtempSync(join(tmpdir(), "hf-generate-test-"));
2237
writeFileSync(join(project, "index.html"), "<html></html>");
@@ -687,3 +702,135 @@ describe("foldShort", () => {
687702
expect(foldShort([0.3], 10)).toEqual([0.3]);
688703
});
689704
});
705+
706+
describe("pickVoice", () => {
707+
const voice = (id: string, name: string) => ({ voice_id: id.repeat(32), name, gender: "female" });
708+
let catalog = [voice("a", "Lin"), voice("b", "Wei")];
709+
const asked: string[] = [];
710+
const posts: { key: string | null; body: Record<string, unknown> }[] = [];
711+
const heygen = () =>
712+
vi.stubGlobal("fetch", async (url: string, init?: RequestInit) => {
713+
asked.push(url);
714+
if (url.includes("/v3/voices?")) return json({ data: catalog });
715+
if (init?.method === "POST") {
716+
const key = new Headers(init.headers).get("Idempotency-Key");
717+
posts.push({ key, body: JSON.parse(String(init.body)) });
718+
}
719+
return json({ data: { id: "j1", status: "pending" } });
720+
});
721+
afterEach(() => {
722+
vi.useRealTimers();
723+
account.scope = "acct";
724+
catalog = [voice("a", "Lin"), voice("b", "Wei")];
725+
asked.length = 0;
726+
posts.length = 0;
727+
});
728+
729+
it("takes a voice id as given, without asking HeyGen", async () => {
730+
heygen();
731+
const r = { type: "voice" as const, out: "vo.mp3", voice: "c".repeat(32) };
732+
expect((await pickVoice(r)).voice_id).toBe("c".repeat(32));
733+
expect(asked).toEqual([]);
734+
});
735+
736+
it("picks HeyGen's first public voice for a language", async () => {
737+
heygen();
738+
const r = { type: "voice" as const, out: join(project, "assets/a.mp3"), voice: "Chinese" };
739+
expect((await pickVoice(r)).name).toBe("Lin");
740+
expect(asked[0]).toBe("https://api.test/v3/voices?type=public&language=Chinese&limit=1");
741+
});
742+
743+
it("lists voices to pick from when none is named", async () => {
744+
heygen();
745+
await expect(pickVoice({ type: "voice", out: "vo.mp3" })).rejects.toMatchObject({
746+
message: "--voice is missing",
747+
fix: expect.stringContaining(`${"a".repeat(32)} (Lin, female)`),
748+
});
749+
});
750+
751+
// The command's own path: pick, then generate; a timeout, a reordered catalog, the same command.
752+
const runOnce = async (r: GenerateRequest) => {
753+
const request = { ...r };
754+
const picked = await pickVoice(request).catch((err: unknown) => err);
755+
if (picked instanceof Error) return picked;
756+
request.voice = (picked as { voice_id: string }).voice_id;
757+
vi.useFakeTimers({ toFake: ["Date", "setTimeout"] });
758+
const run = generate(request).catch((err: unknown) => err);
759+
await vi.advanceTimersByTimeAsync(31 * 60_000);
760+
vi.useRealTimers();
761+
return run;
762+
};
763+
764+
for (const [type, extra] of [
765+
["voice", "mp3"],
766+
["avatar", "mp4"],
767+
] as const) {
768+
it.skipIf(!ffmpeg)(
769+
`keeps the first voice for a ${type} re-run after the catalog reorders`,
770+
async () => {
771+
heygen();
772+
const face = join(project, "face.png");
773+
execFileSync(ffmpeg!, [
774+
"-v",
775+
"error",
776+
"-y",
777+
"-f",
778+
"lavfi",
779+
"-i",
780+
"color=red:s=64x64",
781+
"-frames:v",
782+
"1",
783+
face,
784+
]);
785+
const r: GenerateRequest = {
786+
type,
787+
out: join(project, `assets/${type}-retry.${extra}`),
788+
voice: "Chinese",
789+
script: "你好",
790+
...(type === "avatar" ? { photo: face } : {}),
791+
};
792+
expect(await runOnce(r)).toMatchObject({ message: "HeyGen job j1 is still pending" });
793+
catalog = [voice("b", "Wei"), voice("a", "Lin")];
794+
await runOnce(r);
795+
expect(posts).toHaveLength(2);
796+
expect(posts[1]!.key).toBe(posts[0]!.key);
797+
expect(posts[1]!.body).toEqual(posts[0]!.body);
798+
expect(posts[0]!.body.voice_id).toBe("a".repeat(32));
799+
},
800+
);
801+
}
802+
const shot = (name: string): GenerateRequest => ({
803+
type: "voice",
804+
out: join(project, `assets/${name}.mp3`),
805+
voice: "Chinese",
806+
script: "你好",
807+
});
808+
809+
it("refuses before any pick or POST while the account is unknown, then picks once it is", async () => {
810+
heygen();
811+
account.scope = null;
812+
expect(await runOnce(shot("who-later"))).toMatchObject({
813+
message: expect.stringContaining("did not say which account"),
814+
});
815+
expect(posts).toHaveLength(0);
816+
expect(asked).toEqual([]);
817+
account.scope = "acct";
818+
await runOnce(shot("who-later"));
819+
expect(posts.map((p) => p.body.voice_id)).toEqual(["a".repeat(32)]);
820+
});
821+
822+
it("refuses a re-run while the account is unknown instead of starting another job", async () => {
823+
heygen();
824+
await runOnce(shot("who-lost"));
825+
account.scope = null;
826+
catalog = [voice("b", "Wei"), voice("a", "Lin")];
827+
expect(await runOnce(shot("who-lost"))).toMatchObject({
828+
message: expect.stringContaining("did not say which account"),
829+
});
830+
expect(posts).toHaveLength(1);
831+
account.scope = "acct";
832+
await runOnce(shot("who-lost"));
833+
expect(posts[1]!.key).toBe(posts[0]!.key);
834+
expect(posts[1]!.body).toEqual(posts[0]!.body);
835+
});
836+
});

0 commit comments

Comments
 (0)