Skip to content

Commit ca6ea3a

Browse files
refactor(embedded-captions): fetch PP-MattingV2 model on demand instead of shipping in-tree
The 34 MB ppmattingv2 ONNX was committed as a raw blob (added before the *.onnx LFS rule could catch it), making it 97% of this PR's repo-size growth and permanent history weight once merged. Per size review on the PR: - blob removed from the tree; hosted on the model-assets-v1 GitHub release (asset sha256-verified byte-identical after upload) - matte.cjs resolves: MATTE_MODEL env -> legacy bundled copy if present -> ~/.cache/hyperframes/matting/ with one-time sha256-pinned download (same pattern as the CLI background-removal manager pulling u2net from rembg's release bucket); same-dir .part temp + atomic rename - new `matte.cjs --ensure-model` pre-warm flag; SKILL.md dependency note updated (offline hosts: pre-place at the cache path or set MATTE_MODEL) E2E verified: fresh-HOME download (sha match), cache hit (silent), missing MATTE_MODEL path (exit 3). Author-time fetch only — render path untouched. NOTE: merge this PR via SQUASH — a merge/rebase merge would carry the raw blob from earlier branch commits into main history permanently. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent f97fe36 commit ca6ea3a

2 files changed

Lines changed: 51 additions & 14 deletions

File tree

‎skills/embedded-captions/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -294,6 +294,6 @@ track has its own, much simpler spec → **[references/rail.md](references/rail.
294294
- **Transcription = hyperframes' Whisper** (whisper.cpp — native C++, no API key, no Python). `transcribe.cjs` wraps `hyperframes transcribe`; hyperframes auto-installs whisper.cpp (Homebrew or source build) and the model on first run. Falls back to an existing word-level `transcript.json` if present.
295295
- **Source video** — `matte.cjs` / `transcribe.cjs` auto-resolve `source.mp4` (or glob the clip / read `hyperframes.json`), so `hyperframes init --video X.mp4` needs no manual rename.
296296
- **fps** — `matte.cjs` extracts at the source's native rate and records `matte.fps`; `render-and-composite.sh` uses that so the matte stays frame-aligned.
297-
- `assets/ppmattingv2_stdc1_human_544x960.onnx` (34 MB, Apache-2.0) — **ships with the skill** (no auto-download; the upstream ONNX exports have no stable public URL). `MATTE_MODEL` env can point at another `<H>×<W>.onnx` export (e.g. 1088x1920 for max detail); `MATTE_EMA` tunes temporal smoothing (default 0.5).
297+
- **PP-MattingV2 model** (`ppmattingv2_stdc1_human_544x960.onnx`, 34 MB, Apache-2.0) — **downloaded on demand** by `matte.cjs` (one-time, sha256-pinned) from the repo's `model-assets-v1` GitHub release into `~/.cache/hyperframes/matting/` (the upstream ONNX exports have no stable public URL, so we host the exact blob ourselves). Pre-warm with `node scripts/matte.cjs --ensure-model`; offline hosts: pre-place the file at the cache path or set `MATTE_MODEL`. `MATTE_MODEL` env can point at another `<H>×<W>.onnx` export (e.g. 1088x1920 for max detail); `MATTE_EMA` tunes temporal smoothing (default 0.5).
298298

299299
If a hard dependency is missing, STOP and ask the user — don't silently skip steps.

‎skills/embedded-captions/scripts/matte.cjs‎

Lines changed: 50 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,12 @@
1919
* 1 channel silently produces a garbage matte (top-third smeared + striped).
2020
*
2121
* node matte.cjs <project-dir>
22+
* node matte.cjs --ensure-model (pre-warm: download/verify the model, print its path)
2223
* Reads: <project>/source.mp4 OR <project>/frames_bg/f_%04d.png
2324
* Writes: <project>/frames_fg/f_%04d.png (RGBA, subject opaque), <project>/matte.fps
25+
* Model: downloaded on demand (sha256-pinned) to ~/.cache/hyperframes/matting/ from
26+
* the repo's model-assets-v1 release; offline hosts can pre-place the file
27+
* there or set MATTE_MODEL.
2428
* Env: MATTE_MODEL — path to an alternative ppmattingv2 ONNX (dims parsed from the
2529
* `<H>x<W>.onnx` filename suffix, e.g. 1088x1920 for max detail)
2630
* MATTE_EMA — temporal smoothing weight for the NEW frame (default 0.5;
@@ -62,11 +66,19 @@ function hfResolve(pkg) {
6266
const ort = hfResolve("onnxruntime-node");
6367
const sharp = hfResolve("sharp");
6468

65-
// Bundled with the skill (no auto-download: the upstream ONNX exports have no stable
66-
// public URL). MATTE_MODEL may point at any ppmattingv2 `<H>x<W>.onnx` export.
69+
// Model resolution: MATTE_MODEL env → copy bundled next to the skill (legacy
70+
// checkouts) → ~/.cache/hyperframes/matting (downloaded on demand). The upstream
71+
// PaddleSeg ONNX exports have no stable public URL, so the exact blob is hosted on
72+
// the repo's `model-assets-v1` GitHub release (same pattern as the CLI's
73+
// background-removal manager pulling u2net from rembg's release) and sha256-pinned
74+
// here. MATTE_MODEL may point at any ppmattingv2 `<H>x<W>.onnx` export.
75+
const MODEL_NAME = "ppmattingv2_stdc1_human_544x960.onnx";
76+
const MODEL_URL = `https://github.com/heygen-com/hyperframes/releases/download/model-assets-v1/${MODEL_NAME}`;
77+
const MODEL_SHA256 = "75df19af62aaf0453e24444e7ee3cdf8d8de0f628aa9f59c30248cf6220ee9f6";
78+
const BUNDLED_MODEL = path.resolve(__dirname, "..", "assets", MODEL_NAME);
79+
const CACHED_MODEL = path.join(os.homedir(), ".cache", "hyperframes", "matting", MODEL_NAME);
6780
const MODEL =
68-
process.env.MATTE_MODEL ||
69-
path.resolve(__dirname, "..", "assets", "ppmattingv2_stdc1_human_544x960.onnx");
81+
process.env.MATTE_MODEL || (fs.existsSync(BUNDLED_MODEL) ? BUNDLED_MODEL : CACHED_MODEL);
7082
const EMA = (() => {
7183
const v = parseFloat(process.env.MATTE_EMA || "0.5");
7284
return v > 0 && v <= 1 ? v : 0.5;
@@ -80,14 +92,34 @@ function modelDims(modelPath) {
8092
}
8193
return { MH: parseInt(m[1], 10), MW: parseInt(m[2], 10) };
8294
}
83-
function ensureModel() {
95+
async function ensureModel() {
8496
if (fs.existsSync(MODEL)) return;
85-
console.error(`[matte] model missing: ${MODEL}`);
86-
console.error(` The PP-MattingV2 ONNX ships with this skill (assets/). If you moved the`);
87-
console.error(
88-
` skill, restore assets/ppmattingv2_stdc1_human_544x960.onnx or set MATTE_MODEL.`,
89-
);
90-
process.exit(3);
97+
if (MODEL !== CACHED_MODEL) {
98+
console.error(`[matte] model missing: ${MODEL}`);
99+
console.error(` MATTE_MODEL points at a file that does not exist — fix the path, or`);
100+
console.error(` unset it to auto-download the default model.`);
101+
process.exit(3);
102+
}
103+
console.error(`[matte] model not cached — downloading PP-MattingV2 (~34 MB, one-time)…`);
104+
console.error(` ${MODEL_URL}`);
105+
const res = await fetch(MODEL_URL);
106+
if (!res.ok) {
107+
console.error(`[matte] model download failed (HTTP ${res.status}).`);
108+
console.error(` Download it manually to ${CACHED_MODEL}`);
109+
console.error(` or set MATTE_MODEL to a local copy.`);
110+
process.exit(3);
111+
}
112+
const buf = Buffer.from(await res.arrayBuffer());
113+
const sha = require("crypto").createHash("sha256").update(buf).digest("hex");
114+
if (sha !== MODEL_SHA256) {
115+
console.error(`[matte] sha256 mismatch on downloaded model (got ${sha}) — refusing to use it.`);
116+
process.exit(3);
117+
}
118+
fs.mkdirSync(path.dirname(CACHED_MODEL), { recursive: true });
119+
const tmp = `${CACHED_MODEL}.${process.pid}.part`; // same-dir temp → atomic rename
120+
fs.writeFileSync(tmp, buf);
121+
fs.renameSync(tmp, CACHED_MODEL);
122+
console.error(`[matte] model cached ✓ ${CACHED_MODEL}`);
91123
}
92124

93125
function ensureSource(project) {
@@ -158,9 +190,14 @@ function extractFrames(src, dst, fps) {
158190
}
159191

160192
async function main() {
193+
if (process.argv[2] === "--ensure-model") {
194+
await ensureModel();
195+
console.log(MODEL);
196+
return;
197+
}
161198
const project = path.resolve(process.argv[2] || "");
162199
if (!process.argv[2]) {
163-
console.error("usage: matte.cjs <project-dir>");
200+
console.error("usage: matte.cjs <project-dir> | --ensure-model");
164201
process.exit(1);
165202
}
166203
const src = ensureSource(project);
@@ -171,7 +208,7 @@ async function main() {
171208
console.error(`[matte] no source video found in ${project}`);
172209
process.exit(2);
173210
}
174-
ensureModel();
211+
await ensureModel();
175212

176213
if (fs.existsSync(src) && !fs.existsSync(framesBg)) {
177214
const fps = probeFps(src);

0 commit comments

Comments
 (0)