Skip to content

Commit ae7e530

Browse files
feat(cli): capture reports why a referenced asset is not in the folder (#3598)
Capture drops assets for four reasons and reported none of them, so a folder with thirty images and a folder truncated to thirty images were the same object. Every drop site was a bare `continue`, `break`, `return null` or an empty `catch`, and the only signals downstream were two hand-written warning strings that fired when the budget was already gone before a download pass started, which is the one case where the pass could not say how much it lost. `downloadAssets` and `downloadAndRewriteFonts` now return an `AssetDropCounts` tally beside their result, incremented at the single line that performs each drop. `capture --json` carries it as `dropped`; the human summary prints a `Dropped:` line when it is non-zero. Four reasons, three decisions and one failure: size-floor fetched, then judged too small to be a real asset budget-exhausted the post-navigation clock ran out before this one cap-reached 30 inline SVGs, 30 fonts, or 6 faces per family unavailable the request or the write failed A break now counts everything it did not reach rather than the one it stopped on, because "how many did we lose" is the question and one is never the answer. The two budget warnings are gone. Both existed only to cover the case where the budget ran out before a pass was called, so both passes are now called unconditionally: a zero budget makes each loop break on its first item and record `budget-exhausted` for the rest, which costs no network and produces a real number instead of the word "some". The single remaining warning is derived from the tally, so the prose and the count cannot disagree. Measured on a live capture of a large marketing site: default budget 232 kept, 91 dropped (39 size-floor, 20 cap-reached, 32 unavailable) 15s budget 30 kept, 299 dropped (279 budget-exhausted, 20 cap-reached) Same page, same command, and until now both runs described themselves the same way.
1 parent 84ed587 commit ae7e530

6 files changed

Lines changed: 358 additions & 61 deletions

File tree

‎docs/packages/cli.mdx‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -186,10 +186,35 @@ npx hyperframes capture https://example.com --json
186186
187187
Screenshots: 12
188188
Assets: 45
189+
Dropped: 9 (6 size-floor, 3 cap-reached)
189190
Sections: 15
190191
Fonts: sohne-var
191192
```
192193

194+
`Dropped` is how many assets the page referenced that are **not** in the
195+
folder, and why. It is printed only when it is non-zero, and `--json` always
196+
carries it as a `dropped` object. Without it, a capture of a spare page and a
197+
capture that a limit truncated are the same three-line summary, and the only
198+
way to tell them apart is to open the page yourself.
199+
200+
| Reason | What it means |
201+
| ------------------ | ------------------------------------------------------------------------------------------------------ |
202+
| `size-floor` | Fetched, then judged too small to be a real asset rather than a spacer or tracking pixel. |
203+
| `budget-exhausted` | `--capture-budget` ran out before this one was reached. Raise it, or pass `--skip-vision` to buy time. |
204+
| `cap-reached` | A per-run or per-family limit was already met: 30 inline SVGs, 30 fonts, 6 faces per family. |
205+
| `unavailable` | The request or the write failed: network error, timeout, refused address, bad status, disk. |
206+
207+
Three of those are decisions the capture made and one is a failure it hit, so a
208+
run that is thin with an all-zero `dropped` is a thin page, and a run that is
209+
thin with counts on it was cut short.
210+
211+
```json
212+
{
213+
"assets": 45,
214+
"dropped": { "size-floor": 6, "budget-exhausted": 0, "cap-reached": 3, "unavailable": 0 }
215+
}
216+
```
217+
193218
| Flag | Description |
194219
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
195220
| `--output, -o` | Output directory. Default `./capture`, then `./capture-2/`, `./capture-3/`, … if that name is taken. |
@@ -207,7 +232,8 @@ metadata, and contact sheets — plus whatever Lottie, video, and WebGL context
207232
the page exposed. It is raw material for an agent, not a finished composition;
208233
the `/product-launch-video` workflow uses it when a real product has to appear
209234
on screen. Dynamic sites, protected pages, and unusual media loaders produce
210-
partial results, so read the warnings and contact sheets before you build.
235+
partial results, so read `dropped`, the warnings, and the contact sheets before
236+
you build.
211237

212238
For AI image descriptions, set `GEMINI_API_KEY` in a `.env` file
213239
(~$0.001/image), or `OPENROUTER_API_KEY` to route any vision model through

‎packages/cli/src/capture/assetDownloader.test.ts‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,12 @@ import { tmpdir } from "node:os";
44
import { join } from "node:path";
55
import {
66
downloadAndRewriteFonts,
7+
downloadAssets,
78
isPrivateUrl,
89
safeFetch,
910
toStandaloneSvg,
1011
} from "./assetDownloader.js";
12+
import type { DesignTokens } from "./types.js";
1113

1214
describe("isPrivateUrl — SSRF denylist (security: F-003)", () => {
1315
it("blocks loopback, private, and metadata IPv4", () => {
@@ -184,3 +186,145 @@ describe("downloadAndRewriteFonts — attempt caps", () => {
184186
}
185187
});
186188
});
189+
190+
describe("drop counts — why a referenced asset is not in the capture", () => {
191+
afterEach(() => vi.unstubAllGlobals());
192+
193+
/** `n` @font-face rules, each naming a DIFFERENT family, so only the global cap can bite. */
194+
function fontCss(n: number): string {
195+
return Array.from(
196+
{ length: n },
197+
(_, i) =>
198+
`@font-face { font-family: Family${i}; src: url(https://fonts${i}.example/font-${i}.woff2); }`,
199+
).join("\n");
200+
}
201+
202+
function withTempDir<T>(run: (dir: string) => Promise<T>): Promise<T> {
203+
const dir = mkdtempSync(join(tmpdir(), "hf-drops-"));
204+
return run(dir).finally(() => rmSync(dir, { recursive: true, force: true }));
205+
}
206+
207+
it("counts every face the budget never let it reach, not just the one it stopped on", async () => {
208+
// Four declared, zero budget: the honest number is four, and a warning string could only
209+
// ever have said "some".
210+
await withTempDir(async (dir) => {
211+
const fetchMock = vi.fn();
212+
vi.stubGlobal("fetch", fetchMock);
213+
const { drops } = await downloadAndRewriteFonts(fontCss(4), dir, { remainingMs: () => 0 });
214+
expect(fetchMock).not.toHaveBeenCalled();
215+
expect(drops["budget-exhausted"]).toBe(4);
216+
expect(drops["cap-reached"]).toBe(0);
217+
expect(drops.unavailable).toBe(0);
218+
});
219+
});
220+
221+
it("separates the faces the global cap refused from the ones that failed", async () => {
222+
// 35 declared, cap 30: 30 are attempted and every attempt 503s, 5 are never reached.
223+
await withTempDir(async (dir) => {
224+
vi.stubGlobal(
225+
"fetch",
226+
vi.fn(async () => new Response("no", { status: 503 })),
227+
);
228+
const { drops } = await downloadAndRewriteFonts(fontCss(35), dir);
229+
expect(drops["cap-reached"]).toBe(5);
230+
expect(drops.unavailable).toBe(30);
231+
expect(drops["budget-exhausted"]).toBe(0);
232+
});
233+
});
234+
235+
it("counts the faces the per-family cap refused", async () => {
236+
// 10 rules, all one family, per-family cap 6: 6 attempted, 4 refused.
237+
await withTempDir(async (dir) => {
238+
vi.stubGlobal(
239+
"fetch",
240+
vi.fn(async () => new Response("no", { status: 503 })),
241+
);
242+
const css = Array.from(
243+
{ length: 10 },
244+
(_, i) =>
245+
`@font-face { font-family: Shared; src: url(https://fonts.example/f-${i}.woff2); }`,
246+
).join("\n");
247+
const { drops } = await downloadAndRewriteFonts(css, dir);
248+
expect(drops["cap-reached"]).toBe(4);
249+
expect(drops.unavailable).toBe(6);
250+
});
251+
});
252+
253+
it("reports all zeroes when nothing was refused, which is what makes thin readable", async () => {
254+
// The whole point of the tally: this page declared one face and we have it. A reader can now
255+
// tell this apart from a page that declared thirty and got truncated to one.
256+
await withTempDir(async (dir) => {
257+
vi.stubGlobal(
258+
"fetch",
259+
vi.fn(async () => new Response(new Uint8Array(2048), { status: 200 })),
260+
);
261+
const { css, drops } = await downloadAndRewriteFonts(fontCss(1), dir);
262+
expect(css).toContain("assets/fonts/font-0.woff2");
263+
expect(drops).toEqual({
264+
"size-floor": 0,
265+
"budget-exhausted": 0,
266+
"cap-reached": 0,
267+
unavailable: 0,
268+
});
269+
});
270+
});
271+
272+
/** The two fields `downloadAssets` reads off the token bundle. */
273+
function tokensWithNoSvgs(): DesignTokens {
274+
return { svgs: [], sections: [], ogImage: "" } as unknown as DesignTokens;
275+
}
276+
277+
it("counts an image dropped for being under the raster floor", async () => {
278+
// 9 KB is under the 10 KB floor. Nothing lands, and the reason is now on the record.
279+
await withTempDir(async (dir) => {
280+
vi.stubGlobal(
281+
"fetch",
282+
vi.fn(async () => new Response(new Uint8Array(9000), { status: 200 })),
283+
);
284+
const { assets, drops } = await downloadAssets(tokensWithNoSvgs(), dir, [
285+
{ type: "Image", url: "https://cdn.example/hero.png", contexts: ["img[src]"] },
286+
] as never);
287+
expect(assets).toEqual([]);
288+
expect(drops["size-floor"]).toBe(1);
289+
expect(drops.unavailable).toBe(0);
290+
});
291+
});
292+
293+
it("counts every catalogued image the budget never let it reach", async () => {
294+
await withTempDir(async (dir) => {
295+
const fetchMock = vi.fn();
296+
vi.stubGlobal("fetch", fetchMock);
297+
const catalog = Array.from({ length: 7 }, (_, i) => ({
298+
type: "Image",
299+
url: `https://cdn.example/img-${i}.png`,
300+
contexts: ["img[src]"],
301+
}));
302+
const { assets, drops } = await downloadAssets(
303+
tokensWithNoSvgs(),
304+
dir,
305+
catalog as never,
306+
[],
307+
{ remainingMs: () => 0 },
308+
);
309+
expect(fetchMock).not.toHaveBeenCalled();
310+
expect(assets).toEqual([]);
311+
expect(drops["budget-exhausted"]).toBe(7);
312+
});
313+
});
314+
315+
it("counts the inline SVGs the 30-per-run cap refused", async () => {
316+
// 34 inline SVGs on the page, 30 kept: the four the cap dropped are now countable.
317+
await withTempDir(async (dir) => {
318+
const svgs = Array.from({ length: 34 }, (_, i) => ({
319+
outerHTML: `<svg viewBox="0 0 ${i} 10"><rect width="10" height="10" fill="#abc"/></svg>`,
320+
isLogo: false,
321+
}));
322+
const { assets, drops } = await downloadAssets(
323+
{ svgs, sections: [], ogImage: "" } as unknown as DesignTokens,
324+
dir,
325+
);
326+
expect(assets).toHaveLength(30);
327+
expect(drops["cap-reached"]).toBe(4);
328+
});
329+
});
330+
});

0 commit comments

Comments
 (0)