Skip to content

Commit 7a07ea9

Browse files
fix(cli): fetch a page's assets as the same agent that loaded the page (#3726)
A capture is one session with two halves: Chrome navigates the page with a browser User-Agent, then Node fetches the assets that page referenced. Those halves sent three different identities — "HyperFrames/1.0" from the asset and media downloaders, a bare "Mozilla/5.0" from the stylesheet inliner, and the real Chrome UA from the navigation itself. An origin is free to answer those differently, and anti-bot edges do. Capturing one large site, GET /favicon.svg answers 403 text/html to "HyperFrames/1.0" and 200 image/svg+xml to the UA the very same capture had just navigated with. The favicon ranker had already picked that SVG as the best declared icon; the 403 discarded it and the downloader fell through to the next candidate, so the icon written to assets/ was chosen by the CDN's bot rules rather than by the ranker. The capture reported it as one "unavailable" drop and carried on. Hoist the navigation UA into CAPTURE_USER_AGENT and use it for every out-of-band fetch the capture makes: favicons, images, og:image, fonts, stylesheets, Lottie JSON and videos. One constant is what stops the two halves drifting apart again. Verified end to end against that site: before, assets/favicon.png (the apple-touch icon) plus one unavailable drop; after, assets/favicon.svg, byte identical to the file the site itself serves.
1 parent be86a1e commit 7a07ea9

7 files changed

Lines changed: 112 additions & 18 deletions

File tree

‎packages/cli/src/capture/assetDownloader.test.ts‎

Lines changed: 84 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ import {
1010
toStandaloneSvg,
1111
} from "./assetDownloader.js";
1212
import type { DesignTokens } from "./types.js";
13+
import type { IconCandidate } from "./faviconRanker.js";
14+
import { CAPTURE_USER_AGENT } from "./userAgent.js";
1315

1416
describe("isPrivateUrl — SSRF denylist (security: F-003)", () => {
1517
it("blocks loopback, private, and metadata IPv4", () => {
@@ -187,6 +189,16 @@ describe("downloadAndRewriteFonts — attempt caps", () => {
187189
});
188190
});
189191

192+
function withTempDir<T>(run: (dir: string) => Promise<T>): Promise<T> {
193+
const dir = mkdtempSync(join(tmpdir(), "hf-drops-"));
194+
return run(dir).finally(() => rmSync(dir, { recursive: true, force: true }));
195+
}
196+
197+
/** The two fields `downloadAssets` reads off the token bundle. */
198+
function tokensWithNoSvgs(): DesignTokens {
199+
return { svgs: [], sections: [], ogImage: "" } as unknown as DesignTokens;
200+
}
201+
190202
describe("drop counts — why a referenced asset is not in the capture", () => {
191203
afterEach(() => vi.unstubAllGlobals());
192204

@@ -199,11 +211,6 @@ describe("drop counts — why a referenced asset is not in the capture", () => {
199211
).join("\n");
200212
}
201213

202-
function withTempDir<T>(run: (dir: string) => Promise<T>): Promise<T> {
203-
const dir = mkdtempSync(join(tmpdir(), "hf-drops-"));
204-
return run(dir).finally(() => rmSync(dir, { recursive: true, force: true }));
205-
}
206-
207214
it("counts every face the budget never let it reach, not just the one it stopped on", async () => {
208215
// Four declared, zero budget: the honest number is four, and a warning string could only
209216
// ever have said "some".
@@ -269,11 +276,6 @@ describe("drop counts — why a referenced asset is not in the capture", () => {
269276
});
270277
});
271278

272-
/** The two fields `downloadAssets` reads off the token bundle. */
273-
function tokensWithNoSvgs(): DesignTokens {
274-
return { svgs: [], sections: [], ogImage: "" } as unknown as DesignTokens;
275-
}
276-
277279
it("counts an image dropped for being under the raster floor", async () => {
278280
// 9 KB is under the 10 KB floor. Nothing lands, and the reason is now on the record.
279281
await withTempDir(async (dir) => {
@@ -328,3 +330,75 @@ describe("drop counts — why a referenced asset is not in the capture", () => {
328330
});
329331
});
330332
});
333+
334+
describe("asset fetches present the same identity as the page navigation", () => {
335+
afterEach(() => vi.unstubAllGlobals());
336+
337+
/**
338+
* openai.com's declared icons, in DOM order, with the attributes the page really carries.
339+
* `rankIconCandidates` puts `favicon.svg` first, so that is the file a capture must land.
340+
*/
341+
const OPENAI_ICONS: IconCandidate[] = [
342+
{
343+
rel: "icon",
344+
href: "https://openai.example/favicon.svg",
345+
sizes: null,
346+
type: "image/svg+xml",
347+
},
348+
{
349+
rel: "icon",
350+
href: "https://openai.example/favicon.ico",
351+
sizes: "48x48",
352+
type: "image/x-icon",
353+
},
354+
{
355+
rel: "apple-touch-icon",
356+
href: "https://openai.example/apple-icon.png",
357+
sizes: "180x180",
358+
type: "image/png",
359+
},
360+
];
361+
362+
/**
363+
* Measured against the real origin: `GET /favicon.svg` answers `403 text/html` to
364+
* `User-Agent: HyperFrames/1.0` and `200 image/svg+xml` to the browser UA the capture
365+
* already navigates with. The other two icons are served to either agent.
366+
*/
367+
function serveLikeAnAntiBotEdge() {
368+
return vi.fn(async (url: string, init?: RequestInit) => {
369+
const agent = new Headers(init?.headers).get("user-agent") ?? "";
370+
if (url.endsWith("/favicon.svg") && !agent.startsWith("Mozilla/")) {
371+
return new Response("<html>denied</html>", {
372+
status: 403,
373+
headers: { "content-type": "text/html" },
374+
});
375+
}
376+
const body = new Uint8Array(4096);
377+
return new Response(body, { status: 200, headers: { "content-type": "image/svg+xml" } });
378+
});
379+
}
380+
381+
it("lands the icon the ranker chose from an origin that refuses non-browser agents", async () => {
382+
// With a bot UA the ranker's winner 403s and the capture silently falls through to the
383+
// next candidate, so the file on disk is chosen by the CDN rather than by the ranker.
384+
await withTempDir(async (dir) => {
385+
vi.stubGlobal("fetch", serveLikeAnAntiBotEdge());
386+
const { assets } = await downloadAssets(tokensWithNoSvgs(), dir, [], OPENAI_ICONS);
387+
expect(assets.map((a) => a.localPath)).toEqual(["assets/favicon.svg"]);
388+
expect(assets[0]?.url).toBe("https://openai.example/favicon.svg");
389+
});
390+
});
391+
392+
it("sends one User-Agent for every asset request", async () => {
393+
await withTempDir(async (dir) => {
394+
const fetchMock = serveLikeAnAntiBotEdge();
395+
vi.stubGlobal("fetch", fetchMock);
396+
await downloadAssets(tokensWithNoSvgs(), dir, [], OPENAI_ICONS);
397+
const agents = fetchMock.mock.calls.map((call) =>
398+
new Headers(call[1]?.headers).get("user-agent"),
399+
);
400+
expect(agents).not.toHaveLength(0);
401+
expect(new Set(agents)).toEqual(new Set([CAPTURE_USER_AGENT]));
402+
});
403+
});
404+
});

‎packages/cli/src/capture/assetDownloader.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import { join, extname } from "node:path";
1010
import { createHash } from "node:crypto";
1111
import type { DesignTokens, DownloadedAsset } from "./types.js";
1212
import type { CatalogedAsset } from "./assetCataloger.js";
13+
import { CAPTURE_USER_AGENT } from "./userAgent.js";
1314
import { rankIconCandidates, type IconCandidate } from "./faviconRanker.js";
1415

1516
interface DownloadBudgetOptions {
@@ -503,7 +504,7 @@ async function fetchBuffer(url: string, timeoutMs = 10_000): Promise<Buffer | nu
503504
try {
504505
const res = await safeFetch(url, {
505506
signal: AbortSignal.timeout(timeoutMs),
506-
headers: { "User-Agent": "HyperFrames/1.0" },
507+
headers: { "User-Agent": CAPTURE_USER_AGENT },
507508
});
508509
if (!res || !res.ok) return null;
509510
// Reject XML/HTML error pages disguised as 200 OK (common with S3/CloudFront)

‎packages/cli/src/capture/htmlExtractor.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
import type { Page } from "puppeteer-core";
99
import type { ExtractedHtml } from "./types.js";
1010
import { isPrivateUrl } from "./assetDownloader.js";
11+
import { CAPTURE_USER_AGENT } from "./userAgent.js";
1112

1213
const DEFAULT_SETTLE_TIME = 3000;
1314

@@ -35,7 +36,7 @@ export async function extractHtml(
3536
if (isPrivateUrl(href)) continue;
3637
const res = await fetch(href, {
3738
signal: AbortSignal.timeout(10000),
38-
headers: { "User-Agent": "Mozilla/5.0" },
39+
headers: { "User-Agent": CAPTURE_USER_AGENT },
3940
});
4041
if (!res.ok) continue;
4142
let css = await res.text();

‎packages/cli/src/capture/index.ts‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ import {
2424
totalDrops,
2525
} from "./assetDownloader.js";
2626
import type { IconCandidate } from "./faviconRanker.js";
27+
import { CAPTURE_USER_AGENT } from "./userAgent.js";
2728
import { extractFontMetadata } from "./fontMetadataExtractor.js";
2829
import { normalizeErrorMessage } from "../utils/errorMessage.js";
2930
import { diag } from "../ui/diagnostics.js";
@@ -166,9 +167,7 @@ export async function captureWebsite(
166167

167168
const page1 = await chromeBrowser.newPage();
168169
await page1.setViewport({ width: viewportWidth, height: viewportHeight });
169-
await page1.setUserAgent(
170-
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
171-
);
170+
await page1.setUserAgent(CAPTURE_USER_AGENT);
172171

173172
// Set up hooks BEFORE navigation
174173
await setupAnimationCapture(page1);

‎packages/cli/src/capture/mediaCapture.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import type { Browser, Page } from "puppeteer-core";
1111
import { mkdirSync, writeFileSync, readdirSync, readFileSync, statSync } from "node:fs";
1212
import { join, extname } from "node:path";
1313
import { isPrivateUrl, safeFetch } from "./assetDownloader.js";
14+
import { CAPTURE_USER_AGENT } from "./userAgent.js";
1415

1516
/** Discovered Lottie item from network interception or DOM scan. */
1617
export interface DiscoveredLottie {
@@ -58,7 +59,7 @@ export async function saveLottieAnimations(
5859
// SSRF guard — safeFetch re-checks the denylist on every redirect hop
5960
const res = await safeFetch(lottieItem.url, {
6061
signal: AbortSignal.timeout(requestTimeoutMs),
61-
headers: { "User-Agent": "HyperFrames/1.0" },
62+
headers: { "User-Agent": CAPTURE_USER_AGENT },
6263
});
6364
if (!res || !res.ok) continue;
6465
const buf = Buffer.from(await res.arrayBuffer());
@@ -278,7 +279,7 @@ async function downloadVideoBody(
278279
// Location hop, so a public URL cannot 30x to an internal/metadata host.
279280
const res = await safeFetch(srcUrl, {
280281
signal: AbortSignal.timeout(timeoutMs), // bounded by both per-request and aggregate capture budgets
281-
headers: { "User-Agent": "HyperFrames/1.0" },
282+
headers: { "User-Agent": CAPTURE_USER_AGENT },
282283
});
283284
if (!res || !res.ok || !res.body) return null;
284285
const ct = (res.headers.get("content-type") || "").toLowerCase();
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
/**
2+
* The single identity a capture presents to the origin it is capturing.
3+
*
4+
* A capture is one session with two halves: Chrome navigates the page, then Node fetches the
5+
* assets that page referenced. When those halves send different `User-Agent` headers the origin
6+
* is free to answer them differently, and an anti-bot edge does exactly that — it serves the
7+
* document to the browser and refuses the out-of-band asset fetch.
8+
*
9+
* Measured against openai.com: `GET /favicon.svg` answers `403 text/html` for `HyperFrames/1.0`
10+
* and `200 image/svg+xml` for this string. The favicon ranker had already chosen that SVG as the
11+
* best declared icon; the 403 discarded it and the capture silently fell through to the next
12+
* candidate, so the icon on disk was decided by the CDN's bot rules rather than by the ranker.
13+
*
14+
* One constant, used by both halves, is what keeps that from being reintroduced.
15+
*/
16+
export const CAPTURE_USER_AGENT =
17+
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36";

‎packages/cli/src/commands/capture/video.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import { createWriteStream, existsSync, mkdirSync, readFileSync, unlinkSync } fr
33
import { resolve, join, basename } from "node:path";
44
import { c } from "../../ui/colors.js";
55
import { safeFetch } from "../../capture/assetDownloader.js";
6+
import { CAPTURE_USER_AGENT } from "../../capture/userAgent.js";
67

78
const MAX_VIDEO_BYTES = 250 * 1024 * 1024;
89
const VIDEO_CONTENT_TYPE_RE = /^(video\/|application\/(mp4|octet-stream|x-mpegurl))/i;
@@ -12,7 +13,7 @@ async function streamToFile(url: string, destPath: string): Promise<number> {
1213
// safeFetch re-validates redirect hops; bare redirect:"follow" leaks to private hosts.
1314
const r = await safeFetch(url, {
1415
signal: AbortSignal.timeout(120_000),
15-
headers: { "User-Agent": "HyperFrames/1.0" },
16+
headers: { "User-Agent": CAPTURE_USER_AGENT },
1617
});
1718
if (!r) {
1819
throw new Error(

0 commit comments

Comments
 (0)