Repository navigation
fix(capture): share host policy and block localhost aliases - #3844
Conversation
miga-heygen
left a comment
There was a problem hiding this comment.
Reviewed at head 21fddd6a259371e7fb3c3162797692b63097a870 (5 files, +65/−50), in an isolated worktree with the CLI resolving @hyperframes/engine from this branch's source. No prior reviews on the PR.
Strengths
packages/engine/src/utils/urlDownloader.ts:230-247— one host classifier, normalized before matching (lowercase → strip trailing dot → strip brackets), thenBlockListfor literals. The CLI's hand-rolled IPv4/IPv6 parsers are gone entirely (assetDownloader.ts:629-668deleted); the only remainingisPrivateUrlcallers inpackages/cli/src/capture/{htmlExtractor,lottiePreviewRequests,mediaCapture}.tsall route through the shared policy.packages/cli/src/capture/assetDownloader.test.ts:108-131— the alias test exercises both the initial URL and a public→alias redirect, and assertsfetchwas never called for the initial-alias case. That pins the per-hop check insafeFetch(assetDownloader.ts:659), not just the classifier.notlocalhost.exampleandexample.com.are pinned as allowed (assetDownloader.test.ts:66-67,urlDownloader.test.ts:167), so the suffix match is anchored on a dot boundary.
Verified locally
- CLI
assetDownloader.test.ts39/39; engineurlDownloader.test.ts80/80. - Baseline (main's implementation + this PR's tests): 3 CLI tests fail (
localhost./privateallowed;100.64.0.1and[fec0::1]allowed) and 1 engine test fails. Both new alias regressions are red on the original source, as the description claims. - Mutants against the shared classifier, run under the CLI suite: drop trailing-dot strip → killed; drop
.localhost→ killed; drop.local→ killed (2 tests); drop.internal→ killed (2 tests);safeFetchchecks hop 0 only → killed (2 tests). Drop.toLowerCase()→ survives in both suites, which is expected: WHATWG URL parsing already lowercaseshostname, so that line is a no-op for every caller that passesnew URL(...).hostname. Not a gap, just noting it is defence in depth rather than load-bearing. - Probes through the real classifier:
http://0x7f000001/,http://2130706433/,http://127.1/all canonicalize to127.0.0.1and are blocked;[::ffff:8.8.8.8]is blocked (the deliberate mapped-public restriction the description asks reviewers to weigh);[fe80::1%25eth0]fails URL parsing and is rejected by the catch;internal/localas bare labels are allowed (correct, they are not suffixes).
Previously accepted by capture, now rejected (for the changelog; none of these appear in repo docs, examples or tests that go through the policy):
- IPv4: 100.64.0.0/10, 192.0.0.0/24, 192.0.2.0/24, 192.88.99.0/24, 198.18.0.0/15, 198.51.100.0/24, 203.0.113.0/24, 224.0.0.0/4, 240.0.0.0/4.
- IPv6: fec0::/10, ff00::/8, 2001:db8::/32, and all of ::ffff:0:0/96 including mapped public addresses (the old CLI code allowed
[::ffff:8.8.8.8]). - Names:
*.localhost, and any oflocalhost/*.local/*.internalwritten with a trailing dot. - The
registry.internalhosts inpackages/cli/src/registry/transport.test.tsand thehyperframes.local/studio.localbase URLs inframeCapture.ts:513/domEditingDom.ts:165do not pass through either guard (URL-building only), so nothing in the repo regresses.
Mode split (render vs capture) — assertPublicHttpsUrl keeps its own HTTPS check before delegating to the classifier (urlDownloader.ts:261-266); capture's isPrivateUrl accepts http: and https: (assetDownloader.ts:638). These are two differently named entry points and every render-side caller (downloadToTemp, fetchPublicHttpsText) still goes through the HTTPS-only one, so a caller cannot get the permissive mode by accident. Fine as is.
Nits (non-blocking)
urlDownloader.ts:233—.replace(/\.$/, "")strips exactly one trailing dot, sohttp://localhost../normalizes tolocalhost.and is allowed. I checked: glibcgetaddrinforeturns ENOTFOUND forlocalhost..andfetchfails, so this is not exploitable on Linux today, but/\.+$/costs nothing and removes a resolver-dependent assumption. The media skill (skills/media-use/scripts/lib/media-fetch.mjs:43) has the identical single-dot strip.- Layering note, no refactor requested: the repo now has three deliberate policies — this shared literal-host classifier, the media skill's standalone copy of the same lists, and the catalog script's exact-origin allowlist. The media skill copy is currently byte-equivalent in both the alias set and the subnet lists, but nothing pins that agreement; a small test that imports both and compares the range tables would catch the next drift. The description already documents why the skill keeps its own copy.
- Engine test
urlDownloader.test.ts:158-169asserts on the"private/reserved"message; the CLI test asserts on the boolean plus fetch-call count. Both fine; just noting the engine side does not pin thenotlocalhostboundary the CLI side does.
CI — all required checks green at 21fddd6a (Build, Typecheck, Test, Test: runtime contract, Tests on windows-latest, Render on windows-latest, regression, Semantic PR title).
Verdict: APPROVE
Reasoning: Closes the confirmed localhost. / *.localhost over-http bypass at the classifier, with both new regressions red on the original source and every meaningful mutant killed; the widened deny-set is enumerated above and touches nothing in the repo. Merging is Miguel's call.
— Miga (pr-review)
A captured page could bypass the private-host guard with
http://localhost./...orhttp://foo.localhost/..., including after a public redirect. Normalize trailing dots and block localhost subdomains before requests, using one BlockList-backed host policy shared by capture and rendering instead of the CLI's drifting IPv4/IPv6 classifiers.Capture still accepts public HTTP and HTTPS; rendering remains HTTPS-only. Both now reject
.localand.internalnames, including dotted aliases. Capture gains the engine's reserved-range coverage (CGNAT, benchmark, multicast, class-E, site-local IPv6); it also adopts the engine's existing rejection of all IPv4-mapped IPv6 literals, including mapped public addresses. That compatibility restriction is deliberate and should be reviewed. Public ordinary DNS/IP URLs and trailing-dot public hosts remain supported. DNS resolution remains trusted; this does not claim DNS-rebinding protection. The standalone media skill retains its own policy because it runs outside the workspace package runtime.Addresses the confirmed boundary defect behind #599. No alert dismissal or suppression is included. Download byte budgets, output paths, redirect counts, and rendering behavior are otherwise unchanged.
Validation: full workspace build; 119 focused downloader tests; full CLI 3172 passed / 3 skipped. Both new alias regressions fail against the original source. Tests cover initial and redirected aliases, public-URL compatibility and added reserved ranges. Lint/format, Fallow changed-code gate and signed commit hooks pass (inherited findings excluded, no new suppression).