You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 7a07ea9
Browse filesBrowse the repository at this point in the historyBrowse files
fix(cli): fetch a page's assets as the same agent that loaded the page (#3726)
A capture is one session with two halves: Chrome navigates the page with a
browser User-Agent, then Node fetches the assets that page referenced. Those
halves sent three different identities — "HyperFrames/1.0" from the asset and
media downloaders, a bare "Mozilla/5.0" from the stylesheet inliner, and the
real Chrome UA from the navigation itself.
An origin is free to answer those differently, and anti-bot edges do. Capturing
one large site, GET /favicon.svg answers 403 text/html to "HyperFrames/1.0" and
200 image/svg+xml to the UA the very same capture had just navigated with. The
favicon ranker had already picked that SVG as the best declared icon; the 403
discarded it and the downloader fell through to the next candidate, so the icon
written to assets/ was chosen by the CDN's bot rules rather than by the ranker.
The capture reported it as one "unavailable" drop and carried on.
Hoist the navigation UA into CAPTURE_USER_AGENT and use it for every
out-of-band fetch the capture makes: favicons, images, og:image, fonts,
stylesheets, Lottie JSON and videos. One constant is what stops the two halves
drifting apart again.
Verified end to end against that site: before, assets/favicon.png (the
apple-touch icon) plus one unavailable drop; after, assets/favicon.svg, byte
identical to the file the site itself serves.
0 commit comments