Repository navigation
launch --headless sends "HeadlessChrome" in the User-Agent, and some sites block on that alone #12
Description
What happens
chrome-agent launch --headless (0.6.1, Chrome 153, Linux) sends this User-Agent, both in the
request header and in navigator.userAgent:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/153.0.0.0 Safari/537.36
At least one large site blocks on that string alone. Headless, https://www.imdb.com/chart/tvmeter/
returns 403 with a page titled 403 Forbidden and none of the 100 list rows. Headed, the same page
loads normally.
The only difference is the string. Launching headless with -- --user-agent= set to the same value,
with HeadlessChrome/ replaced by Chrome/, loads the full page. No JavaScript overrides and no
fingerprint profile are involved. The sec-ch-ua header is "Chromium";v="153", "Not_A Brand";v="8"
in all three cases below, so the client hints do not give headless away; only the UA string does.
| launch | status | rows | title | User-Agent sent |
|---|---|---|---|---|
--headless |
403 | 0 | 403 Forbidden |
...HeadlessChrome/153.0.0.0... |
--headless -- --user-agent=...Chrome/153.0.0.0... |
202 | 100 | Most popular TV shows |
...Chrome/153.0.0.0... |
| headed | 202 | 100 | Most popular TV shows (after the instance marker) |
...Chrome/153.0.0.0... |
It is easy to misdiagnose. Relaunching headed works, so the natural conclusion is "this site needs a
real window", when the only thing that changed was the UA string. That is where I landed first.
Why raise it here
fingerprint.py makes the case for leaving the JavaScript environment alone and spoofing only what
can be set cleanly at launch. --user-agent is exactly that kind of flag, but launch_browser only
passes it when a fingerprint profile is given, so plain --headless sends the string that is
simplest to sniff.
Options, roughly smallest first
- Document it in AGENTS.md next to
launch --headless: what the UA looks like, and the
-- --user-agent=...workaround. - In headless mode, pass
--user-agentwithHeadlessChrome/rewritten toChrome/, unless a
fingerprint profile or an explicit--user-agentis already given. This needs the browser
version at launch time, for example from the binaryfind_chrome_binary()returns. - The same as 2, behind a flag, for anyone who wants the honest headless UA.
This changes default behaviour, which is why I'm opening an issue rather than a PR. Happy to send a
PR for whichever you prefer.
Method
chrome-agent 0.6.1, Chrome 153.0.8010.52, Fedora, Linux 7.2, 2026-09-26. Each case was launched
through the chrome-agent CLI as a fresh instance, twice, with the cache disabled, waiting up to
25 s for 100 rows to render. Status and sent headers come from Network.responseReceived and
Network.requestWillBeSentExtraInfo for the chart document. The headless block also reproduced on
chrome-agent 0.5.7 with Chrome 151 in August. Happy to share the script.
Activity
Document the HeadlessChrome User-Agent and how to replace it
captivus commentedon Sep 28, 2026
Thanks for such a thorough report, @JayOfTheKeyboard. The side-by-side table and the header capture made this easy to confirm. I reproduced it on Chrome 151: plain Chrome launched with
--headless=new, without chrome-agent involved at all, sends the sameHeadlessChrome/151.0.0.0UA. So chrome-agent is passing along Chrome's own behaviour, not introducing it.I've gone with option 1, documentation, and here's why. chrome-agent's default is to launch Chrome as Chrome launches itself, and anything beyond that is opt-in. Both opt-in routes already exist: a
--fingerprintprofile sets the UA, and the--passthrough takes--user-agentdirectly. Rewriting the UA by default would make headless chrome-agent behave differently from headless Chrome, and it would add moving parts: reading the browser version before launch (whichchrome.exe --versiondoesn't support on Windows), and keeping a generated UA out of session snapshots so a restore after a Chrome update doesn't pin an old version.Your point about misdiagnosis shaped the docs most. The new Gotchas entry in AGENTS.md opens with "a site that fails headless but works headed may just be sniffing the User-Agent" and says not to switch to headed. The fix it gives derives the UA from the running browser (
Browser.getVersion, thenHeadlessChrome/→Chrome/), so the version and platform always match the installed Chrome instead of going stale in a hardcoded string. There's a matching paragraph in the README's fingerprinting section. Landed in ab3f3be.If this turns out to bite people often, I'm open to revisiting option 3 (an opt-in flag) later. Closing for now. Thanks again!