How much of the web is ready for agents, and how do you know?

Bar length is how common a finding is; bar colour is how serious — green is low severity, red is critical. The two are independent: the most common defect here is medium, and the critical ones are rare.

What a single crawl cannot see

85.4% of measured sites carry a defect today. 100% carried one at least once between 2026-08-14 and 2026-11-14. 7 look clean now and did not for at least one day in that window.

Both numbers describe the same sites over 93 days, 2026-08-14 to 2026-11-14. A one-off crawl at any budget can only ever produce the first row — the second needs the same sites to have been measured before the question was asked. Sites added after the window opened are excluded (1314) because they have had fewer chances to be seen broken. No scanned domain is named here or anywhere else on this site.

Arrivals across the network

Unlike everything else here, this is not a probe. These are visits nobody asked for, recorded at the origin on 12 participating sites over the last 7 days.

Verified means the source address sits inside the range that crawler’s operator publishes. Unverifiable means the operator publishes no range at all — that is not an accusation, and it is excluded from any forged rate rather than counted against the crawler.

Forged crawler identities

13.2% of the requests that named themselves as a known crawler here were not that crawler. 21499 of 163477 checkable claims came from an address outside the range the operator publishes.

A user-agent is a claim, not an identity. Verified means the source IP sits inside a range the operator publishes — OpenAI, Anthropic, Google, Microsoft, Perplexity and Apple all publish one. Unverifiable is not forgery: some operators publish no range at all, so their requests can be neither confirmed nor accused, and they are excluded from the rate rather than counted against it. This is traffic to this site only, and this site is small — it is a floor on the problem, not a survey of the web.

Coverage matters as much as the counts: a defect rate measured entirely on one platform is a fact about that platform. CrawlCheck deliberately widens the sample rather than scanning the same kind of site repeatedly, and the queue below is what it has not reached yet.

Crawlers we have actually seen

Everything else on this page comes from probes — we send a request wearing a crawler’s name and record what comes back. This table is the opposite: visits nobody asked for, on 4 sites we operate, over 91 days. Verified means the request came from an address range that crawler’s operator publishes: flagged by Cloudflare or, from 20 September 2026, confirmed against the operator’s own published list when Cloudflare does not flag it. Cloudflare never flagged a PerplexityBot request here, although most came from Perplexity’s published ranges, so days before that date undercount operators Cloudflare does not flag.

How to read this, and how not to. 363034 visits across a handful of sites is a sample, not a census — it says what reached these sites, not what the web receives. And unverified is not proof of an impostor: plenty of legitimate traffic arrives from ranges nobody publishes, and our own testing appears in these counts as unverified because it is. What the column does show is that the name in a user-agent string is a claim, and it is checkable.

Platform, rendering, size and business type are read from what each site discloses in its own response. Most sites now answer from behind a CDN, which replaces the origin identity in the headers, so unknown here means the site did not disclose one — not that we did not look. We would rather publish a large unknown than a confident guess.

Each row counts sites, not pages, and a site is counted once per finding however many times it was scanned. A finding is recorded only when the check actually returned an answer: lookups that failed on our side are excluded rather than counted as clean, which is why the totals here can be smaller than the scan count. Domains are never named, in aggregate or individually, and any operator can exclude a domain from this dataset in one line of robots.txt — see the policy page.

Every figure on this page is recomputed when the page is requested, so a number quoted elsewhere is a snapshot of the moment it was read. The same counts are served as JSON at /api/public/counts with an at timestamp — no key, CORS open — so anyone quoting us can check the current value instead of taking ours. A record is only worth more than a single reading if the reading can be repeated.

Run a scan