CrawlCheck

Findings · 2026-10-03 · By

Release notes 3 Oct 2026: Resolve, census, rulebook

What changed on crawlcheck.io between 29 September and 3 October 2026: a signed answer AI agents can read before acting, the first agentic-web census and a correction to it, a public rulebook, receipts, disputes and independent observers.

Between 29 September and 3 October 2026, crawlcheck.io gained Resolve, one signed answer an AI agent can read about a domain before acting on it, with no overall score. The first agentic-web census measured 991 sites from the Tranco top 1,000, and its first write-up was corrected the same day. Every finding CrawlCheck can raise is now listed in a public rulebook, a fix ends in a signed before-and-after receipt, and anyone can dispute a finding in a public ledger.

What changed, and what it means for you
Resolve
What it means for you
An agent reads one signed answer about a domain before it fetches, quotes or acts on it
Agentic-web census
What it means for you
The same frozen list of top sites measured repeatedly, so figures can be compared run to run
Public rulebook
What it means for you
Every finding CrawlCheck can raise, with what it means, how serious it is and how to fix it
Signed receipts
What it means for you
A fix ends in a before-and-after record anyone can verify offline
Dispute ledger
What it means for you
Anyone can challenge a finding; the verdict is published and signed
Independent observers
What it means for you
Readings from separate networks; disagreements are shown, never merged
Export or delete your data
What it means for you
A site owner can take or remove everything held about their domain
Evidence and Intelligence
What it means for you
Two new paid lines; nothing that was free moved behind them

Second note in the Releases series. Every line names something that shipped; nothing here is planned. The previous note is 29 September.

ChangeWhat it means for you
ResolveAn agent reads one signed answer about a domain before it fetches, quotes or acts on it
Agentic-web censusThe same frozen list of top sites measured repeatedly, so figures can be compared run to run
Public rulebookEvery finding CrawlCheck can raise, with what it means, how serious it is and how to fix it
Signed receiptsA fix ends in a before-and-after record anyone can verify offline
Dispute ledgerAnyone can challenge a finding; the verdict is published and signed
Independent observersReadings from separate networks; disagreements are shown, never merged
Export or delete your dataA site owner can take or remove everything held about their domain
Evidence and IntelligenceTwo new paid lines; nothing that was free moved behind them

What is Resolve? #

Resolve answers one question for an AI agent: what should it know about this domain before acting? The answer covers the crawl policy the site declares, what each crawler identity was actually served, the machine files, who the site says it is and whether outside sources confirm it, and any open findings. There is deliberately no overall score; an agent decides on the section its action depends on. Every answer is signed with a published key and can be checked without contacting this site. A domain that has never been measured says so and is queued; an opted-out domain says only that.

The census, and the correction to it #

The first census froze one Tranco top-1,000 list so every run measures the same sites: 991 members after nine excluded hosts, counted and never named. Each site is fetched as a browser and as the main AI crawlers, and its robots.txt is read for 12 AI agents per RFC 9309.

Correction, 3 October: the first version of this note said the homepages that did not answer with a page were walling off datacenter traffic. The records said otherwise. 287 of them were timeouts caused by a defect in our own measurement, fixed the same morning. About 244 were infrastructure names with no website, and about 100 actually refused. Since then every delivery figure reads refused from this vantage, never blocked. The census figures now live in the census posts, starting with which AI crawlers the top 10,000 sites block most, and on the dataset page.

Rules, receipts and disputes in public #

The rulebook lists every finding CrawlCheck can raise; a finding that is not in it cannot appear in a report. A fix now ends in a signed receipt holding the before and after readings. When a receipt is lined up against a site's own traffic or leads, the page says plainly that a coincidence in time is not proof.

Anyone can dispute a finding. A site owner's dispute is accepted at once; anyone else first proves they control the domain. The finding is re-checked against its original evidence and a fresh reading, and the verdict (upheld, corrected, fixed since, vantage-dependent or inconclusive) is signed. The first resolution on the ledger was against one of our own sites, and it was upheld. Signed records and daily seals are listed on the proof page.

Second and third networks #

Independent observers on separate networks, one of them the browser extension, now re-read pages. A finding two networks reproduce reads cross-vantage confirmed; one they disagree on reads observer disagreement and is shown as such.

Developers, privacy and security #

A typed SDK with the offline verifier shipped, along with published test fixtures so a third-party verifier can prove it agrees with ours. No median, percentile or count is shown for a group of sites too small to stay anonymous. An outside security audit of the running service produced eight items, all closed.

Pricing and the site #

Two paid lines, Evidence and Intelligence, are described on the pricing page; every surface they gate is new. Findings and guides now read as one library, the glossary gives each term its own page, and the navigation lost half its items without losing any page. Scoring did not change. Two rules were revised and say so on every card they touch: robots.txt matching follows RFC 9309 wildcards and anchors, and a stale-cache finding is confirmed before it is reported.

Live, as you read this: the corpus now holds 566,333 domains across 8,961 scans. The figures in this piece were measured on the date above; this line is not.

CrawlCheck’s dataset when this page loaded: 566,333 sites measured.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

What is CrawlCheck Resolve?

One signed answer about a domain that an AI agent can read before fetching, quoting or acting on it. It covers declared crawl policy, what each crawler identity was served, machine files, identity confirmation and open findings, with no overall score.

Can a Resolve answer be verified without contacting CrawlCheck?

Yes. Every answer is signed with a published key, and the offline verifier checks the signature with no call to crawlcheck.io.

What was corrected in the first census write-up?

It said the homepages that did not answer were walling off datacenter traffic. Many were timeouts caused by a defect in CrawlCheck's own measurement, fixed the same morning; others were infrastructure names with no website. Delivery is now reported as refused from this vantage, never as blocked.

Who can dispute a CrawlCheck finding?

Anyone. A site owner's dispute is accepted at once; anyone else first proves control of the domain. The finding is re-checked against its original evidence and a fresh reading, and the verdict is signed.

Did scoring change in this release?

No. Two rules were revised, robots.txt matching and the stale-cache check, and each card they touch says so.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works