Specification · v1.4.0One signed answer before an agent acts.
What an AI agent should know about a domain before it reads, cites, connects or transacts, as one signed document. Any tool may read it, cache it, verify it offline and build on it. This page is the specification; the change log at the end is its history.
1
document version; never changed in place
17
sections an agent can decide on
0
overall scores, on purpose
For agents
Ask, decide, verify
Three calls cover the path from a domain name to an action you can account for.
1 · Ask
Resolve the domain
GET /api/v1/resolve?domain= returns the signed answer; a never-measured domain comes back partial in about a second and upgrades when the full scan finishes. A site may staple its own answer in a header, and the DNS form needs no HTTP client.
EndpointsThe document
2 · Decide
Get a receipt
POST /api/v1/preflight turns the answer into one decision for the action you are about to take, signed, with the rules that decided it. The visited site can check the receipt you present.
Decision rule tablesQuickstart
3 · Verify
Check it offline
Recompute the digest, check the Ed25519 signature against the published key, and confirm the answer’s leaf in the witnessed log. No call to CrawlCheck is needed.
SignatureThe verifier
JSON Schema · machine-readable spec · API reference with worked examples · offline verifier · signing keys · Resolve over DNS · observer protocol
| Method | Path | Purpose |
|---|
| GET | /api/v1/resolve?domain= | One signed ResolveV1 answer. Also /v1/resolve/{domain} and /api/v1/resolve/{domain}. Cached at the edge for 5 minutes. Add as_of= for the answer that was in force at a past time, as it was signed then. |
| POST | /api/v1/resolve | Batch: {"domains": [..]}, 1-100 names, one ResolveV1 per name in the order sent. |
| GET | /api/v1/resolve-prefix/{prefix} | Private lookup: send 2-8 hex characters of sha256(domain) and pick your answer out of every measured domain under that prefix, so the domain you are about to act on is never sent. |
| POST | /api/v1/preflight | A signed decision for one action {domain, action, agent?, template?, policy?}: allow, warn, require_confirmation, block or unsupported, with reason codes and an expiry. |
| GET | /api/v1/decisions/{sha256} | The visited site checks a receipt an agent presents (the CrawlCheck-Receipt header names its digest). |
| GET | /api/v1/resolve/archive/{sha256} | The exact signed answer a decision receipt was made from. |
| GET | /api/v1/staple?domain= | A site’s own signed verdict to serve in the CrawlCheck-Staple header, so an agent needs no lookup. |
| GET | /dns-query | Resolve over DNS-over-HTTPS: a signed TXT record at <domain>._v.crawlcheck.io. |
| GET | /api/v1/changes?since= | Change feed: one event per section that moved when an answer was rebuilt. Poll with the last next_since; re-resolve only the domains listed. |
| GET | /api/v1/log | The append-only answer log: the latest signed tree head, inclusion proofs and the witness we run on GitHub. |
| GET | /api/v1/snapshot | Every public answer, once a day, signed: for agents that want the whole set offline. |
| GET / POST | /api/v1/visitor | Visitor Resolve: for a site, whether a request really comes from the agent its user agent names. |
| POST | /api/v1/outcome | An agent reports what happened when it acted. Self-reported; counted, never used to change an answer. |
| GET | /api/v1/outcomes?domain= | What agents reported for a domain over 30 days, including disagreements with the answer. |
| GET | /api/v1/domain?domain= | The domain’s public registry record (RDAP) and its timeline. |
| GET | /api/v1/accuracy?domain= | What Google’s AI states about a claimed business against what the business publishes. |
MCP: the tool resolve_domain on https://crawlcheck.io/mcp returns the same document.
kind: "crawlcheck-resolve", v: 1. status is measured, partial (a domain never measured before, read live in about a second; assurance says how much was read and upgrade when the full scan replaces it), not_measured (a measurement is requested), opted_out or withheld (the answer is held back, for example because the opt-out list could not be read or no signing key is available; ask again shortly). Treat any status you do not recognise as not measured: never act on it. Each section carries state: declared (what the site says about itself), observed (what CrawlCheck was served) or not_measured, and observed_at. There is deliberately no overall score: an agent decides on the section its action depends on, or takes the decision in actions.
| Section | Answers |
|---|
| crawl_policy | robots.txt as declared, per crawler, with signal conflicts |
| delivery | what each identity (browser, each AI crawler) was actually served: status, words, blocker, redirect |
| machine_files | llms.txt, sitemaps, well-known files: status and hashes |
| capabilities | what the site declares agents can do, by safety class, with verification and mismatches |
| mcp_registry | remote MCP servers listed for the domain and what each answered to the handshake |
| entity | who the site says it is and whether external sources confirm it back |
| self_claims | what the site says about itself (phone, address, hours, prices) and where each claim is published |
| impersonation | whether the domain looks built to be mistaken for a known brand |
| control | whether control of the domain has been proven (DNS TXT, meta tag, well-known file or Cloudflare zone), by whom, and the organization behind it when published; it never changes a decision |
| read_equivalence | whether the main text AI crawlers are served matches what a browser gets |
| content_clock | when the main content last changed, from a hash of the content alone |
| rights | one permission answer per use: train, ground, summarize, cite, transact |
| findings | open findings, each with its rule code and revision; in full |
| actions | a decision for read, cite, connect, transact and administer, made from the sections above |
| absences | a signed record of what was looked for and not found, and where |
| freshness, observers, confidence, contradictions | how old the reading is, from how many vantages, and where they disagree |
| evidence, transparency | the report and manifest the answer was built from, and its leaf and inclusion proof in the answer log |
sha256 is the SHA-256 of the canonical JSON of every field except sha256 and signature (keys sorted at every level, no whitespace). signature.sig is Ed25519 over the message prefix crawlcheck-data-v1\n followed by that digest. The key is named by signature.kid and published in /.well-known/http-message-signatures-directory; a retired key leaves that directory when its 7-day overlap ends and stays in the rotation record, so what it signed before then still verifies. Which key signs which kind of document, from when, and what takes over if a key is compromised is set by one rule set: the signing policy.
Online: node crawlcheck-verify.mjs --online answer.json fetches the key directory, so keys retired or declared compromised since are known. Offline: save the directory once (curl -o keys.json https://crawlcheck.io/.well-known/http-message-signatures-directory) and run node crawlcheck-verify.mjs --keys keys.json answer.json; nothing is fetched, so a key compromised after you saved the file is not known until you refresh it. With neither, the signature and digest are checked but the issuer is unknown and the answer is not accepted. Node 20+, no dependencies; the same checks run in @crawlcheck/sdk and crawlcheck on PyPI. It recomputes the digest, checks the signature against the published key and reports which check failed. transparency carries the answer’s leaf and inclusion path, which fold to a signed tree head a witness we run on GitHub co-signs. Every answer is also a leaf in that day’s Merkle root, anchored with OpenTimestamps; /api/seal/lookup?id=&day= says whether a leaf is in the sealed root.
v in the document changes only when a field is removed or its meaning changes; such a change ships as a new version beside the old one, never in place. Added fields and added endpoints raise the minor number of this specification. Readers must ignore fields they do not know.
| Version | Date | Change |
|---|
| 1.4.0 | 2026-10-09 | Degraded mode: a source that could not be read is named in degraded.sources (with assurance degraded) instead of being taken for an empty one; the actions that depend on it follow the published failure policy (/spec/failure-policy, signed at /api/v1/failure-policy), failing closed where the missing source could hold a block. Added status withheld (the opt-out list could not be read), the section disputes (open disputes add the doubt disputed to read and cite), and the doubt observers_disagree (a finding seen from one network and not another lowers read, cite and connect allow to warn). |
| 1.3.0 | 2026-10-09 | Added the section control: whether control of the domain has been proven to CrawlCheck (DNS TXT record, homepage meta tag, well-known file or Cloudflare zone), by a licence holder or a public registry claim, with the organization behind it when its holder made one public. Informational only: no decision reads it. |
| 1.2.0 | 2026-10-07 | Added to the document: status partial (a never-measured domain read live in about a second, with assurance and upgrade), and the sections impersonation, mcp_registry, content_clock, self_claims, read_equivalence, rights, actions (a decision per read, cite, connect, transact, administer), absences (signed absence) and transparency (the answer's leaf and inclusion proof in the append-only log). Added endpoints: preflight decisions POST /api/v1/preflight; receipt checks GET /api/v1/decisions/{sha256}; the exact answer a receipt was made from GET /api/v1/resolve/archive/{sha256}; the answer in force at a past time GET /api/v1/resolve?as_of=; private lookup GET /api/v1/resolve-prefix/{prefix}; site staples GET /api/v1/staple; Resolve over DNS GET /dns-query; the transparency log GET /api/v1/log; the daily signed snapshot GET /api/v1/snapshot; Visitor Resolve GET/POST /api/v1/visitor. The JSON Schema now describes impersonation and transparency. No existing field changed meaning; v stays 1. |
| 1.1.0 | 2026-10-03 | Added: path form GET /v1/resolve/{domain}; batch POST /api/v1/resolve (1-100 domains); change feed GET /api/v1/changes; agent outcome reports POST /api/v1/outcome and GET /api/v1/outcomes; domain lifecycle GET /api/v1/domain; AI accuracy GET /api/v1/accuracy. Answers, change batches, outcome tallies and lifecycle records join the daily sealed Merkle root. No field of ResolveV1 changed. |
| 1.0.0 | 2026-10-02 | First publication: GET /api/v1/resolve, ResolveV1 document with declared / observed / not_measured sections and no overall score, Ed25519 signature, JSON Schema at /schemas/resolve.json, MCP tool resolve_domain, offline verifier. |