kind string | v int64 | built_at string | days list | newest timestamp[s] | fields list | panel list | licence string | scope string | sites int64 | parts list | signature dict |
|---|---|---|---|---|---|---|---|---|---|---|---|
crawlcheck-ai-consent | 1 | 2026-11-17T08:46:47.960Z | [
"2026-11-17T00:00:00",
"2026-11-16T00:00:00",
"2026-11-15T00:00:00",
"2026-11-14T00:00:00",
"2026-11-13T00:00:00",
"2026-11-12T00:00:00",
"2026-11-11T00:00:00"
] | 2026-11-17T00:00:00 | [
[
"domain",
"Registrable domain"
],
[
"read_at",
"When robots.txt and the homepage were read (UTC, ISO 8601)"
],
[
"lane",
"saas, gov, local or filler"
],
[
"kind",
"Site kind from the page"
],
[
"robots",
"read = robots.txt parsed; unread = absent or unreadab... | [
{
"agent": "GPTBot",
"company": "OpenAI",
"use": "training"
},
{
"agent": "OAI-SearchBot",
"company": "OpenAI",
"use": "search"
},
{
"agent": "ChatGPT-User",
"company": "OpenAI",
"use": "user fetch"
},
{
"agent": "ClaudeBot",
"company": "Anthropic",
"use":... | https://crawlcheck.io/datasets/ai-consent#licence | free sample: 1000 sites, SaaS and government lanes, evenly spread | 1,000 | [
{
"lane": "saas+gov",
"file": "sample.ndjson",
"rows": 1000,
"bytes": 490842,
"sha256": "24800beec028bd3332b791b65f6a725605b2eeb63612bf81ee4bfb29b6f61cf5"
}
] | {
"alg": "Ed25519",
"kid": "1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps",
"message": "crawlcheck-ai-consent/v1:4e0fab58ab386bd075146cf0699fc98c1fd185205e0a70d703c473aa83e9fa97",
"manifest_sha256": "4e0fab58ab386bd075146cf0699fc98c1fd185205e0a70d703c473aa83e9fa97",
"sig": "s3txVg79ZEpGvg2naHwSar0LbOxK6JDBu6HER... |
CrawlCheck: AI consent data (sample edition)
What each site's robots.txt tells named AI agents, its Content-Signal line and llms.txt, signed; the free sample edition.
Mirrored from crawlcheck.io/datasets/ai-consent by the CrawlCheck Worker; the site is the canonical copy and is updated first. Measured by CrawlCheck's own observer. Free under CC BY 4.0: credit "CrawlCheck" with the URL of the file.
- Catalogue of every dataset: https://crawlcheck.io/datasets
- How the data is made: https://crawlcheck.io/how-it-works
- Downloads last month
- 677