source string | retrieved string | license string | count int64 | sent_by_scanner int64 | identity_types dict | verification_methods dict | scanner_profile dict | schema dict | agents list |
|---|---|---|---|---|---|---|---|---|---|
https://crawlcheck.io/ai-crawlers | 2026-11-18T15:20:33.066Z | https://creativecommons.org/licenses/by/4.0/ | 116 | 15 | {
"user_triggered_fetch": 10,
"answer_engine_crawler": 6,
"training_crawler": 36,
"tool_or_developer_fetch": 27,
"search_index_crawler": 29,
"control_token": 2,
"link_preview_fetcher": 6
} | {
"user_agent_string_only": 67,
"published_ip_feed": 22,
"operator_documented": 2,
"reverse_dns": 19,
"none_control_token": 2,
"asn_range": 1,
"web_bot_auth": 2,
"operator_published_ranges": 1
} | {
"version": 1,
"scope": "full_scan_initial_identity_comparison",
"profiles": [
{
"id": "anon",
"label": "Unnamed client",
"role": "control",
"crawler_token": null
},
{
"id": "gptbot",
"label": "GPTBot",
"role": "train",
"crawler_token": "GPTBot"
},
... | {
"identity_type": [
"training_crawler",
"search_index_crawler",
"answer_engine_crawler",
"user_triggered_fetch",
"link_preview_fetcher",
"tool_or_developer_fetch",
"control_token"
],
"verification_method": [
"published_ip_feed",
"asn_range",
"reverse_dns",
"operator_pu... | [
{
"token": "Meta-ExternalFetcher",
"operator": "Meta",
"description": "Meta AI live fetch",
"purpose": "answer",
"identity_type": "user_triggered_fetch",
"identity_label": "user-triggered fetch",
"verification_method": "user_agent_string_only",
"verification_source": null,
"verif... |
CrawlCheck: AI crawler catalogue
Every named AI, search and training agent: operator, purpose, and how its traffic can be verified.
Mirrored from crawlcheck.io/ai-crawlers by the CrawlCheck Worker; the site is the canonical copy and is updated first. Measured by CrawlCheck's own observer. Free under CC BY 4.0: credit "CrawlCheck" with the URL of the file.
- Catalogue of every dataset: https://crawlcheck.io/datasets
- How the data is made: https://crawlcheck.io/how-it-works
- Downloads last month
- 3,155