kind string | v int64 | source string | license string | read_at string | every_hours int64 | note string | feeds list | log list |
|---|---|---|---|---|---|---|---|---|
crawlcheck-ip-range-history | 1 | https://crawlcheck.io/data/ip-ranges | https://creativecommons.org/licenses/by/4.0/ | 2026-11-09T13:25:24.342Z | 24 | Each operator feed is re-read once a day. A change lists the prefixes added and removed (first 40 of each). The first reading of a feed is a baseline, not a change. | [
{
"feed": "openai",
"operator": "OpenAI · GPTBot",
"url": "https://openai.com/gptbot.json",
"prefixes": 18,
"changes": 0,
"last_change": null,
"tracked_since": null
},
{
"feed": "openai_search",
"operator": "OpenAI · OAI-SearchBot",
"url": "https://openai.com/searchbot.js... | [
{
"at": "2026-11-09T13:25:24.401Z",
"feed": "commoncrawl",
"url": "https://index.commoncrawl.org/ccbot.json",
"kind": "baseline",
"n": 5
},
{
"at": "2026-11-06T21:33:23.354Z",
"feed": "openai_user",
"url": "https://openai.com/chatgpt-user.json",
"kind": "change",
"before"... |
CrawlCheck: Crawler IP-range history
Every change AI and search operators make to the IP ranges their crawlers use, read once a day from the operators' own published feeds: each feed's current prefix count, and each change with the prefixes added and removed.
Mirrored from crawlcheck.io/data/ip-ranges by the CrawlCheck Worker; the site is the canonical copy and is updated first. Read from each operator's published IP-range feed (URLs in the file). Free under CC BY 4.0: credit "CrawlCheck" with the URL of the file.
- Catalogue of every dataset: https://crawlcheck.io/datasets
- How the data is made: https://crawlcheck.io/how-it-works
- Downloads last month
- 150