Dataset Viewer
Auto-converted to Parquet Duplicate
kind
string
v
int64
source
string
license
string
read_at
string
every_hours
int64
note
string
feeds
list
log
list
crawlcheck-ip-range-history
1
https://crawlcheck.io/data/ip-ranges
https://creativecommons.org/licenses/by/4.0/
2026-11-09T13:25:24.342Z
24
Each operator feed is re-read once a day. A change lists the prefixes added and removed (first 40 of each). The first reading of a feed is a baseline, not a change.
[ { "feed": "openai", "operator": "OpenAI · GPTBot", "url": "https://openai.com/gptbot.json", "prefixes": 18, "changes": 0, "last_change": null, "tracked_since": null }, { "feed": "openai_search", "operator": "OpenAI · OAI-SearchBot", "url": "https://openai.com/searchbot.js...
[ { "at": "2026-11-09T13:25:24.401Z", "feed": "commoncrawl", "url": "https://index.commoncrawl.org/ccbot.json", "kind": "baseline", "n": 5 }, { "at": "2026-11-06T21:33:23.354Z", "feed": "openai_user", "url": "https://openai.com/chatgpt-user.json", "kind": "change", "before"...

CrawlCheck: Crawler IP-range history

Every change AI and search operators make to the IP ranges their crawlers use, read once a day from the operators' own published feeds: each feed's current prefix count, and each change with the prefixes added and removed.

Mirrored from crawlcheck.io/data/ip-ranges by the CrawlCheck Worker; the site is the canonical copy and is updated first. Read from each operator's published IP-range feed (URLs in the file). Free under CC BY 4.0: credit "CrawlCheck" with the URL of the file.

Downloads last month
150