Dataset Viewer
Auto-converted to Parquet Duplicate
kind
string
v
int64
week
string
state
string
from
string
to
string
built_at
string
frames
list
reading
string
method
string
page
string
editions
list
feed
string
code
dict
sha256
string
signature
dict
crawlcheck-census-weekly
1
2026-W47
week_to_date
2026-11-16T00:00:00.000Z
2026-11-16T22:19:44.965Z
2026-11-16T22:19:44.965Z
[ { "frame": "top1k", "label": "top 1,000, daily", "runs_this_week": 1, "run": { "run": 46, "finished_at": "2026-11-16T09:07:40.387Z" }, "compared_with": { "run": 45, "finished_at": "2026-11-15T08:59:28.147Z" }, "sites": 991, "figures": [ { "id...
Each frame compares the last run that finished in this ISO week (UTC) with the last run that finished before it. Change is like-for-like: points = (sites gained - sites lost) / sites compared, counting only sites measured in both runs and inside the figure's denominator both times, so a site that did not answer in one ...
https://crawlcheck.io/api/census
https://crawlcheck.io/data/census/weekly?week=2026-W47
[ { "week": "2026-W40", "url": "https://crawlcheck.io/api/v1/census/weekly?week=2026-W40", "archive": "https://crawlcheck.io/api/v1/census/weekly/2026-W40.json" }, { "week": "2026-W41", "url": "https://crawlcheck.io/api/v1/census/weekly?week=2026-W41", "archive": "https://crawlcheck.io/api...
https://crawlcheck.io/api/v1/census/weekly/feed.xml
{ "build_id": "da1f5abd3ddea180", "script_sha16": "9a5c143d61af0ffc", "state": "attested", "deployed": { "at": "2026-11-16T22:03:11.675Z" }, "build_log_entry": 2630, "score_version": 43, "scoring_policy_sha256": "3e08fccfbbce356dc82ecd302b32a226a36fc0fb3150a4ab72248fb5795372af", "decision_rules_sh...
45c496311e449c6d484ce74f64879d735cd175e4eb0fd8132f4ec0cec1bb3450
{ "v": 1, "alg": "Ed25519", "kid": "1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps", "over": "sha256", "message": "crawlcheck-data-v1\n45c496311e449c6d484ce74f64879d735cd175e4eb0fd8132f4ec0cec1bb3450", "sig": "_hN1WHOVVRpaoUergkMbWPUgVXPbj-82RhiIK3I954jJ3sKcfvS6Ul3GkF1aXH1QVC4EuEYTSgIX46MdnGKrDg", "signed_at...

CrawlCheck: Agentic-web census, weekly

The top sites on a fixed frame: crawler policy, machine files, declared capabilities and whether AI crawlers get what a browser gets, sealed once a week.

Mirrored from crawlcheck.io/data/census/weekly by the CrawlCheck Worker; the site is the canonical copy and is updated first. Measured by CrawlCheck's own observer. Free under CC BY 4.0: credit "CrawlCheck" with the URL of the file.

Downloads last month
2,636