Repository navigation
Breaking, and the break is the point. Every per-submission GitHub repository and every Cloudflare deployment this benchmark created has been deleted. Fourteen public pokedex-* repos of model output were counting against the owner's GitHub language statistics — the benchmark read as a TypeScript project because twelve models happened to reach for React — and a dozen deployments were being kept alive to prove a point about a one-shot build.
Nothing was lost. Everything got better.
The archive is the artifact
archive/repos/<repo>.zip is a complete git clone of each submission — working tree and full history:
unzip archive/repos/pokedex-opus-5-ultracode.zip
cd pokedex-opus-5-ultracode
git log # the model's own commits, intactZips are binary, so Linguist attributes none of it. And because a zip costs nothing in language stats, nothing is stripped any more — which is exactly why fable-5-ultracode's committed 23 MB of PokéAPI JSON and opus-5-ultracode's 41 MB CSV dump came along, and why both of those build offline.
archive/shots/ holds every deployment's last frames, captured minutes before it was removed. archive/MANIFEST.json records each zip's sha256, HEAD commit and commit count, and is regenerated by opening every zip and running git fsck on the restored clone.
The in-tree vendored copies under submissions/ are gone too, for the same reason. Only ENTRY.md remains there.
Run all thirteen yourself
cd selfhost && docker compose up -d
open http://localhost:9080A gallery on :9080; thirteen Pokédexes on :9081–:9093. Each container unzips its archived clone, runs that submission's own build chain, and serves it with wrangler dev — the real Workers runtime (workerd), including the model's own Worker code, asset routing, SPA fallback and KV bindings. Nothing is fetched from GitHub or Cloudflare.
13/13 verified booting, painting real Pokédex content, and throwing zero page errors — more than the live board could claim at the end, where one of twelve had already rotted away from its own source.
One source of truth
The archive is not a backup sitting beside the real thing; it is the real thing.
compute-metrics.mjsreads straight out of the zips and reproduces every recorded LOC, file count and stack detection byte-for-byte.- Grading unpacks from the same zips into a read-only judge sandbox.
validate.mjsrecomputes eachvendorHashfrom the archive and fails if the graded artifact ever changed.
The loop no longer accumulates
scripts/takedown.mjs retires a finished run — but refuses to delete anything until it has opened the zip, git fsck'd the restored clone, and confirmed its HEAD is the commit the remote is still serving. A stale archive is a hard error, not a silent loss.
The one-shot prompt is untouched. Models still self-provision a repo and a deployment, because that is part of what is graded. They just do not outlive their own scorecard.
Breaking changes to submissions.json
| 2.x | 3.0.0 |
|---|---|
sourceRepo |
formerRepo — the repo no longer exists |
liveUrl |
formerLiveUrl — the deployment no longer exists |
| — | archive { repo, zip } |
| — | selfhost { port, serve, dir, build, network, verified } |
| — | vendorHash — fingerprint of the archived artifact |
scripts/verify-live.mjs |
deleted; the drift it caught is now structurally impossible |
Also in this release: gen-selfhost.mjs, gen-archive.mjs, verify-selfhost.mjs, capture-shots.mjs and takedown.mjs; the judge containment from aceacc1 repointed at the archive; and Ox Alpha (max) as the 13th submission.