Skip to content

3.0.0

Latest

Choose a tag to compare

@guitaripod guitaripod released this 13 Sep 06:46

Breaking, and the break is the point. Every per-submission GitHub repository and every Cloudflare deployment this benchmark created has been deleted. Fourteen public pokedex-* repos of model output were counting against the owner's GitHub language statistics — the benchmark read as a TypeScript project because twelve models happened to reach for React — and a dozen deployments were being kept alive to prove a point about a one-shot build.

Nothing was lost. Everything got better.

The archive is the artifact

archive/repos/<repo>.zip is a complete git clone of each submission — working tree and full history:

unzip archive/repos/pokedex-opus-5-ultracode.zip
cd pokedex-opus-5-ultracode
git log            # the model's own commits, intact

Zips are binary, so Linguist attributes none of it. And because a zip costs nothing in language stats, nothing is stripped any more — which is exactly why fable-5-ultracode's committed 23 MB of PokéAPI JSON and opus-5-ultracode's 41 MB CSV dump came along, and why both of those build offline.

archive/shots/ holds every deployment's last frames, captured minutes before it was removed. archive/MANIFEST.json records each zip's sha256, HEAD commit and commit count, and is regenerated by opening every zip and running git fsck on the restored clone.

The in-tree vendored copies under submissions/ are gone too, for the same reason. Only ENTRY.md remains there.

Run all thirteen yourself

cd selfhost && docker compose up -d
open http://localhost:9080

A gallery on :9080; thirteen Pokédexes on :9081–:9093. Each container unzips its archived clone, runs that submission's own build chain, and serves it with wrangler dev — the real Workers runtime (workerd), including the model's own Worker code, asset routing, SPA fallback and KV bindings. Nothing is fetched from GitHub or Cloudflare.

13/13 verified booting, painting real Pokédex content, and throwing zero page errors — more than the live board could claim at the end, where one of twelve had already rotted away from its own source.

One source of truth

The archive is not a backup sitting beside the real thing; it is the real thing.

  • compute-metrics.mjs reads straight out of the zips and reproduces every recorded LOC, file count and stack detection byte-for-byte.
  • Grading unpacks from the same zips into a read-only judge sandbox.
  • validate.mjs recomputes each vendorHash from the archive and fails if the graded artifact ever changed.

The loop no longer accumulates

scripts/takedown.mjs retires a finished run — but refuses to delete anything until it has opened the zip, git fsck'd the restored clone, and confirmed its HEAD is the commit the remote is still serving. A stale archive is a hard error, not a silent loss.

The one-shot prompt is untouched. Models still self-provision a repo and a deployment, because that is part of what is graded. They just do not outlive their own scorecard.

Breaking changes to submissions.json

2.x 3.0.0
sourceRepo formerRepo — the repo no longer exists
liveUrl formerLiveUrl — the deployment no longer exists
— archive { repo, zip }
— selfhost { port, serve, dir, build, network, verified }
— vendorHash — fingerprint of the archived artifact
scripts/verify-live.mjs deleted; the drift it caught is now structurally impossible

Also in this release: gen-selfhost.mjs, gen-archive.mjs, verify-selfhost.mjs, capture-shots.mjs and takedown.mjs; the judge containment from aceacc1 repointed at the archive; and Ox Alpha (max) as the 13th submission.