Skip to content

Latest commit

 

History

History
84 lines (67 loc) · 6.44 KB

RESULTS.md

File metadata and controls

84 lines (67 loc) · 6.44 KB

CoalBoard benchmark records

2026-09-16 — CoalBoard audit mode vs cloudflare/security-audit-skill vs a solo control

Measured: 2026-09-16 (UTC) · CoalBoard v2.4.2, audit mode · cloudflare/security-audit-skill commit c1c8a8c (quick profile) · a plain solo -p control · model ids claude-opus-5 / claude-opus-4-8 / claude-haiku-4-5-20251001 (verified 2026-09-16 against Anthropic’s models overview) throughout, n = 3 rounds per arm, one seeded target (withheld) — full stamp incl. Claude Code version in the linked record below.

TL;DR (disclosure: we build CoalBoard, one of the three arms compared): the skill's quick profile matched a single-agent review of the same model tier on recall (0.467 median, 15 seeds, n = 3) at ≈ 14.5× its median cost. CoalBoard's audit mode reached 0.867 median recall at ≈ 3.9× less median cost than the skill. Precision held ≥ 0.857 median strict across all three arms, zero decoy false positives in any round. n = 3, one target — a measured signal, not a verdict; the "theirs weaker" reading holds at the pooled median and in 2 of 3 individual rounds (see the full record for the round that didn't clear it).

Full record: SECURITY-AUDIT-2026-09-16.md — design, every per-arm table, the pre-registered decision-rule outputs, a 13-seed sensitivity recompute, host-load and scoring caveats, and a standing re-run offer to the vendor. Raw safe-to-publish artefacts: results/security-audit-2026-09-16/.

Addendum 1 (2026-09-16 19:20–20:45 UTC, scored 2026-09-17, judge model claude-haiku-4-5-20251001, CoalBoard v2.4.2, permission regime acceptEdits; full stamp in the addendum) — the judge chair: reseats the JUDGE at the weakest model tier (a haiku main) over strong-tier lenses forced there by two research-only levers not shipped in the product (three convened runs), against three whole-weak-tier floors, to ask how much of a verdict survives unattended when the reader deciding it is weak. Weak-judge/strong-lens median recall 0.867 at strict precision ≈0.947 (vs the strong-judge board's own 0.867/0.889) — JUDGE-ROBUST by the pre-registered rule, read with its limits: the verdict held on RETENTION (38 of 39 lens-union seeds kept across the three runs, one true seed silently dropped), the precision is largely the strong lenses' own, whether a weak judge rejects a look-alike stays untested (no strong lens proposed one), and the whole-weak-tier board read "in between". Workflow completion: 10 of 12 collected (the two misses, both on the third-party skill's own floor arm, are a directory-scope miss, not a failure to finish — that same arm also carried a handful of real permission denials this round, disclosed rather than smoothed over). A prior same-day pass under a mismatched permission regime is named and set aside as confounded — see the addendum for both lessons it left standing. Details: SECURITY-AUDIT-2026-09-16.md's Addendum 1; raw artefacts results/security-audit-2026-09-16/round4/.


2026-07-03 — solo-vs-board, strong-solo and weak-solo, mirrored

Measured: 2026-07-03 · CoalBoard v1.5.5 (the skill installed at run time) · two arms on two platforms: Claude Code / Opus 4.8 (strong solo) and Antigravity / Gemini 3.5 Flash (weak solo) · the 5 tasks.md error-not-allowed tasks, judge-run scoring per README.md.

TL;DR: Opus 4.8: solo 4/5 · board 5/5—a strong solo already catches the four reasoning traps; the board's irreducible win is T3, the version-sensitive fact (run-the-check beats training-stale memory). Gemini 3.5 Flash: solo ~4/15 (27%) · board 5/5—a weak solo misses reasoning traps too, so the board recovers ALL of them. The board's margin scales inversely with solo strength; T3 is the win at both ends. Board = solo + ground-truth execution, not an omniscient LLM.

The paired result

Base model (solo arm) Solo Board Board's added value
Opus 4.8 (strong) 4/5 (T3 miss: committed stale Node 22, 0/3) 5/5 small—only T3, the external fact
Gemini 3.5 Flash (weak) ~4/15 (27%) (every task ≤1/3) 5/5 large—all four reasoning traps + T3

The single task the board wins regardless of base model is T3: no model reasons its way past a training-cutoff gap; only the board's RUN-the-check step (a live fetch—Node 24, EOL 2028-04-30, cited) does. On reasoning traps (T1 crypto / T2 compound / T4 race / T5 headings) the board's edge exists only where the solo model is weak.

Full runs (the raw, dated records)

Honest scope

Small n (K=1–3 per cell, majority verdicts—a direction, not a rate). The Claude Code arm is same-family on both sides (Opus lenses share Opus blind spots—the correlated-blind-spot ceiling applies; the board's reasoning-trap coverage cannot exceed what its lenses can see). The T3 gold is time-varying—re-fetch nodejs.org / endoflife.date to re-score. A 5-task sample, not a guarantee: the board's honest claim stays bounded-cost + zero-breakage + improved correctness, never a defect-rate number.