Do your "different" AI models actually think differently? Measure it.
Multi-model stacks — ensembles, mixture-of-agents, judge panels, adversarial review — all rest on one assumption: the models fail differently. Almost everyone infers that from the lab logo. This tool measures it instead.
The headline finding so far: you cannot infer decorrelation from lineage, lab, or country. You have to measure it. Our own earlier small-run "confirmation" that origin predicts prior-distance failed to replicate at larger n. Details in LEADERBOARD.md.
- 40 contested-judgment probes (forced-choice + 0-10 scalar). No right answers — only priors.
- Every model answers every probe. Per model pair: mean disagreement = decorrelation, in [0,1].
- Judge-free. Agreement is computed arithmetically. No LLM judge sits in the loop, so no judge injects its own priors into the metric.
- Two pre-registered structural checks ship with the harness:
- V1 validity — same-lab model pairs must score LOW. If they don't, the metric is broken and the run is suspect.
- V2 bloc — tests whether same-country pairs are closer than cross-country pairs. (In our runs this flips sign across rosters. That instability is the finding.)
export OPENROUTER_API_KEY=sk-or-...
python3 decorr_eval.py --self-test # no API calls
python3 decorr_eval.py --cofailure-golden # no API calls
python3 decorr_eval.py --dry-run # see the plan
python3 decorr_eval.py --agents kimi,glm,grok # small first run (~$0.05)
python3 decorr_eval.py # full 10-model roster (~$1-2)No dependencies beyond the Python standard library. scipy is optional (exact confidence interval in the co-failure report). Answers are cached under runs/<tag>/, so re-runs only pay for new cells.
| Flag | What it measures |
|---|---|
--confidence |
Choice + stated confidence 0-100. Metric becomes total-variation distance between confidence distributions — a 51/49 mind and a 99/1 mind that both "pick A" are not the same mind. |
--calibration |
Known-answer battery (probes/calibration.json). Separates well-calibrated (confident when right) from assertive-style (confident regardless) from humble-style (unsure even when right). |
--cofailure-golden |
Reproduces Proposition 3 of arXiv 2606.27288 on this harness: pairwise correlation provably cannot identify β = Pr[all models wrong]. You must count all-wrong events directly. The golden builds two pools with identical marginals and pairwise stats but different β, and asserts the direct count separates them. |
--batch |
Same run through the OpenRouter Batch API at ~50% token pricing. One batch per model, resumable across interruptions (runs/<tag>/_batches.json). Models without a :batch endpoint fall back to the sync lane automatically. Cheaper, slower wall-clock. |
- A single-curator probe battery leaks the curator's priors into the test set. Ours had two curators from different model lineages, which helps but doesn't fix it. PRs adding probes are the best contribution you can make.
- n=40 probes is enough to rank models and kill bad inferences, not enough for tight confidence intervals on small deltas.
- This measures prior-distance on disputable judgments. It does not measure co-failure on ground-truth tasks (that needs a known-answer battery — see
--calibrationand the β lane) or actual ensemble lift.
This builds on real theory by other people:
- arXiv 2606.27288 — the co-failure ceiling and the Prop-3 impossibility result (pairwise stats can't recover β)
- arXiv 2605.24048 — complementarity-aware ensembles
Our contribution is a standing, reproducible harness and leaderboard — not the theory.
MIT.