Bounty Operator

AI model benchmark

Paydirt · release 2026-10

22 models on 18 held pairs: which one finds the planted bug, leaves the fixed code alone and catches an overclaimed report, and what a run costs.

See the model leaderboardRead the method
Test setup and recorded cost
Models ranked
22
Held pairs
18
Inputs per model
36
Recorded inference cost
$273.01
Published
3 October 2026
Harness
omp 18.4.4

The harness requests each model’s highest reasoning effort. A pair counts only when both answers are right. Full benchmark method

Model leaderboard

Each model alone on the raw arm: the shared harness and answer format, with no Bounty Operator profile added. Ranked by score and grouped by tier.

#ModelEffort sent
Tier 1intervals overlap tier leader
1GPT-6 Astraopenai/gpt-6-astraopenaiScore94.417 of 18 pairsRecall100%Fool’s gold8%Challenge100%Failures0%Cost / run$0.52Time2m 21sEffortmax
1GPT-6.1 Solopenai/gpt-6.1-solopenaiScore94.417 of 18 pairsRecall100%Fool’s gold8%Challenge100%Failures0%Cost / run$0.108Time2m 34sEffortmax
3Kimi K3moonshotai/kimi-k3moonshotaiOpen weightScore88.916 of 18 pairsRecall83%Fool’s gold0%Challenge100%Failures0%Cost / run$0.22Time4m 02sEffortmax
3GPT-5.6 Terraopenai/gpt-5.6-terraopenaiScore88.916 of 18 pairsRecall92%Fool’s gold8%Challenge100%Failures0%Cost / run$0.433Time5m 09sEffortmax
5Claude Opus 5.5anthropic/claude-opus-5.5anthropicScore83.315 of 18 pairsRecall83%Fool’s gold8%Challenge100%Failures11%Cost / run$0.882Time5m 58sEffortmax
5GPT-6 Lunaopenai/gpt-6-lunaopenaiScore83.315 of 18 pairsRecall92%Fool’s gold8%Challenge92%Failures0%Cost / run$0.012Time2m 40sEffortmax
7Claude Fable 5.1anthropic/claude-fable-5.1anthropicScore77.814 of 18 pairsRecall83%Fool’s gold8%Challenge92%Failures8%Cost / run$1.38Time4m 17sEffortmax
7DeepSeek V4 Pro 0813deepseek/deepseek-v4-pro-0813deepseekOpen weightScore77.814 of 18 pairsRecall75%Fool’s gold8%Challenge100%Failures0%Cost / run$0.088Time2m 00sEffortmax
7DeepSeek V4.1 Flashdeepseek/deepseek-v4.1-flashdeepseekOpen weightScore77.814 of 18 pairsRecall92%Fool’s gold8%Challenge100%Failures6%Cost / run$0.063Time5m 13sEffortmax
7Hy4 previewtencent/hy4-previewtencentOpen weightScore77.814 of 18 pairsRecall75%Fool’s gold8%Challenge100%Failures0%Cost / run$0.087Time4m 38sEfforthighasked max
7GLM 5.3z-ai/glm-5.3z-aiOpen weightScore77.814 of 18 pairsRecall83%Fool’s gold17%Challenge100%Failures0%Cost / run$0.105Time5m 18sEffortmax
12Qwen3.8 27Bqwen/qwen3.8-27bqwenOpen weightScore72.213 of 18 pairsRecall58%Fool’s gold0%Challenge100%Failures0%Cost / run$0.085Time2m 58sEfforthighasked maxmixed: high 28, xhigh 8
12GLM 5.3 Flashz-ai/glm-5.3-flashz-aiOpen weightScore72.213 of 18 pairsRecall75%Fool’s gold0%Challenge83%Failures6%Cost / run$0.016Time6m 09sEffortmax
14Qwen3.8 Max (0902)qwen/qwen3.8-max-0902qwenScore66.712 of 18 pairsRecall75%Fool’s gold25%Challenge100%Failures0%Cost / run$0.169Time5m 05sEffortxhighasked max
15MiniMax M3minimax/minimax-m3minimaxOpen weightScore61.111 of 18 pairsRecall58%Fool’s gold17%Challenge100%Failures8%Cost / run$0.039Time2m 51sEfforthighasked max
Tier 2intervals overlap tier leader
16Gemini 3.8 Flashgoogle/gemini-3.8-flashgoogleScore55.610 of 18 pairsRecall58%Fool’s gold17%Challenge100%Failures22%Cost / run$0.204Time2m 55sEfforthighasked max
16Grok 4.7x-ai/grok-4.7x-aiScore55.610 of 18 pairsRecall67%Fool’s gold17%Challenge75%Failures31%Cost / run$0.373Time12m 10sEffortxhighasked max
18Claude Haiku 4.5anthropic/claude-haiku-4.5anthropicScore44.48 of 18 pairsRecall33%Fool’s gold0%Challenge83%Failures0%Cost / run$0.165Time47sEfforthighasked max
18Nemotron 3 Ultranvidia/nemotron-3-ultra-550b-a55bnvidiaOpen weightScore44.48 of 18 pairsRecall33%Fool’s gold17%Challenge92%Failures3%Cost / run$0.045Time5m 08sEfforthighasked max
20Gemini 3.1 Pro Previewgoogle/gemini-3.1-pro-previewgoogleScore38.97 of 18 pairsRecall83%Fool’s gold0%Challenge100%Failures36%Cost / run$0.147Time8m 09sEfforthighasked max
21Claude Sonnet 5.5anthropic/claude-sonnet-5.5anthropicScore33.36 of 18 pairsRecall83%Fool’s gold0%Challenge83%Failures36%Cost / run$1.09Time12m 01sEffortmax
21Mistral Medium 3.5mistralai/mistral-medium-3-5mistralaiScore33.36 of 18 pairsRecall33%Fool’s gold42%Challenge83%Failures3%Cost / run$0.174Time1m 45sEfforthighasked max

A tier starts at its highest-scoring model and takes in every model whose 95% interval overlaps that model’s. These are descriptive groups; overlapping intervals do not establish equal performance.

What each column means
Paydirt Score
100 × pairs right ÷ 18, on the raw arm. A pair is right only when both of its inputs are right. The bar is the 95% interval.
Recall
Planted bugs hit ÷ planted bugs, on the vulnerable twins.
Fool’s gold
Fixed twins whose patched code was reported as a bug rated medium, high, critical or unrated ÷ fixed twins.
Challenge accuracy
The mean of two shares: overclaimed drafts caught with a quote of a false statement, and accurate drafts accepted.
Failure rate
Inputs that ended with no readable answer (refusal, timeout, truncation, a provider error of the model’s own, an answer sheet that does not parse) ÷ inputs.
Cost per run
The mean recorded cost per input in US dollars, including preserved retries. Unpreserved attempts are excluded, so the recorded cost can be lower than total account spend.
Median time
Wall time of the median run.
Effort sent
The reasoning effort the recorded requests carried. The harness asks every model for its highest.

What the numbers say

  1. 15 models share the top tier, from 61.1 to 94.4: each one's 95% interval overlaps the leader's.
  2. In that tier, GPT-6 Luna costs $0.012 a run and Claude Fable 5.1 $1.38, 118.5 times as much.
  3. Across the 22 ranked models, a run costs from $0.012 (GPT-6 Luna) to $1.38 (Claude Fable 5.1).
  4. Eleven of the 22 ranked models returned a readable answer on all 36 inputs.

Which model to use for what

The highest score on each kind of task, the best of the models under $0.10 a run, and the best open-weight model. Equal scores go to the cheaper run.

Solidity review

10 pairs

  • Best scoreUnder $0.10 a run

    GPT-6 Luna

    openai/gpt-6-luna

    Score
    909 of 10
    Median cost per run
    $0.011
    Open the workbench

  • Open weight

    Kimi K3

    moonshotai/kimi-k3

    Score
    808 of 10
    Median cost per run
    $0.215
    Open the workbench

TypeScript API review

2 pairs

  • Best scoreUnder $0.10 a runOpen weight

    GLM 5.3 Flash

    z-ai/glm-5.3-flash

    Score
    1002 of 2
    Median cost per run
    $0.014
    Open the workbench

Challenging a draft report

6 pairs

  • Best scoreUnder $0.10 a runOpen weight

    MiniMax M3

    minimax/minimax-m3

    Score
    1006 of 6
    Median cost per run
    $0.034
    Open the workbench

Each model on its own. A score counts only the pairs of that task. A cost is the median over every input of the model’s run, other tasks included. The workbench adds its own profile to the model you pick there; these scores do not measure that profile.

Pair by pair

Every ranked model on each of the 18 held pairs, on the raw arm. Right means right on both inputs of the pair. Failed means at least one input ended with no readable answer.

SolidityTypeScript APIDraft report
Modelhard-01hard-02hard-03hard-04sol-02sol-03sol-05sol-09sol-10sol-11ts-02ts-03ch-02ch-05ch-06ch-07hard-07hard-08Right
GPT-6 Astra17
GPT-6.1 Sol17
Kimi K316
GPT-5.6 Terra16
Claude Opus 5.515
GPT-6 Luna15
Claude Fable 5.114
DeepSeek V4 Pro 081314
DeepSeek V4.1 Flash14
Hy4 preview14
GLM 5.314
Qwen3.8 27B13
GLM 5.3 Flash13
Qwen3.8 Max (0902)12
MiniMax M311
Gemini 3.8 Flash10
Grok 4.710
Claude Haiku 4.58
Nemotron 3 Ultra8
Gemini 3.1 Pro Preview7
Claude Sonnet 5.56
Mistral Medium 3.56

Pair names only: the held pairs are not published. Bug classes of the scored find pairs (the label of a held pair is not published, the distribution is): language semantics 2, oracle staleness and decimals 2, signature replay and domain separation 2, access control on sibling functions 1, cross-chain message replay 1, initialisation and upgrade gaps 1, queue and epoch boundary errors 1, reward checkpoint order 1, share inflation and rounding direction 1. No class appears more than 2 times.

Right, wrong and not finished

Every ranked model on each of its 36 inputs, on the raw arm. An input that ends with no readable answer scores as wrong, so a low rank can mean wrong answers or unfinished ones. This table tells them apart.

Of 792 inputs, 646 were answered right and 85 wrong. 61 were not finished: 21 timed out, 21 cut off at the output limit, 10 gave no readable answer sheet, 7 ended in a provider error and 2 came back empty. Five of the answers with no readable sheet are the blocked answers of this release.

Gemini 3.1 Pro Preview answered no input wrong. The 13 it lost were not finished: 12 timed out and 1 ended in a provider error.

Claude Sonnet 5.5 answered no input wrong. The 13 it lost were not finished: 11 cut off at the output limit and 2 timed out. It is the workbench’s default model on OpenRouter, where a review is one request and not an agent run at the highest effort.

ModelRightWrongNot finishedWhy not finished
GPT-6 AstraRight35Wrong1Not finished0Why not finished
GPT-6.1 SolRight35Wrong1Not finished0Why not finished
Kimi K3Right34Wrong2Not finished0Why not finished
GPT-5.6 TerraRight34Wrong2Not finished0Why not finished
Claude Opus 5.5Right31Wrong1Not finished4Why not finished2 cut off at the output limit and 2 gave no readable answer sheet
GPT-6 LunaRight33Wrong3Not finished0Why not finished
Claude Fable 5.1Right32Wrong1Not finished3Why not finished3 gave no readable answer sheet
DeepSeek V4 Pro 0813Right32Wrong4Not finished0Why not finished
DeepSeek V4.1 FlashRight32Wrong2Not finished2Why not finished1 timed out and 1 cut off at the output limit
Hy4 previewRight32Wrong4Not finished0Why not finished
GLM 5.3Right32Wrong4Not finished0Why not finished
Qwen3.8 27BRight31Wrong5Not finished0Why not finished
GLM 5.3 FlashRight31Wrong3Not finished2Why not finished2 gave no readable answer sheet
Qwen3.8 Max (0902)Right30Wrong6Not finished0Why not finished
MiniMax M3Right28Wrong5Not finished3Why not finished2 came back empty and 1 timed out
Gemini 3.8 FlashRight24Wrong4Not finished8Why not finished6 cut off at the output limit and 2 timed out
Grok 4.7Right23Wrong2Not finished11Why not finished6 ended in a provider error, 3 timed out and 2 gave no readable answer sheet
Claude Haiku 4.5Right26Wrong10Not finished0Why not finished
Nemotron 3 UltraRight24Wrong11Not finished1Why not finished1 gave no readable answer sheet
Gemini 3.1 Pro PreviewRight23Wrong0Not finished13Why not finished12 timed out and 1 ended in a provider error
Claude Sonnet 5.5Right23Wrong0Not finished13Why not finished11 cut off at the output limit and 2 timed out
Mistral Medium 3.5Right21Wrong14Not finished1Why not finished1 cut off at the output limit

How it is kept honest

Each claim, and the mechanism behind it.

Blocked answers: Anthropic’s cyber safeguards blocked 5 answers in this release: 3 from Claude Fable 5.1 and 2 from Claude Opus 5.5, all on the raw arm, the model alone. The benchmark account is not in Anthropic’s Cyber Verification Program. The blocks count as misses in this release.

Hashes this release ran under

WhatSHA-256
Protocol74aad5311600bf53862cb0c5e09dec6e6af0f63feaff7d6805d021905e0cfd35
Arm raw (the core)5fea9eb00e3f2bb24d3965bf6854d39742d3324b1b826492eaa1ef51dfe09768
Arm solidityad6a658a538ec5ef84c3025febf9dea8a5e076127e2b11e9dcfe346cb2d38a15
Arm generald7f99fc3799b5bf3914fd3aba5668fdf9467a391a1607f896c0e9d26083a01fc
Arm report4550f8a9890f6efd39f757fcea9c5b7ea870a7beccf9fc84c715d66c7a7f6448

Check it yourself

Download latest.json, the per-model files and the archive, unpack the archive and run these from its folder.

  1. Check the published numbers

    Needs Node 22 or later and nothing else. It recomputes every published number from the per-run outcomes, recomputes the hashes from the files in the archive, and re-scores every run on a public case from its stored output.

    Terminal
    node bench/bench.mjs verify --results latest.json --archive paydirt-2026-10-public.tar.gz
node bench/bench.mjs verify --results practice/latest.json --archive paydirt-2026-10-public.tar.gz

  2. Check the public cases

    Needs Foundry for the Solidity proofs. Every planted bug must pass its proof on the vulnerable twin and fail it on the fixed one.

    Terminal
    node bench/bench.mjs lint --proofs

  3. Run a model on the practice set

    Needs omp 18.4.4 and an OpenRouter key. Models are not deterministic: a rerun lands near the published numbers, not on them.

    Terminal
    node bench/bench.mjs run --set public --models <slug> --run-id mine
node bench/bench.mjs score --set public --run-id mine

The practice set

6 public pairs, published with their answer keys, proofs and the raw output of every run on them, so every outcome here can be re-scored end to end. This is not the Paydirt Score.

ModelPairs rightRecallFool’s goldCost per run
GPT-6 LunaPairs right6 of 6Recall100%Fool’s gold0%Cost per run$0.0086
DeepSeek V4.1 FlashPairs right5 of 6Recall100%Fool’s gold0%Cost per run$0.02
GLM 5.3 FlashPairs right4 of 6Recall100%Fool’s gold0%Cost per run$0.011

practice/latest.json

Not run

3 models of the release’s list have no score. A model with no answer is listed here, not given a low score.

WhyModels
Its hosts returned tool calls as plain text on 35 of 36 initial inputs. The batch could not be completed and is excluded from the ranking.Model
  • gpt-oss-120bopenai/gpt-oss-120bRun tier 1
OpenRouter serves it only to accounts that have made an age confirmation, which the benchmark account has not made.Model
  • Muse Spark 1.3meta/muse-spark-1.3Run tier 2
Its runs were not finished when this release was published: 35 of 36 inputs completed.Model
  • Gemini 3.5 Flash Litegoogle/gemini-3.5-flash-liteRun tier 2

Release notes

Google Gemini 3.5 Flash Lite has 35 finalized raw inputs and one unresolved input after repeated provider failures. Its results are shown for inspection, but it is not ranked or recommended.

Questions

Why 18 pairs and one run per input?

Each pair is an original case with an executable proof for every planted bug and every decoy, and the scored pairs are held, so the set grows slowly. With 18 pairs the 95% interval is wide; it is published so that a two-pair difference is not read as a ranking. Every model answers every input once. The release’s credit covered one repeat for many models rather than three for a handful, so the score carries the noise of one run per input: the interval covers which pairs were drawn, not how a model varies between runs.

Why is a model missing from the table?

Models run in tiers, the cheapest first inside a tier, until the release’s credit is used. A model the credit did not reach, or whose host returned no answer, is listed under Not run with the reason. A model whose runs had not finished at publication is listed there too, with how many of its inputs were completed. None of them is given a low score.

Is the benchmark tuned to Bounty Operator?

Bounty Operator is not a row. The ranked score comes from the raw arm, which sends no product text: the same system prompt, task and answer sheet to every model. The profile arms, which add a Bounty Operator core profile, are in the results file: download it, the file keeps every published number. Each pair started as a draft by a model of one of nine vendors. Sessions of Claude Opus 5.5, a ranked model, directed by the benchmark’s owner, then checked, repaired and proved every pair and wrote two of the hard pairs. Those two count as Anthropic-drafted, and the results file gives every model’s score without the pairs its own vendor drafted. Two ranked models, DeepSeek V4.1 Flash and GLM 5.3 Flash, ran the pilot on the find pairs of that time, and while nine pairs were written, drafts were sent to a few ranked models to see whether they could be decided both ways; the method names the models and the pairs whose design changed after such a check.

Does Bounty Operator make a model score higher?

This release does not answer that, and the page makes no claim. Four of the 22 ranked models also ran with the Bounty Operator profiles added, on all 18 pairs, once per input. That is too little to judge the product by. Every number from those runs is in the results file.

What does held mean?

A held case’s files are not published. Before the first scored run each held case got a salted SHA-256 commitment (36 in this release); when a held case moves to the public set, its salt is published with it so the commitment can be checked. Held does not mean unseen by the model providers: a held case is sent to the providers that serve each model, as every prompt is.

When does it update?

With each release. A release spends a fixed amount of credit in run order: tier 1 first, the cheapest model first inside a tier. Held pairs move to the public set over time and new held pairs replace them. A change to the harness, the routing policy, the task text or a set changes every hash, and no earlier run counts after it.