blackwall run gpt-oss-120b
Each model is loaded through llama-swap and measured for speed and answer quality the same way every time, on one NVIDIA RTX PRO 6000 Blackwell with 96 GB of memory. One leaderboard comes out, with an honest margin on every number.
The green line is the wall: nothing here is both faster and smarter than a model on it. Whiskers are 95% intervals.
Qwen 3.8 27B leads at 81.0 quality, with Nemotron 3.5 Super VL 120B inside its margin of error. Nemotron 3.5 Bolt 74B is the fastest at 267 tok/s. Qwen 3.8 27B spends the least energy per correct answer, 2.67 Wh.
| # | Model | Quality, ±95% | Know | Math | Code | Long | Tools | tok/s | Wh/correct |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Qwen 3.8 27B Uncensored · SGLang · FP8 · DFlash2 · 512K | 81.0 ±2.7 | 61.2 | 81.0 | 76.2 | 86.4 | 100.0 | 157 | 2.67 |
| 2 | Nemotron 3.5 Super VL 120B vLLM · NVFP4 · MTP · 256K | 79.2 ±2.5 | 61.2 | 88.9 | 83.2 | 65.5 | 97.4 | 187 | 3.18 |
| 3 | Muse Glimmer 30B SGLang · FP8 · DFlash2 · 128K | 71.6 ±3.1 | 57.8 | 84.1 | 62.4 | 56.4 | 97.4 | 143 | 2.73 |
| 4 | Nemotron 3.5 Bolt 74B llama.cpp · IQ2 · DFlash · 256K | 71.4 ±2.9 | 52.8 | 84.1 | 69.3 | 53.7 | 97.4 | 267 | 6.54 |
| 5 | GPT-OSS 120B SGLang · MXFP4 · DFlash · 128K | 71.2 ±2.7 | 51.4 | 87.3 | 83.2 | 37.7 | 96.5 | 206 | 5.99 |
| 6 | Kolibri 1 78B-A3B llama.cpp · Q5_K_M · 160K | 67.7 ±3.2 | 46.4 | 86.5 | 78.2 | 33.5 | 93.9 | 172 | 12.00 |
| 7 | MiMo-V2.6 Distill Qwen 9B llama.cpp · Q5_K_M · 256K | 50.0 ±3.8 | 39.7 | 41.3 | 35.6 | 55.2 | 78.1 | 164 | 17.92 |
Max effort, suite full/v1, one RTX PRO 6000 Blackwell. A snapshot of the live board from 31 Oct 2026, refreshed with every deploy. Open the live page for rank ranges, the weights lab, profiles and compare.
The live page. Rank badges show the range each model lands in when the benchmark is rerun, hatched rows are runs still in progress.
install + first run
uv tool install git+https://github.com/guitaripod/blackwall
blackwall doctor gpt-oss-120b # loads it and checks tokens, tool calls and a 16k prompt
blackwall run gpt-oss-120b # detached; closing the terminal only detaches you
blackwall wait # follow it, however often it gives way to other GPU work
blackwall ls --effort max # ranked table in the terminal
blackwall board --open # the leaderboard page
blackwall compare a b # which differences between two models are realNinety-six gigabytes is a different place from a 24 GB gaming card: a 120B fits at MXFP4 and a 27B fits at half a million tokens of context. Every speed and energy figure comes off the same hardware, and each row names its card, driver and server build.
The rank badge is the span each model lands in across 1,600 simulated reruns of the quality index, so #1-2 means a tie, not a winner. The headline only says "leads" when the lead survives the noise.
The index averages knowledge, math, code, long context and tool use equally. Five sliders and presets change that, the board and the chart move as you drag, and your rank always sits next to the default one.
Strengths and gaps, the scores inside each task, how often answers were cut off, the data checks and the command behind the run. Pin up to four models for a head-to-head that calls a difference only when its interval clears zero.
The worker is a systemd user unit. It saves every answer, gives way when something else needs the GPU, and continues after a crash. A run in progress sits at the foot of the board as a provisional row with the span its index can still land in.
An Atom feed with one entry per finished run, a board.json whose keys are only ever added, and an address that carries the whole view, so a link shares exactly what you were looking at.
Speed: 1 up to 32 requests at once, first-token wait and decode speed from 1k to 250k-token prompts, GPU power. Greedy decoding, streamed timings, nvidia-smi.
SuperGPQA: 521 graduate-level questions with ten options, spread over its 13 disciplines. Scored on the final Answer line.
AIME + HMMT 2026: 63 problems, two samples each, a 64k-token budget. The last boxed answer, compared symbolically with Math-Verify.
LiveCodeBench v6: 101 problems from January to April 2025. Hidden tests run in a network-less Docker container.
MRCR, 8 needles: 60 conversations at 16k, 32k, 64k and 128k tokens. OpenAI’s grade: the requested prefix, then the difflib ratio.
Graphwalks: 48 graphs from 8k to 64k tokens. F1 of the node set on the last line.
Tool use: 38 episodes, three runs each, against a simulated workspace. Scored on the workspace’s end state and the reply, not the exact calls.
The index: the equal-weight mean of five groups, knowledge, math, code, long context (MRCR and Graphwalks) and tool use. It appears once every task has run. Each reasoning effort is ranked on its own.
Datasets are pinned by sha256, question subsets use fixed seeds, and every run records the harness, the suite version and a fingerprint of the launch command.
Every task carries a 95% interval and the index bootstraps the questions of every task. blackwall compare pairs two runs question by question.
When the server stops answering the run stops and says why, with the kernel’s Xid line if there is one, instead of recording silence as wrong answers.
A prompt that does not fit the context scores zero instead of disappearing, and the board names the task it cost.
Energy is integrated across the quality tasks and divided by correct answers, so a fast but wasteful model is visible.
When a task is not separating the models, all near the ceiling or apart by less than the noise, the board marks it.
feed.xml: one Atom entry per finished run, marked new best or debut.
board.json: every number on the page, plus the history of all finished runs, under a schema whose keys are only ever added.
Releases: what changed in Blackwall itself. A change to how scores are computed bumps the suite version, so old and new results are never ranked together.
No. It runs on any NVIDIA GPU llama-swap can serve. The board labels every row with the card and driver it came from and never compares speed, peak throughput or energy across cards. The ranking here is one card’s numbers; a different card, quantization or server setting will move them.
Linux with systemd user units, an NVIDIA GPU, llama-swap in front of SGLang, llama.cpp or vLLM, Docker for grading LiveCodeBench, Python 3.12 with uv, and about 1 GB of disk for the pinned datasets.
A full run at medium effort takes three to six hours in the background. --quick takes 45 to 90 minutes and is ranked as provisional; --smoke is a pipeline check and is never ranked.
Because it is one. With a few hundred questions per task two scores a couple of points apart are inside each other’s noise, so the page shows the range of ranks a model lands in when the benchmark is rerun, and the headline says "leads" only when the lead holds.
SGLang and llama.cpp are the tested servers. vLLM speaks the same API and has a model on the board, but it does not report how many requests it takes at once, so Blackwall sends four.
Read it as an upper bound. LiveCodeBench has published nothing newer than April 2025, so models trained after mid-2025 may have seen its problems. blackwall sources checks for newer releases and for pinned files that moved.
Yes. blackwall export --out site --site-url https://you.example/bench/ writes the page, the data and the feed. The database is the record; everything else is a projection of it.
No. Blackwall is an independent project and is not affiliated with or endorsed by NVIDIA or CD PROJEKT RED. NVIDIA, RTX and Blackwell are trademarks of NVIDIA Corporation.
Blackwall is an independent project and is not affiliated with or endorsed by NVIDIA or CD PROJEKT RED. NVIDIA, RTX and Blackwell are trademarks of NVIDIA Corporation. GPL-3.0.