| E20 |
Unsloth UD-Q3_K_XL (13.15 GB) context on this machine; the subject's IQ3_S file (12.12 GB) for comparison |
Unsloth fits 32K only; 64K and 48K spill 452–484 MiB. IQ3_S (12,120,016,416 bytes) fits 64K |
A |
results/qwen38-27b-unsloth-q3kxl/arena-2026-10/frozen.json (tried); results/qwen38-27b-iq3s/arena-2026-10/frozen.json; file size in configs/qwen38-27b-iq3s/README.md |
| E21 |
Ornith-1.5-35B-A3B passed the settings check, then spilled mid-run |
shared memory 294 → 359 MiB; decode ~99 tok/s, falling to ~27 tok/s in two windows |
B (invalid run) |
results/ornith-1.5-35b-a3b-apex/arena-2026-10/invalid-spill-1/INVALID.md, decode-over-time.txt |
| E22 |
Ornith-1.5-35B-A3B 10Q (from the invalid run; correctness is unaffected by speed) |
32/40 using 57,346 tokens (4.8× the subject) |
B |
…/invalid-spill-1/10q_grades_r1.json |
| E24 |
Reasoning effort: medium + short-think vs the vendor default xhigh (10Q, 4× budget, 1 run each) |
35/40 with 9,878 tokens vs 30/40 with 25,688 tokens (38 % of the tokens); thinking off 16/40 |
B (preliminary) |
results/qwen38-27b-iq3s/2026-10-03-prelim/effort-sweep.md |
| E25 |
Identical temperature-0 request from a fresh server: adaptive MTP vs fixed draft length 5 |
adaptive: 885–1,219 tokens across 5 runs (answer 5); fixed: 3,080 tokens ×5 (answer 7); a long-running adaptive server gave the same length and answer as the fixed runs |
B (dedicated experiment, 1 prompt) |
results/qwen38-27b-iq3s/determinism-2026-10/README.md |
| E30 |
Pi coding agent + subject builds the dashboard task from scratch (1 run, production server) |
320 s, 28 turns, 27 tool calls (read 2, bash 16, write 5, edit 4), 0 tool errors, 25,678 output tokens; hidden browser checks 20/20 on the third grader revision (18/18 → 19/19 → 20/20; each revision only tightened checks after the run). TASK.md gave the model every selector; only the grader code was hidden |
B (dedicated experiment, 1 run) |
results/qwen38-27b-iq3s/pi-2026-10/dashboard-r1/summary.json, grade-v3/ |
| E31 |
The dashboard grader separates correct from broken work |
reference solution 20/20; copy with 6 injected defects 13/20, all 6 caught; reference and grader written by the same side; known gap for near-equal values |
B |
harness/tests/dashboard/README.md |
| E32 |
HumanEval+ checker validity |
passes all 164 canonical solutions |
A |
harness/tests/humanevalplus_grader_check.json |
| E33 |
Code-review planted bugs are real |
the 6 Python/SQL bugs reproduce when run; the 4 JS/Go/C bugs were not executed |
A |
harness/tests/code_review_bugs_check.json |
| E23 |
Large contexts on this machine (subject) |
131,072: loads, one test request decoded at 5.9 tok/s; 122,880: shared memory 485 MiB, 6.2–7.7 tok/s; 98,304: 421 MiB, 41–74 tok/s; 65,536: 169 MiB, 86–92 tok/s. Overflow into system RAM is inferred from the shared-memory counter |
B (preliminary) |
…/2026-10-03-prelim/ctx-sweep/summary.txt, qwen38-v3-128k.log |