benchmark: re-run the labeled corpus on the v0.18 candidate and compare with W27 #312

v0.18 release-exit measurement. Run after the behavior-changing adoption work so the comparison represents the candidate, not an intermediate state. The 1.0 must-block and needs-human bars make this P1 release evidence, not optional analysis.

Summary

Re-run the labeled corpus and report the movement. This is the last remaining measurement obligation from #292, split out because it is a measurement task with a pass/fail bar, not a design question.

Why now

The most recent re-evaluation on disk is benchmark/miner/results/2026-W27-reeval.csv (2026-07-08, run against v0.15.0). Both #296 and #297 landed after it — 2026-07-28 and 2026-07-29 — and both changed cold-start behavior. No numbers have been produced since.

What to measure

Re-run the 19 labeled decided PRs at fixed SHAs against current main, and report against the W27 baseline:

• must_block_caught and needs_human_caught must hold at 1.0. These are hard bars; a regression here blocks the release regardless of what else improves.
• benign_escalation_rate — W27 measured 0.286 and attributed most of it to a cold-start whole-repo-surface artifact. That attribution now needs re-testing: the insufficient_evidence attaches a human-review notice to every turn in first-adoption repos #292 work turned out not to be about that artifact, so the explanation may not survive.
• blocked_recall — 0.0 on real history at W27 (the gate abstained rather than blocked on both real must_block cases). Report whether it moved.

Usable outcomes and coverage

The W27 files have 19 case rows, 17 verifier results and two baseline failures. Report all attempted cases and their denominators; an abstention can satisfy a non-passing safety catch while still failing to give the user a useful review.

Alongside the existing four metrics:

• Report counts and denominators for terminal allowed/complete outcomes, named human-review outcomes, blocked outcomes, insufficient_evidence/coverage misses, input/base failures and product errors. Keep execution success separate from operational completion.
• For benign cases, report how many actually reach the allowed terminal outcome, how many are unnecessarily escalated, and how many receive no answer. Do not discard IE/error cases to improve a rate.
• For human-review cases, report whether the output names the changed capability and an actionable request; a raw escalation is not proof that a reviewer can finish the workflow.
• Preserve W27's original labels and fixed SHAs for comparison. If insufficient_evidence is a statement about the reader, not the change: drop it from ground truth, score it as a coverage miss #520 introduces a different scoring/ground-truth model, publish an explicit separate mapping/view; do not silently relabel the baseline or change the approved 1.0 safety catch bars.
• Measure after the candidate's behavior-changing fixes, including applicable Scope the verdict to the diff by default, with an opt-in whole-tree mode #515/Deprecate instruction-weakening judgments for prose edits; retain structured permission and execution changes #516/insufficient_evidence is a statement about the reader, not the change: drop it from ground truth, score it as a coverage miss #520 work. The design-partner pilot stops at one PR: measure reviewer value and the next eligible change #521 measures actual human use separately; this benchmark cannot claim retention.

Acceptance

• A committed 2026-W##-reeval.csv + .jsonl under benchmark/miner/results/, pinned to the same SHAs as W27 so the comparison is like-for-like.
• A short written delta versus W27 covering the four existing metrics plus usable-outcome/coverage counts, explicit denominators, and the two original baseline failures.
• must_block_caught / needs_human_caught at 1.0, or an explicit regression report if not.

Notes

Per the memory of the W27 run: stripe/agent-toolkit redirects to stripe/ai, so the corpus covers 8 distinct repositories, not 9. Watch for CRLF handling in the miner's scoring path.

Split out of #292.