Repository navigation
Commit 958a907
fix(benchmark): record exposure per slot, and bind profile and origin to the PR (#456)
Addresses the five review findings. Each was reproduced before the fix and
re-checked after; the guard sweep is nine perturbations plus two policy moves.
**[P1] Exposure is now recorded, and it decides the split.** Forcing every
external PR to `either` was wrong: holdout means the engine was never tuned
on the case, which is a fact about this project's history, not about where
the bytes live. Six candidates are engine-development inputs -- #3020,
#3076, #4489 and #1745 through `test_surface_exclusions`, `triggers.py`,
`test_init_scaffold_disclosure` and `test_declaration_questionnaire`;
stripe/ai#232 through a committed fixture tree; and
openai-agents-python#3392, which produced
`test_capability_change_schema_hash_parity.py` and the fix behind it. A new
`exposure` column carries them, `split_eligibility` is derived from it, and
a detector re-reads the tree as a *floor*, so a row may declare more than is
findable but never less. `maintainer_walk` has no detector -- mcp-grafana#1080
appears nowhere in this repository and still drove `tool_sources[].binding` --
so `diff_substance` implies it, closing the one exposure anybody could drop.
Three `mcp_openapi_declared_binding` cells gain a third slot whose only job
is to be holdout-eligible.
**[P2] The holdout floor is computed.** `ceil(count * fraction)` from the
policy, not a hard-coded one. At 0.60 the inventory now fails in 14 cells.
**[P2] Profile and merge state are bound to the register.** Every sourced
candidate has a register entry; swapping two candidates' profiles now fails.
For the five source-type profiles a sample's declared manifest type is
checked too; `coding_agent_trust_roots` and `multi_agent_handoffs` are
scenario profiles with nothing mechanical to check against, which the
register says.
**[P1] awslabs/mcp#4489 was misdescribed and misplaced.** It adds two
literal FastMCP entrypoints, `budget-actions` and `budget-notifications`,
taking the budget surface 1 -> 3; both read-only. Static literals are exactly
what a grep-the-literal extraction reads, so this never supported an
`insufficient_evidence` hypothesis. Moved to the reserve as a `passed`
alternate; the cell is now a gap leading to a genuinely non-enumerable
surface.
**[P2] An origin is a fact about the PR.** adk-python#6605 was closed
without merge, so the gap naming it plans `rejected_or_reverted`. Checking
this also caught adk-samples#1745, which is still **open** -- not history at
all -- so it leaves its slot for the reserve.
**Recommendation taken.** `human_label` is renamed `miner_label` and
disclosed as *not verifier-independent*: the labeling worksheet carries
`head_decision`, `verify_verdict` and `verify_can_merge`, and LABELING.md
tells the labeler those are enough to label without opening the diff. That
biases cell targeting; it does not reach the corpus label, which Amendment 1
raters produce blind. A test fails if the worksheet ever stops exposing
verdicts, so the disclosure cannot outlive its cause.
One judgment is flagged rather than settled: `benchmark_scored` does not by
itself block holdout, on the rule that being measured is not being tuned on.
If any W24-W26 scoring drove engine changes not traceable to a test or a
walk, those subjects need reclassifying, and only the owner can say.
Refs #456
Co-Authored-By: Claude Opus 5 <[email protected]>1 parent c03ca30 commit 958a907
4 files changed
Lines changed: 657 additions & 245 deletions
File tree
- benchmark/safety-qualification
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
23 | 40 | | |
24 | 41 | | |
25 | 42 | | |
| |||
0 commit comments