Skip to content

Commit 958a907

Browse files
fix(benchmark): record exposure per slot, and bind profile and origin to the PR (#456)
Addresses the five review findings. Each was reproduced before the fix and re-checked after; the guard sweep is nine perturbations plus two policy moves. **[P1] Exposure is now recorded, and it decides the split.** Forcing every external PR to `either` was wrong: holdout means the engine was never tuned on the case, which is a fact about this project's history, not about where the bytes live. Six candidates are engine-development inputs -- #3020, #3076, #4489 and #1745 through `test_surface_exclusions`, `triggers.py`, `test_init_scaffold_disclosure` and `test_declaration_questionnaire`; stripe/ai#232 through a committed fixture tree; and openai-agents-python#3392, which produced `test_capability_change_schema_hash_parity.py` and the fix behind it. A new `exposure` column carries them, `split_eligibility` is derived from it, and a detector re-reads the tree as a *floor*, so a row may declare more than is findable but never less. `maintainer_walk` has no detector -- mcp-grafana#1080 appears nowhere in this repository and still drove `tool_sources[].binding` -- so `diff_substance` implies it, closing the one exposure anybody could drop. Three `mcp_openapi_declared_binding` cells gain a third slot whose only job is to be holdout-eligible. **[P2] The holdout floor is computed.** `ceil(count * fraction)` from the policy, not a hard-coded one. At 0.60 the inventory now fails in 14 cells. **[P2] Profile and merge state are bound to the register.** Every sourced candidate has a register entry; swapping two candidates' profiles now fails. For the five source-type profiles a sample's declared manifest type is checked too; `coding_agent_trust_roots` and `multi_agent_handoffs` are scenario profiles with nothing mechanical to check against, which the register says. **[P1] awslabs/mcp#4489 was misdescribed and misplaced.** It adds two literal FastMCP entrypoints, `budget-actions` and `budget-notifications`, taking the budget surface 1 -> 3; both read-only. Static literals are exactly what a grep-the-literal extraction reads, so this never supported an `insufficient_evidence` hypothesis. Moved to the reserve as a `passed` alternate; the cell is now a gap leading to a genuinely non-enumerable surface. **[P2] An origin is a fact about the PR.** adk-python#6605 was closed without merge, so the gap naming it plans `rejected_or_reverted`. Checking this also caught adk-samples#1745, which is still **open** -- not history at all -- so it leaves its slot for the reserve. **Recommendation taken.** `human_label` is renamed `miner_label` and disclosed as *not verifier-independent*: the labeling worksheet carries `head_decision`, `verify_verdict` and `verify_can_merge`, and LABELING.md tells the labeler those are enough to label without opening the diff. That biases cell targeting; it does not reach the corpus label, which Amendment 1 raters produce blind. A test fails if the worksheet ever stops exposing verdicts, so the disclosure cannot outlive its cause. One judgment is flagged rather than settled: `benchmark_scored` does not by itself block holdout, on the rule that being measured is not being tuned on. If any W24-W26 scoring drove engine changes not traceable to a test or a walk, those subjects need reclassifying, and only the owner can say. Refs #456 Co-Authored-By: Claude Opus 5 <[email protected]>
1 parent c03ca30 commit 958a907

4 files changed

Lines changed: 657 additions & 245 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 32 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -4,22 +4,39 @@
44

55
- **The pre-1.0 qualification corpus now has a committed sourcing plan.** (#456)
66
`benchmark/safety-qualification/strata-inventory.csv` maps the known candidate
7-
pool onto all 28 profile × decision cells the `pre_1_0` policy requires — 56
7+
pool onto all 28 profile × decision cells the `pre_1_0` policy requires — 59
88
slots, each either a pinned candidate, an identified-but-unpinned one, or a gap
9-
with the lead that would close it — so Cut B mines the empty cells instead of
10-
re-finding the full ones. It is a plan, not evidence: it carries no label, no
11-
verdict and no receipt, nothing in it reaches the qualification runner, and it
12-
is not an admissible rater input, because it names a target decision for every
13-
slot. `tests/test_strata_inventory.py` derives the grid from
14-
`pre_release_safety_requirements()` rather than restating it, so a policy move
15-
fails the inventory instead of leaving it silently aimed at the wrong shape,
16-
re-reads every cited miner label and pinned SHA from its source, refuses an
17-
origin the candidate's own nature contradicts, and recomputes every number in
18-
the accompanying reading so the prose cannot drift from the plan. The plan's
19-
own shape is the finding: 28 of 56 slots have a candidate, the origin floor
20-
(23 qualifying cases) is the binding constraint rather than the case count,
21-
`n8n` has one sourced slot of eight, and the sweeps that produced the pool ran
22-
before #403 — so their zero-trigger repositories are unexplored, not empty.
9+
with the lead that would close it. It is a plan, not evidence: it carries no
10+
label, no verdict and no receipt, nothing in it reaches the qualification
11+
runner, and it is not an admissible rater input, because it names a target
12+
decision for every slot.
13+
14+
Each slot records its **exposure** — whether the engine was developed against
15+
that candidate — and holdout eligibility is derived from it rather than from
16+
where the candidate's bytes live. That is what the plan turns on: every vendor
17+
MCP server the adoption walks measured produced an issue, a fix, or a
18+
regression test, so the four best-understood candidates in the pool can only
19+
ever be tuning cases, and three `mcp_openapi_declared_binding` cells carry a
20+
third slot whose only job is to be holdout-eligible. The cell targeting drawn
21+
from miner labels is disclosed as **not verifier-independent**: the labeling
22+
worksheet shows the engine's own verdict columns, which biases which cell a
23+
candidate is aimed at (it does not reach the corpus label, which Amendment 1
24+
raters produce blind).
25+
26+
`tests/test_strata_inventory.py` derives the grid and the holdout floor from
27+
`pre_release_safety_requirements()` rather than restating them, so a policy
28+
move fails the inventory instead of leaving it silently aimed at the wrong
29+
shape. It re-reads every cited miner label, pinned SHA, declared exposure, and
30+
candidate profile and merge state from the source that records it — so an
31+
origin cannot outrun the PR that supplies it (an open PR is not history, and a
32+
closed-unmerged one is not `real_history`), and a profile cannot be changed on
33+
one side only.
34+
35+
The plan's own shape is the finding: 26 of 59 slots have a candidate, the
36+
origin floor (23 qualifying cases) is the binding constraint rather than the
37+
case count, `n8n` has one sourced slot of eight, and the sweeps that produced
38+
the pool ran before #403 — so their zero-trigger repositories are unexplored,
39+
not empty.
2340

2441
- **Reviewed risk overrides no longer masquerade as scan observations.** (#460)
2542
`risk_overrides.tags` is excluded from `effect_readings` and the derived

0 commit comments

Comments
 (0)