[python] Migrate from API to MCP - #232

* discovery: skip symlink loops instead of crashing detect/init

Found by the first real history-mining run: stripe/ai ships a looping
symlink (llm/ai-sdk/LICENSE), and Path.resolve() surfaces ELOOP as
RuntimeError on CPython — git-based candidate discovery
(_git_candidate_files) crashed with a traceback, taking down `detect`
(the advertised safe first touch) and `init` cold-start on any repo
containing such a path.

Per-entry guard: a symlink loop or unreadable entry skips that entry,
never crashes discovery. Regression test builds a git workspace with a
self-referencing symlink and asserts detect_workspace still classifies
the project.

Co-Authored-By: Claude Fable 5 <[email protected]>

* Add the merged-PR history miner + first real run (WS-P / P1-P2)

benchmark/miner/: replays merged-PR history of public agent repos through
the real engine (trigger → check → init cold-start → scan → capability
diff) and emits one metadata row per PR. One run produces the three
artifacts the roadmap needs: demand receipts (candidates), the
labeled-corpus candidate pool, and the measured real-world IE rate.

Maintainer benchmark tool: gh/git/network allowed here, never in src/
(static-only invariant is scoped to src/agents_shipgate/). No CLI
surface, no schema, not in the wheel. Engine calls go through
sys.executable -m agents_shipgate with AGENTS_SHIPGATE_AGENT_MODE=0
pinned. Monorepo fallback retries at the deepest common directory of
the changed files (the stripe/ai#232 pattern).

First committed run (results/2026-W24-mined.csv, 121 rows over
stripe/ai + openai/openai-agents-python + crewAIInc/crewAI-examples):
108/121 organic trigger-skips; IE on decided = 3/7; stripe/ai#232
ground truth reproduced (insufficient_evidence, 5 review items) via
retry_at:tools/python; README documents why blocked rows are cold-start
gate state, not per-PR receipts (that's miner v0.2 = verify path).

Tests are network-free (local git fixtures): end-to-end evaluation,
trigger-skip, error rows, CSV schema roundtrip, IE-rate summary.

Co-Authored-By: Claude Fable 5 <[email protected]>

* miner: fetch cached clones, resolve rev expressions, LF-only CSV (PR #212 review)

Three review findings:

- P2: ensure_clone reused a cached .miner-work clone without fetching,
  and _cmd_mine silently `continue`d when resolve_base couldn't find a
  new merge commit locally — weekly reruns would drop the newest PRs
  from the corpus with no trace, biasing row counts and IE/trigger
  metrics. Cached clones now `git fetch origin` before resolution, and
  unresolved candidates become explicit error rows
  (notes=merge_commit_not_in_clone) via unresolved_candidate_row().

- P3: the committed PR#232 row stored base_sha as `<sha>^1` while the
  row contract says the columns are commit ids. evaluate_pr now
  rev-parses base/head expressions to full SHAs up front (unresolvable
  revs become error rows with unresolvable_{base,head}_sha notes).
  Regenerated results: base_sha is now 5af4bcd1…, verdict unchanged.

- P3: csv default dialect writes CRLF, failing `git diff --check` on
  every committed row. write_csv pins lineterminator="\n"; both result
  files regenerated LF-only (verified: git diff --check clean).

Tests: LF-only assertion, rev-expression resolution (HEAD/HEAD^1 → 40-
char SHAs), unresolved-candidate error row; bad-sha test updated for
the earlier failure point. Summary stats unchanged (121 rows, 7
decided, IE 3/7).

* miner v0.2: per-PR verify receipts, baseline-gated (WS-P)

Adds the receipt columns the v0.1 README promised: real
`verify --base <base'> --head <head''>` per evaluated PR with the
cold-start manifest committed onto BOTH sides.

Three correctness lessons are baked in (each caught on real history):

- Re-parented head: head'' = head-tree-with-manifest parented on base'
  (commit-tree), so verify's three-dot diff is exactly the PR's delta —
  an independently-parented manifest commit showed up as "shipgate.yaml
  added" and falsely fired the trust-root signal.
- Baseline-gated: without a base-tree baseline the receipt included
  pre-existing blockers (a docs-only PR scored a blocked receipt from
  standing surface). `baseline save` runs on the base tree and verify
  gets `--baseline`, so receipts reflect the PR's new findings
  (release_decision.py demotes baseline-matched blockers).
- Honest failure rows: base-side scan/baseline failures (e.g. the PR
  added the example the manifest references — nothing to scan at base)
  leave verify_* empty with notes naming the step.

Validation on stripe/ai#232 (real trees): with a directory-scoped SDK
source, verify returns merge_verdict=blocked with blocker
SHIP-VERIFY-CAPABILITY-SCOPE-BROADENED and can_merge_without_human=
false — the silent least-privilege removal is caught on real history.
Cold-start stops at insufficient_evidence + cannot-merge because init
scopes the source to main.py while the bounded constructor lives in
support_agent.py (file-vs-directory init scoping; tracked with the
init source-quality fix).

Mining also exposed an engine inconsistency (separate fix tracked):
report-level capability_change marks every cap "broadened (schema_hash
changed, unknown direction)" on a docs-only diff while the lock-diff
artifact from the same run says unchanged — README documents the
verify_cap_* caveat until that lands.

Results regenerated in place at schema 0.2 (121 rows; same headline
stats: 108 trigger-skip, IE-on-decided 3/7).

Co-Authored-By: Claude Fable 5 <[email protected]>

* miner: don't hide a PR's real manifest add/edit in the verify receipt (PR #213 review)

Review P2: _verify_pr copied head's shipgate.yaml onto base whenever base
lacked one. Correct for the miner's synthetic cold-start manifest, but wrong
when the PR itself adds/edits a real shipgate.yaml — mirroring head's manifest
onto base erases the very trust-root diff verify exists to report
(SHIP-VERIFY-TRUST-ROOT-TOUCHED / policy weakening), so verify_trust_root_touched
came back false on a manifest-adding PR.

Fix: branch on the manifest's kind.
- injected (init wrote it): force the identical manifest onto both sides so it
  cancels out of the diff (unchanged behavior).
- preexisting + present at base: PR edited a tracked manifest — keep base's
  real one, baseline-gate.
- preexisting + absent at base: PR ADDED the manifest — read the real base
  commit (no manifest), no baseline; the addition stays a visible trust-root
  diff.

The kind is decided from git history at the real base SHA (`cat-file -e
<base>:<manifest>`), NOT base_wt on disk: _capability_delta already copies the
manifest into the base worktree for its export, so a filesystem check
mis-reads a PR-added manifest as preexisting. verify reads the base commit, so
that stray working-tree copy is irrelevant to the diff.

Validated: on a PR that adds shipgate.yaml the miner now matches plain
`agents_shipgate verify --base --head` exactly (blocked,
trust_root_touched=true, can_merge=false). The injected path is unchanged —
the committed corpus (all init_status=written + PR#232) reproduces
byte-identically, so the results files are not regenerated.

Test: a manifest-adding PR keeps verify_trust_root_touched=true.

…ep) (#217)

* miner: labeling worksheet + verdict-accuracy scoring (WS-P / P3-P4 prep)

The mined corpus is evidence; a human label is ground truth. This adds the
machinery that turns the corpus into the verdict-accuracy benchmark — the
roadmap's gating proof — so the only remaining input is the human labels.

- benchmark/miner/labels.py: build_worksheet (one row per engine-engaged PR
  with blank label/rationale columns), load_labels (adjudicated CSV; rejects
  unknown labels), score (label → gate-action confusion matrix + headline
  metrics: blocked_recall, must_block_caught, needs_human_caught,
  benign_escalation_rate, ie_rate_on_safe), render_matrix_markdown. Scores the
  per-PR receipt (verify_verdict) with cold-start (head_decision) fallback.
- rows.read_jsonl: forward/backward-compatible loader.
- CLI: `labels` (worksheet) and `score` (confusion matrix) subcommands.
- LABELING.md: the three-label rubric (safe_to_merge / needs_human /
  must_block), two-labeler + adjudication process, negative-control sampling,
  metric definitions, and stripe/ai#232 as the worked anchor (must_block).
- Committed turnkey worksheet for the W24 run (13 engine-engaged rows, blank
  labels) so the human pass is copy-and-fill.

All deterministic and network-free: reads the committed JSONL, never re-mines.
Tests (network-free, synthetic rows+labels) pin the confusion-matrix math, the
receipt/cold-start fallback, unknown-label rejection, unmatched-label flagging,
and LF-only worksheet output. Scored end-to-end against the real W24 corpus.

Co-Authored-By: Claude Fable 5 <[email protected]>

* miner score: fail loud on a malformed/wrong labels file (PR #217 review)

Review P2: `score` publishes the headline accuracy table, but it succeeded
(exit 0) on a labels file that produced an empty/partial matrix — a header
typo (`url` not `pr_url`) silently read zero labels and printed all-null
metrics, and a stale/wrong pr_url only warned.

load_labels now hard-errors (instead of silent partial reads) on:
- a missing required header (pr_url / label),
- a filled label with no pr_url (unattributable),
- a duplicate pr_url (ambiguous ground truth).
(Unknown label values already raised; blank-label worksheet rows are still
skipped as "not yet labeled".)

The `score` CLI now returns exit 1 — and prints a clean error, not a
misleading table — when the labels file is malformed, when any labeled
pr_url is absent from the results, or when nothing is labeled yet.
--allow-unmatched and --allow-empty are explicit escape hatches.

Tests: load_labels rejects bad header / blank-pr_url / duplicate; CLI exits
1 on bad header, unmatched label, and zero labeled rows, 0 with the escape
flags and on a clean file. Reproductions from the review now fail loud.

…ock (#261)

* Grow the labeled accuracy corpus: W24+W25 labeled, first real must_block

Label the already-mined-but-unlabeled W24 and W25 decided rows, taking the
real-history labeled decided corpus from 10 (W26) to 19 unique PRs — no
re-mining, since the network/compute-heavy mining was already committed.

Method: same two-independent-labeler + adjudication protocol as W26.
Disagreement 0/9 on the 9 new PRs (the 6 stripe/ai rows in W24 are the same
PRs already labeled in W26 and reuse those labels). AI-generated labels,
pending human spot-check.

Findings (published in benchmark/miner/README.md, surfaced in the README
banner):
- The first real-history must_block rows appear: stripe/ai#232 (drops the
  StripeAgentToolkit least-privilege actions/permissions bounds) and
  crewAIInc/crewAI-examples#169 (wires Slack post + Trello writes + a Gmail
  draft tool). W26 had none.
- The mining-era gate did NOT auto-pass either (must_block_caught 2/2) but
  ABSTAINED (insufficient_evidence) rather than blocking — blocked_recall 0/2
  on real history. The config-bound gap, now measured on real PRs, not
  assumed. Reliable blocked_recall (1.0) stays on the constructed stratum.
- Corrected the stale "zero of 15 decided are must_block" claim in the
  constructed-adversarial section.
- Spot-check: stripe/ai#232 re-evaluated on the released v0.15.0 engine
  returns human_review_required (was insufficient_evidence) — it engages and
  routes to a human rather than abstaining; not a full block and not
  attributed to a specific finding. A W27 re-mine on v0.15.0 (which also
  clears the since-fixed scan_failed crashes) is the follow-up.

Regression pins (tests/test_miner_corpus.py): the W24/W25 labeled scores now
reproduce from the committed labels+corpus, so the published accuracy numbers
can't silently drift. Labels CSVs are LF.

Co-Authored-By: Claude Fable 5 <[email protected]>

* Corpus review fixes: 19 labeled ≠ 19 decided; retire stale W25 prose; tighten pins

Review on #261:

- P1: "19 decided PRs" overstated the scored corpus — 4 of the 19 unique
  labeled rows are W26 goose scan_failed (unscored). README banner and the
  miner README now say "19 unique labeled engine-engaged PRs — 15 with a
  scored verdict, 4 scan_failed/unscored", with the 14/3/2 label mix.
- P2: the W25 findings section still read "only 15 decided (none must_block)"
  and implied constructed is the sole must_block source, contradicting the
  new labels section above it. Updated: 15 carry a verdict; labeling later
  found 2 must_block (the original "none unsafe" reading predated the labels);
  constructed stays where blocked_recall=1.0 is *proven*.
- P3: the score pins now use benchmark.miner.labels.load_labels (not a local
  CSV reader), assert unmatched_labels == [], and a new
  test_aggregate_labeled_corpus_split pins the 19→15+4 split and the
  14/3/2 label distribution — so "19 decided" can't drift back into the docs.

Full CI-style suite + ruff green.

Co-Authored-By: Claude Fable 5 <[email protected]>

* Fix mangled three-strata sentence in the W25 findings

Re-review P3: the must_block caveat added in the prior fix was inserted
into the middle of the three-strata list, orphaning "and harness
transcripts" onto the constructed-stratum proof sentence. Restore the
list (mined-real; constructed-adversarial; harness transcripts for
workflow-evidence replay), then state the must_block caveat as its own
sentence: real history yielded 2 must_block rows on labeling but the
mining-era gate abstained on both, so reliable blocked_recall=1.0 still
comes from the constructed stratum.

…recall still 0.0 (#263)

* W27 re-eval: the 19 labeled PRs on the v0.15.0 engine — abstain→review, not →block

Re-runs the 19-PR labeled corpus (same base→head SHAs as W24–W26) through the
released v0.15.0 engine; the only variable is the engine version. Committed as
2026-W27-reeval.{jsonl,csv} (off the *-mined glob by design) and pinned by
test_w27_reeval_* reading the committed artifact.

Result (9/19 moved):
- Scan crashes cleared 4/4 — the #256 action_id crash-degrade holds on real
  history; every goose scan_failed row now evaluates.
- Both real must_block PRs move off abstention: stripe/ai#232 IE→
  human_review_required (clean receipt), crewAIInc#169 IE→review_required
  (cold-start; verify receipt fails base_lock_failed on both eras). But neither
  is a hard block, so real-history blocked_recall STAYS 0.0 — reliable 1.0
  remains the constructed stratum's to prove.
- #232's move is driven by SHIP-SCOPE-TOOLKIT-UNBOUNDED (substantively the
  least-privilege removal), NOT the new #256 SHIP-CAP-CONFIG-BINDING-* checks,
  which do not fire — honest: right catch, different mechanism.
- benign_escalation_rate 0.0→0.286 (4/14 safe→review), dominated by the
  cold-start-scans-the-whole-repo artifact (openai enumerates 150+ test
  fixtures; goose#9717 its ~240-tool MCP API though the PR removes capability),
  not a product precision regression.

Also corrects a data-integrity finding: the W26 "stripe/agent-toolkit" stratum
is stripe/ai (gh follows the transfer redirect into the monorepo), so the
corpus spans 8 distinct repos, not 9. Updates the top-level README banner and
the miner README (run table + note, forward-reference, 9→8 repos).

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* W27 review: match goose repo name to the artifact; drop "capability-bearing" overstatement

- The committed 2026-W27-reeval artifact records the goose rows as
  aaif-goose/goose (GitHub redirects the former block/goose there — verified
  `gh api repos/block/goose` → aaif-goose/goose). Make the W27 run-table row,
  delta table, and crash bullet use aaif-goose/goose with a "formerly
  block/goose" note, so the docs match the data file. W26 references stay
  block/goose (that is what W26's artifact recorded).
- The four additional review outcomes are all labeled `safe` (and mostly
  cold-start whole-repo artifacts), so the reading no longer calls them
  "capability-bearing changes" — now "four additional (safe) rows."

…blocks #232

Working the two "next legs" from the W27 re-eval — turn a true must_block from
review into block, and measure escalation under an adopter-realistic scope —
shows they are one lever: manifest scope. It is NOT a severity problem.

- The engine already carries the critical, release-blocking diff check for
  least-privilege weakening (SHIP-VERIFY-CAPABILITY-SCOPE-BROADENED). Measured on
  the REAL stripe/ai#232 clone trees: a manifest scoped to the one agent file
  (type: openai_agents_sdk, path: support_agent.py) → base insufficient_evidence,
  base→head BLOCKED (scope_bound_removed: [billing_portal_sessions:create,
  customers:read, invoices:read] → []), 3 review items. The whole-repo cold-start
  (init emits 16+ sources incl. the toolkit library + every example) dilutes to
  review_required + 8-9 whole-surface items — the W27 verdict AND the
  benign_escalation. So both W27 legs are the whole-monorepo cold-start artifact;
  scoping to the agent fixes both. Real-history blocked_recall 0 → 1 under
  adopter-realistic scoping.
- Honest boundary: the other must_block (crewAIInc#169) is a net-new addition,
  not a removal (outside the check by design), and its Slack/Trello writes are
  flow-level requests.post/chat_postMessage helpers, not statically-enumerable
  crewai tools — so no scope blocks it. Adopter-scoped blocked_recall is 1/2.

Adds test_pr232_scoped_verdict_is_precise_not_whole_surface pinning the
precision half (block half already pinned by test_pr232_bound_removal_blocks_
release); documents the finding in the miner README W27 follow-up section; and
corrects the top-level README banner to call the 0.0 a cold-start scoping
artifact. Full suite green.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

… to the PR (#456)

Addresses the five review findings. Each was reproduced before the fix and
re-checked after; the guard sweep is nine perturbations plus two policy moves.

**[P1] Exposure is now recorded, and it decides the split.** Forcing every
external PR to `either` was wrong: holdout means the engine was never tuned
on the case, which is a fact about this project's history, not about where
the bytes live. Six candidates are engine-development inputs -- #3020,
#3076, #4489 and #1745 through `test_surface_exclusions`, `triggers.py`,
`test_init_scaffold_disclosure` and `test_declaration_questionnaire`;
stripe/ai#232 through a committed fixture tree; and
openai-agents-python#3392, which produced
`test_capability_change_schema_hash_parity.py` and the fix behind it. A new
`exposure` column carries them, `split_eligibility` is derived from it, and
a detector re-reads the tree as a *floor*, so a row may declare more than is
findable but never less. `maintainer_walk` has no detector -- mcp-grafana#1080
appears nowhere in this repository and still drove `tool_sources[].binding` --
so `diff_substance` implies it, closing the one exposure anybody could drop.
Three `mcp_openapi_declared_binding` cells gain a third slot whose only job
is to be holdout-eligible.

**[P2] The holdout floor is computed.** `ceil(count * fraction)` from the
policy, not a hard-coded one. At 0.60 the inventory now fails in 14 cells.

**[P2] Profile and merge state are bound to the register.** Every sourced
candidate has a register entry; swapping two candidates' profiles now fails.
For the five source-type profiles a sample's declared manifest type is
checked too; `coding_agent_trust_roots` and `multi_agent_handoffs` are
scenario profiles with nothing mechanical to check against, which the
register says.

**[P1] awslabs/mcp#4489 was misdescribed and misplaced.** It adds two
literal FastMCP entrypoints, `budget-actions` and `budget-notifications`,
taking the budget surface 1 -> 3; both read-only. Static literals are exactly
what a grep-the-literal extraction reads, so this never supported an
`insufficient_evidence` hypothesis. Moved to the reserve as a `passed`
alternate; the cell is now a gap leading to a genuinely non-enumerable
surface.

**[P2] An origin is a fact about the PR.** adk-python#6605 was closed
without merge, so the gap naming it plans `rejected_or_reverted`. Checking
this also caught adk-samples#1745, which is still **open** -- not history at
all -- so it leaves its slot for the reserve.

**Recommendation taken.** `human_label` is renamed `miner_label` and
disclosed as *not verifier-independent*: the labeling worksheet carries
`head_decision`, `verify_verdict` and `verify_can_merge`, and LABELING.md
tells the labeler those are enough to label without opening the diff. That
biases cell targeting; it does not reach the corpus label, which Amendment 1
raters produce blind. A test fails if the worksheet ever stops exposing
verdicts, so the disclosure cannot outlive its cause.

One judgment is flagged rather than settled: `benchmark_scored` does not by
itself block holdout, on the rule that being measured is not being tuned on.
If any W24-W26 scoring drove engine changes not traceable to a test or a
walk, those subjects need reclassifying, and only the owner can say.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>

…rpus (#456) (#477)

* feat(benchmark): commit the Cut A strata inventory for the pre-1.0 corpus (#456)

Maps the known candidate pool onto all 28 profile x decision cells the
`pre_1_0` policy requires, so Cut B mines the empty cells instead of
re-finding the full ones. 56 slots, two per cell: each is a pinned
candidate, an identified-but-unpinned one, or a gap carrying the lead
that would close it.

It is a sourcing plan, not evidence. No label, no verdict, no receipt;
nothing in it reaches the qualification runner. It is also not an
admissible rater input -- it names a target decision for every slot, so
Amendment 1's blindness condition makes a session that has read it
unable to produce an admissible label. `target_basis` is a closed
vocabulary with no verifier-derived member, because a corpus assembled
to match the engine's own verdicts cannot measure the engine.

`tests/test_strata_inventory.py` derives the grid from
`pre_release_safety_requirements()` rather than restating it, closing
the one definition site the decision document says no gate can detect.
It also re-reads every cited miner label and pinned SHA from its source,
and refuses the two escapes: restating a labeled subject's basis as
`diff_substance`, or dropping the pins off a subject an existing sweep
already resolved.

The plan's shape is the finding: 28 of 56 slots have a candidate, the
origin floor (23 qualifying cases) binds well before the case count,
`n8n` has one sourced slot of eight, and the sweeps behind the pool ran
before #403 -- so their zero-trigger repositories are unexplored rather
than empty.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>

* fix(benchmark): close the origin-floor rename and reground the candidate notes (#456)

Review of the inventory guards found four issues, all reproduced by
perturbing the CSV and confirmed fixed the same way.

1. Nothing bound an in-tree candidate to `synthetic`, so relabeling the
   twelve `samples/` slots `real_history` reported 41 of 56 qualifying
   origins against a floor of 23 while adding no real evidence -- the
   origin floor satisfied by renaming. What a candidate *is* now decides
   both its origin and its split, checked against the path, and the
   holdout test consumes that decision instead of restating it.
2. `sample_design` was the one basis whose evidence was never
   cross-checked: a row could cite any directory that happened to exist.
   It must now cite the sample it is about.
3. Slot numbering was validated in file order, so re-sorting the CSV to
   review origin coverage failed a test on byte-identical content. Cell
   membership and 1..n numbering are now checked as sets; naming another
   cell, or numbering 1,4, still fails.
4. The `.template.csv` skip in the label reader was unreachable -- the
   `*.labels.csv` glob never yields one. Replaced with a comment saying
   why the pattern, not the branch, is what excludes them.

Also regrounds all fifteen `human_label` notes in the miner rationales
they cite. Several paraphrases were wrong in a way that matters for a
sourcing artifact: #3451 is a least-privilege tightening release, not a
"runtime handling fix"; #232 removes the toolkit's action and permission
bounds rather than merely migrating to MCP; #312 rewires the skill supply
chain from an authenticated endpoint to an unauthenticated fetch, which
is why it is a trust-root case at all.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>

* test(benchmark): bind the inventory's reading to the plan it describes (#456)

Second review round. `strata-inventory.md` restates the CSV in three
tables -- 28 of 56 sourced, the per-profile shortfall, the per-outcome
scarcity -- and those numbers are what a corpus owner reads to decide
where to mine. Nothing kept them in sync, so the first added candidate
would leave the reading behind and send the next person to a cell the
plan calls empty and the file calls full.

Every number on the page is now recomputed from the CSV. The per-outcome
figures move from prose into a table so one parser covers all three, and
a duplicate numeric row label fails rather than silently shadowing the
table nobody then checks.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>

* test(benchmark): scope the register check to the register (#456)

Third review round, two narrow fixes.

The `diff_substance` check searched the whole document, so a candidate
listed only under **Reserve** -- the section for candidates deliberately
*not* placed in a cell -- satisfied a row claiming a register entry. It
now reads only the register section, and a per-profile row missing from
the reading fails with a message instead of a KeyError.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>

* fix(benchmark): record exposure per slot, and bind profile and origin to the PR (#456)

Addresses the five review findings. Each was reproduced before the fix and
re-checked after; the guard sweep is nine perturbations plus two policy moves.

**[P1] Exposure is now recorded, and it decides the split.** Forcing every
external PR to `either` was wrong: holdout means the engine was never tuned
on the case, which is a fact about this project's history, not about where
the bytes live. Six candidates are engine-development inputs -- #3020,
#3076, #4489 and #1745 through `test_surface_exclusions`, `triggers.py`,
`test_init_scaffold_disclosure` and `test_declaration_questionnaire`;
stripe/ai#232 through a committed fixture tree; and
openai-agents-python#3392, which produced
`test_capability_change_schema_hash_parity.py` and the fix behind it. A new
`exposure` column carries them, `split_eligibility` is derived from it, and
a detector re-reads the tree as a *floor*, so a row may declare more than is
findable but never less. `maintainer_walk` has no detector -- mcp-grafana#1080
appears nowhere in this repository and still drove `tool_sources[].binding` --
so `diff_substance` implies it, closing the one exposure anybody could drop.
Three `mcp_openapi_declared_binding` cells gain a third slot whose only job
is to be holdout-eligible.

**[P2] The holdout floor is computed.** `ceil(count * fraction)` from the
policy, not a hard-coded one. At 0.60 the inventory now fails in 14 cells.

**[P2] Profile and merge state are bound to the register.** Every sourced
candidate has a register entry; swapping two candidates' profiles now fails.
For the five source-type profiles a sample's declared manifest type is
checked too; `coding_agent_trust_roots` and `multi_agent_handoffs` are
scenario profiles with nothing mechanical to check against, which the
register says.

**[P1] awslabs/mcp#4489 was misdescribed and misplaced.** It adds two
literal FastMCP entrypoints, `budget-actions` and `budget-notifications`,
taking the budget surface 1 -> 3; both read-only. Static literals are exactly
what a grep-the-literal extraction reads, so this never supported an
`insufficient_evidence` hypothesis. Moved to the reserve as a `passed`
alternate; the cell is now a gap leading to a genuinely non-enumerable
surface.

**[P2] An origin is a fact about the PR.** adk-python#6605 was closed
without merge, so the gap naming it plans `rejected_or_reverted`. Checking
this also caught adk-samples#1745, which is still **open** -- not history at
all -- so it leaves its slot for the reserve.

**Recommendation taken.** `human_label` is renamed `miner_label` and
disclosed as *not verifier-independent*: the labeling worksheet carries
`head_decision`, `verify_verdict` and `verify_can_merge`, and LABELING.md
tells the labeler those are enough to label without opening the diff. That
biases cell targeting; it does not reach the corpus label, which Amendment 1
raters produce blind. A test fails if the worksheet ever stops exposing
verdicts, so the disclosure cannot outlive its cause.

One judgment is flagged rather than settled: `benchmark_scored` does not by
itself block holdout, on the rule that being measured is not being tuned on.
If any W24-W26 scoring drove engine changes not traceable to a test or a
walk, those subjects need reclassifying, and only the owner can say.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>

* test(benchmark): read the two register tables apart, and score samples too (#456)

Two loose spots in the guards added for the review, found on a self-review
pass and each reproduced by perturbation.

The register parser read the Candidate register and the Reserve as one
table. They share a `State` column, but the Reserve's second column is an
*origin*, not a profile — so a reserved candidate was filed with
`real_history` as its profile name, and a slot filled from the Reserve
would have been "registered" against a profile that is not one. The two
sections are now read apart, a Reserve entry carries no profile, and a
sourced row that resolves to one fails.

The exposure detector applied `benchmark_scored` only to external
candidates, so the constructed sweep's scoring of `fixture://` samples went
undeclared on four rows. Applied uniformly. It changes no eligibility —
those rows are `shipped_sample` already — but a detector that skips a whole
class of participation is a detector nobody can reason about.