feat(billing-cost-management): add budget-actions and budget-notifications operations - #4489

… to the PR (#456)

Addresses the five review findings. Each was reproduced before the fix and
re-checked after; the guard sweep is nine perturbations plus two policy moves.

**[P1] Exposure is now recorded, and it decides the split.** Forcing every
external PR to `either` was wrong: holdout means the engine was never tuned
on the case, which is a fact about this project's history, not about where
the bytes live. Six candidates are engine-development inputs -- #3020,
#3076, #4489 and #1745 through `test_surface_exclusions`, `triggers.py`,
`test_init_scaffold_disclosure` and `test_declaration_questionnaire`;
stripe/ai#232 through a committed fixture tree; and
openai-agents-python#3392, which produced
`test_capability_change_schema_hash_parity.py` and the fix behind it. A new
`exposure` column carries them, `split_eligibility` is derived from it, and
a detector re-reads the tree as a *floor*, so a row may declare more than is
findable but never less. `maintainer_walk` has no detector -- mcp-grafana#1080
appears nowhere in this repository and still drove `tool_sources[].binding` --
so `diff_substance` implies it, closing the one exposure anybody could drop.
Three `mcp_openapi_declared_binding` cells gain a third slot whose only job
is to be holdout-eligible.

**[P2] The holdout floor is computed.** `ceil(count * fraction)` from the
policy, not a hard-coded one. At 0.60 the inventory now fails in 14 cells.

**[P2] Profile and merge state are bound to the register.** Every sourced
candidate has a register entry; swapping two candidates' profiles now fails.
For the five source-type profiles a sample's declared manifest type is
checked too; `coding_agent_trust_roots` and `multi_agent_handoffs` are
scenario profiles with nothing mechanical to check against, which the
register says.

**[P1] awslabs/mcp#4489 was misdescribed and misplaced.** It adds two
literal FastMCP entrypoints, `budget-actions` and `budget-notifications`,
taking the budget surface 1 -> 3; both read-only. Static literals are exactly
what a grep-the-literal extraction reads, so this never supported an
`insufficient_evidence` hypothesis. Moved to the reserve as a `passed`
alternate; the cell is now a gap leading to a genuinely non-enumerable
surface.

**[P2] An origin is a fact about the PR.** adk-python#6605 was closed
without merge, so the gap naming it plans `rejected_or_reverted`. Checking
this also caught adk-samples#1745, which is still **open** -- not history at
all -- so it leaves its slot for the reserve.

**Recommendation taken.** `human_label` is renamed `miner_label` and
disclosed as *not verifier-independent*: the labeling worksheet carries
`head_decision`, `verify_verdict` and `verify_can_merge`, and LABELING.md
tells the labeler those are enough to label without opening the diff. That
biases cell targeting; it does not reach the corpus label, which Amendment 1
raters produce blind. A test fails if the worksheet ever stops exposing
verdicts, so the disclosure cannot outlive its cause.

One judgment is flagged rather than settled: `benchmark_scored` does not by
itself block holdout, on the rule that being measured is not being tuned on.
If any W24-W26 scoring drove engine changes not traceable to a test or a
walk, those subjects need reclassifying, and only the owner can say.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>

…rpus (#456) (#477)

* feat(benchmark): commit the Cut A strata inventory for the pre-1.0 corpus (#456)

Maps the known candidate pool onto all 28 profile x decision cells the
`pre_1_0` policy requires, so Cut B mines the empty cells instead of
re-finding the full ones. 56 slots, two per cell: each is a pinned
candidate, an identified-but-unpinned one, or a gap carrying the lead
that would close it.

It is a sourcing plan, not evidence. No label, no verdict, no receipt;
nothing in it reaches the qualification runner. It is also not an
admissible rater input -- it names a target decision for every slot, so
Amendment 1's blindness condition makes a session that has read it
unable to produce an admissible label. `target_basis` is a closed
vocabulary with no verifier-derived member, because a corpus assembled
to match the engine's own verdicts cannot measure the engine.

`tests/test_strata_inventory.py` derives the grid from
`pre_release_safety_requirements()` rather than restating it, closing
the one definition site the decision document says no gate can detect.
It also re-reads every cited miner label and pinned SHA from its source,
and refuses the two escapes: restating a labeled subject's basis as
`diff_substance`, or dropping the pins off a subject an existing sweep
already resolved.

The plan's shape is the finding: 28 of 56 slots have a candidate, the
origin floor (23 qualifying cases) binds well before the case count,
`n8n` has one sourced slot of eight, and the sweeps behind the pool ran
before #403 -- so their zero-trigger repositories are unexplored rather
than empty.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>

* fix(benchmark): close the origin-floor rename and reground the candidate notes (#456)

Review of the inventory guards found four issues, all reproduced by
perturbing the CSV and confirmed fixed the same way.

1. Nothing bound an in-tree candidate to `synthetic`, so relabeling the
   twelve `samples/` slots `real_history` reported 41 of 56 qualifying
   origins against a floor of 23 while adding no real evidence -- the
   origin floor satisfied by renaming. What a candidate *is* now decides
   both its origin and its split, checked against the path, and the
   holdout test consumes that decision instead of restating it.
2. `sample_design` was the one basis whose evidence was never
   cross-checked: a row could cite any directory that happened to exist.
   It must now cite the sample it is about.
3. Slot numbering was validated in file order, so re-sorting the CSV to
   review origin coverage failed a test on byte-identical content. Cell
   membership and 1..n numbering are now checked as sets; naming another
   cell, or numbering 1,4, still fails.
4. The `.template.csv` skip in the label reader was unreachable -- the
   `*.labels.csv` glob never yields one. Replaced with a comment saying
   why the pattern, not the branch, is what excludes them.

Also regrounds all fifteen `human_label` notes in the miner rationales
they cite. Several paraphrases were wrong in a way that matters for a
sourcing artifact: #3451 is a least-privilege tightening release, not a
"runtime handling fix"; #232 removes the toolkit's action and permission
bounds rather than merely migrating to MCP; #312 rewires the skill supply
chain from an authenticated endpoint to an unauthenticated fetch, which
is why it is a trust-root case at all.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>

* test(benchmark): bind the inventory's reading to the plan it describes (#456)

Second review round. `strata-inventory.md` restates the CSV in three
tables -- 28 of 56 sourced, the per-profile shortfall, the per-outcome
scarcity -- and those numbers are what a corpus owner reads to decide
where to mine. Nothing kept them in sync, so the first added candidate
would leave the reading behind and send the next person to a cell the
plan calls empty and the file calls full.

Every number on the page is now recomputed from the CSV. The per-outcome
figures move from prose into a table so one parser covers all three, and
a duplicate numeric row label fails rather than silently shadowing the
table nobody then checks.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>

* test(benchmark): scope the register check to the register (#456)

Third review round, two narrow fixes.

The `diff_substance` check searched the whole document, so a candidate
listed only under **Reserve** -- the section for candidates deliberately
*not* placed in a cell -- satisfied a row claiming a register entry. It
now reads only the register section, and a per-profile row missing from
the reading fails with a message instead of a KeyError.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>

* fix(benchmark): record exposure per slot, and bind profile and origin to the PR (#456)

Addresses the five review findings. Each was reproduced before the fix and
re-checked after; the guard sweep is nine perturbations plus two policy moves.

**[P1] Exposure is now recorded, and it decides the split.** Forcing every
external PR to `either` was wrong: holdout means the engine was never tuned
on the case, which is a fact about this project's history, not about where
the bytes live. Six candidates are engine-development inputs -- #3020,
#3076, #4489 and #1745 through `test_surface_exclusions`, `triggers.py`,
`test_init_scaffold_disclosure` and `test_declaration_questionnaire`;
stripe/ai#232 through a committed fixture tree; and
openai-agents-python#3392, which produced
`test_capability_change_schema_hash_parity.py` and the fix behind it. A new
`exposure` column carries them, `split_eligibility` is derived from it, and
a detector re-reads the tree as a *floor*, so a row may declare more than is
findable but never less. `maintainer_walk` has no detector -- mcp-grafana#1080
appears nowhere in this repository and still drove `tool_sources[].binding` --
so `diff_substance` implies it, closing the one exposure anybody could drop.
Three `mcp_openapi_declared_binding` cells gain a third slot whose only job
is to be holdout-eligible.

**[P2] The holdout floor is computed.** `ceil(count * fraction)` from the
policy, not a hard-coded one. At 0.60 the inventory now fails in 14 cells.

**[P2] Profile and merge state are bound to the register.** Every sourced
candidate has a register entry; swapping two candidates' profiles now fails.
For the five source-type profiles a sample's declared manifest type is
checked too; `coding_agent_trust_roots` and `multi_agent_handoffs` are
scenario profiles with nothing mechanical to check against, which the
register says.

**[P1] awslabs/mcp#4489 was misdescribed and misplaced.** It adds two
literal FastMCP entrypoints, `budget-actions` and `budget-notifications`,
taking the budget surface 1 -> 3; both read-only. Static literals are exactly
what a grep-the-literal extraction reads, so this never supported an
`insufficient_evidence` hypothesis. Moved to the reserve as a `passed`
alternate; the cell is now a gap leading to a genuinely non-enumerable
surface.

**[P2] An origin is a fact about the PR.** adk-python#6605 was closed
without merge, so the gap naming it plans `rejected_or_reverted`. Checking
this also caught adk-samples#1745, which is still **open** -- not history at
all -- so it leaves its slot for the reserve.

**Recommendation taken.** `human_label` is renamed `miner_label` and
disclosed as *not verifier-independent*: the labeling worksheet carries
`head_decision`, `verify_verdict` and `verify_can_merge`, and LABELING.md
tells the labeler those are enough to label without opening the diff. That
biases cell targeting; it does not reach the corpus label, which Amendment 1
raters produce blind. A test fails if the worksheet ever stops exposing
verdicts, so the disclosure cannot outlive its cause.

One judgment is flagged rather than settled: `benchmark_scored` does not by
itself block holdout, on the rule that being measured is not being tuned on.
If any W24-W26 scoring drove engine changes not traceable to a test or a
walk, those subjects need reclassifying, and only the owner can say.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>

* test(benchmark): read the two register tables apart, and score samples too (#456)

Two loose spots in the guards added for the review, found on a self-review
pass and each reproduced by perturbation.

The register parser read the Candidate register and the Reserve as one
table. They share a `State` column, but the Reserve's second column is an
*origin*, not a profile — so a reserved candidate was filed with
`real_history` as its profile name, and a slot filled from the Reserve
would have been "registered" against a profile that is not one. The two
sections are now read apart, a Reserve entry carries no profile, and a
sourced row that resolves to one fails.

The exposure detector applied `benchmark_scored` only to external
candidates, so the constructed sweep's scoring of `fixture://` samples went
undeclared on four rows. Applied uniformly. It changes no eligibility —
those rows are `shipped_sample` already — but a detector that skips a whole
class of participation is a detector nobody can reason about.

…orpus (#456) (#489)

* feat(benchmark): settle the Cut B sourcing contract — constructions, pins, and claims (#456)

Cut B is staffed as three parallel sessions coordinating through the
inventory CSV. Before they start, the contract they claim slots against
has to exist: where a corpus-built synthetic lives (never samples/),
what makes it a case (base/ and head/ that differ, CASE.md beside them,
no engine output inside), how real history enters (a sweep, then a miner
label, so the slot stays holdout-eligible), and the pin form for
closed-unmerged and reverted PRs, which the merge-commit convention
cannot express.

The guard gains a constructed_design basis, a by-name engine_tests
detector for constructions, and reads n8n as the top-level section it
is rather than a tool-source type it is not.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* feat(benchmark): enumerate closed-unmerged and reverted PRs in the miner (#456)

The safety-qualification corpus needs a rejected_or_reverted vein that
merged history cannot supply. `mine --state closed` enumerates PRs closed
without merging, pinned at the fork point and the PR head;
`--state reverted` finds merged PRs a later Revert PR undid, pinned like
any merged PR with the revert recorded in notes; `--pr N` mines one named
PR in whichever state it is in. Merged enumeration is untouched.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* docs(benchmark): bring the labeling guide to the four-way corpus vocabulary (#456)

Restructure benchmark/miner/LABELING.md so the top of the file is a
self-contained rater rubric for passed / review_required /
insufficient_evidence / blocked, with a first-draft rule for the
review_required vs insufficient_evidence line, constructed illustrations
that name no corpus case, a mapping to the miner's three labels, and the
JSON output contract. The miner process moves below a 'not a rater input'
heading and the real-PR anchor that stated a corpus case's label and
verifier verdicts is removed.

A guard test keeps the guide free of inventory candidate refs (both
spellings), slot samples, and SHIP- check IDs, and requires the four
decisions as headings above the miner section.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* feat(benchmark): add the rater packet builder and session harnesses for Cut C (#456)

build_packet.py turns an external (clone + base/head SHAs) or constructed
(base/ + head/ trees) case into the three inputs Amendment 1 condition 2
allows: repo/ (the head tree, no .git, verifier output and CASE.md
excluded), diff.patch, a byte copy of LABELING.md, plus a role-specific
TASK.md and a MANIFEST.json hashing every file. It refuses a source tree
that carries the strata inventory.

run_rater.py runs one fresh read-only session per family (claude -p with
Read/Grep/Glob only and no MCP; codex exec --sandbox read-only), archives
the complete transcript content-addressed, and parses the final message
into an IndependentHumanLabelV1 with reviewer_id <family>:<model>:<session>.
Anything but exactly one valid JSON object with a known decision fails
closed. --home-mode isolated runs --bare in an empty HOME (needs
ANTHROPIC_API_KEY); shared keeps the caller's HOME under file-level
memory checks. --dry-run prints the command and environment.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* docs(benchmark): pick the five calibration cases (#456)

Name the five non-corpus cases for the Amendment 1 calibration round:
four merged reserve PRs pinned as merge commit + first parent from a
clone (mongodb-js/mongodb-mcp-server#1417, awslabs/mcp#4489,
stripe/ai#353, openai/openai-agents-python#3518) and one constructed
blocked-shaped LangChain case under calibration/cal-5/, since no reserve
PR has that shape. Each entry records why the diff tests the guide and
where two raters are expected to diverge; no label or verdict is
recorded. The file discloses that the chooser had read the inventory,
which is why none of these can ever become a corpus case.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* fix(benchmark): skip a revert title that names something that is not a PR (#456)

aaif-goose/goose has a merged revert whose quoted title carries a
truncated `(#65…` reference; `gh pr view 65` fails and the whole
reverted enumeration aborted. One bad reference now skips that revert.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* feat(benchmark): construct the MCP declared-binding synthetic case for Cut B (#456)

mcp_export_adds_undeclared_tool: a tool server with a complete root declaration publishes a fourth tool the declaration omits (the #432 shape), built outside samples/ so the slot stays holdout-eligible.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* feat(benchmark): construct the OpenAI Agents SDK synthetic cases for Cut B (#456)

sdk_agent_adds_ticket_update_tool adds a scoped Zendesk write with no approval policy; sdk_agent_loads_tools_from_registry replaces the literal tools list with a runtime registry so the bound surface is not enumerable.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* feat(benchmark): construct the LangChain and CrewAI synthetic cases for Cut B (#456)

crewai_tools_from_factory builds the crew's tools from a profile factory; langchain_agent_adds_refund_tool adds a Stripe refund tool with neither an approval policy nor idempotency evidence.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* feat(benchmark): construct the Google ADK synthetic cases for Cut B (#456)

adk_agent_adds_calendar_toolset adds a domain-scoped calendar write through a resolved McpToolset (the adk-samples#1975 shape); adk_billing_sub_agent_refund puts a Stripe refund on a sub-agent (the adk-samples#1745 shape) with no approval policy.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* feat(benchmark): construct the n8n synthetic cases and reserves for Cut B (#456)

Five slot cases on one support-orders workflow family: a POST tool with no approval, a Code Tool whose request target is built at run time, a Call Workflow Tool with an expression target, a DELETE tool behind a public webhook, and a disabled send-and-wait approval node. Two reserves: a read-only GET tool and an allowlisted MCP Client Tool write.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* feat(benchmark): construct the multi-agent handoff synthetic cases and reserve for Cut B (#456)

A triage root with an Agents SDK handoff to a billing sub-agent: a write added on the sub-agent, handoffs resolved from an environment route list, and an approvals sub-agent that decides the refund requests the root submits. One reserve adds a scoped read on the sub-agent.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* feat(benchmark): claim the Session A constructions in the strata inventory (#456)

Fifteen synthetic gap slots move to pinned with target_basis constructed_design, exposure none and their CASE.md as evidence; the register gains one row per construction and three reserve rows; the summary tables and the shape prose are recomputed from the CSV.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* feat(benchmark): sweep n8n, unwalked MCP servers and the rejected vein for Cut B (#456)

2026-W36-cutb: 912 rows over 21 repositories, the first post-#403 run.
Every n8n repository this project had never mined, eight unwalked MCP
servers, and closed-unmerged / reverted / named PRs from the new miner
states. Labeled from the PR diffs as one session's Cut B cell-targeting
labels, not adjudicated. Named off the *-mined glob because it is a
selected, capability-dense sample (trigger-skip 0.71), not the unselected
history the cross-run noise-bound guard measures; the README says why.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* feat(benchmark): claim the slots sourced by the W36 sweep (#456)

Eighteen slots pinned from 2026-W36-cutb: seventeen of the eighteen gaps
Cut B session B owned, plus a third google_adk x insufficient_evidence
slot because adk-python#6605 turned out to be engine_tests exposure.
Every rejected_or_reverted slot is filled from five repositories; n8n has
its first two real-history candidates. The guard gains a `reverted` state
(landed history whose rejection came afterwards) mapping to
rejected_or_reverted only. langchain_crewai.insufficient_evidence.1 stays
a gap with the re-mine's findings as its lead. Summary tables recomputed.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* docs(benchmark): strike the calibration cases from the reserve and reread the shape after Cut B (#456)

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* fix(benchmark): close three holes in the blind-rater boundary (#456)

Review findings on #489, all in the Cut C rater harness.

1. The packet is re-hashed at launch. _check_packet() proved five paths
   existed and parsed the manifest; it never compared the packet against
   it. Any edit landing after the build -- a note under repo/, a rewritten
   guide, a doctored diff -- reached the rater while the label recorded the
   hash of a manifest describing a packet that no longer existed. prepare()
   now calls verify_manifest() and refuses before a session is launched.

2. Isolated mode gives codex its own home. HOME was replaced and then
   CODEX_HOME pointed back at the caller's real profile, which is where
   codex reads global AGENTS.md, AGENTS.override.md, and a config.toml
   that can mount MCP servers and enable web search -- so the mode named
   isolated was neither blind nor packet-limited. It now builds a fresh
   Codex home per run holding one written config (no MCP servers, web
   search off, read-only sandbox, no approvals, no history) and
   authenticates through OPENAI_API_KEY, symmetric with the Claude path;
   shared mode keeps the real profile only after proving it is silent.

3. Symlinks cannot carry unmanifested content. The copy preserved links
   verbatim and the hasher skipped them, so a link out of the tree was
   content the rater's read tools resolve and the manifest cannot
   describe -- and the packet still verified. A link that escapes the tree
   or dangles now refuses the build; one that stays inside is materialised
   so its bytes are hashed; a link found in a packet is tamper, not a skip.

Each fix is covered by tests that fail without it.