Skip to content

feat(benchmark): commit the Cut A strata inventory for the pre-1.0 corpus (#456) - #477

Merged
pengfei-threemoonslab merged 6 commits into
mainfrom
claude/issue-456-review-d41603
Aug 31, 2026
Merged

pengfei-threemoonslab merged 6 commits into
mainfrom
claude/issue-456-review-d41603

Conversation

@pengfei-threemoonslab

@pengfei-threemoonslab pengfei-threemoonslab commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

Delivers Cut A of #456 — the strata inventory the issue's execution plan puts first ("first, cheap, decides everything else") and the third box on its Done-when list.

Review round 4 (owner, changes requested). All five findings were real and are fixed in 958a907; each was reproduced by perturbation before the fix and re-checked after. The recommendation on human_label was taken. Two of the requested checks caught further errors of the same kind — see What the review changed.

What lands

benchmark/safety-qualification/strata-inventory.csv maps the known candidate pool onto all 28 profile × decision cells the pre_1_0 policy requires. 59 slots — three mcp_openapi_declared_binding cells carry a third — each one of:

  • pinned — ready to hand to a rater: an external PR with both full SHAs, or an in-tree sample, which this repository's own history pins;
  • unpinned — candidate identified, base/head not yet resolved. An abbreviated SHA is a display form, never a pin;
  • gap — no candidate, carrying the lead that would close it.

strata-inventory.md is how to read and maintain it; README.md points at both from the corpus runbook.

What it is not

Not evidence. No label, no verdict, no receipt. Nothing in it reaches the qualification runner and nothing in it can qualify a release.

Not a rater input. Amendment 1 condition 2 gives a rater session exactly three things: the pinned repository state, the PR diff, and benchmark/miner/LABELING.md. The inventory names a target decision for every slot, so a session that has read it produces no admissible label.

Not verdict-aimed — but miner_label is not verifier-independent either, and the register says so. No target_basis is taken from verifier output. But the labeling worksheet carries head_decision, verify_verdict and verify_can_merge, and LABELING.md tells the labeler those are enough to label without opening the diff — so a miner_label row's cell targeting was made with the verdict in view. That biases which cell a candidate is aimed at; it does not reach the corpus label, which Amendment 1 raters produce blind. Every such row carries miner_label in exposure, and a test fails if the worksheet ever stops exposing verdicts, so the disclosure cannot outlive its cause.

Exposure decides holdout eligibility

Holdout means evidence the engine was never tuned on — a fact about this project's development history, not about where a candidate's bytes live. Each slot records its exposure; split_eligibility is derived from it.

exposure Blocks holdout?
engine_tests — named in this repository's tests/ or src/ yes
maintainer_walk — walked during development; findings drove engine changes yes
shipped_sample — an in-tree samples/ path yes
benchmark_scored — appears in a committed miner sweep no
miner_label — a committed label from a verifier-exposed worksheet no

A detector re-reads the tree on every run as a floor — a row may declare more than is findable, never less. maintainer_walk has no detector (grafana/mcp-grafana#1080 appears nowhere in this repository and still drove tool_sources[].binding), so diff_substance implies it.

The guard

tests/test_strata_inventory.py (19 tests) derives the 28 cells and the holdout floor from pre_release_safety_requirements(). docs/release-evidence-policy-decision.md names this directory as the one definition site whose omission no gate can detect. Verified by perturbing the policy: stratum count 2 → 3, an eighth profile, origin floor 23 → 40, and holdout fraction 0.20 → 0.60 (which now fails in 14 cells) each fail the inventory.

It re-reads what the inventory claims rather than trusting transcription — every cited miner label, every pinned SHA, every declared exposure, and every candidate's profile and merge state, each from the source that records it. Escapes closed explicitly: restating a labeled subject's basis as diff_substance; dropping pins the sweep already resolved; claiming a qualifying origin for a shipped sample; declaring less exposure than the tree shows; and planning an origin the named PR cannot supply.

The plan's shape is the finding

Slots with a candidate 26 of 59
Planned as a qualifying origin 32 (floor 23) — 14 sourced, 18 still to find
Planned synthetic 27 (ceiling 33)
Engine-development inputs 17
  • Every vendor MCP server this project has walked is disqualified from holdout use. #3020, #3076, awslabs/mcp#4489 and mcp-grafana#1080 each produced an issue, a fix, or a regression test. The best-understood material and the admissible material are close to disjoint — the most expensive fact in this plan, and why three MCP cells need a third slot.
  • The origin floor binds, not the case count. The plan reaches 32 only by committing to mine 18 more — over half.
  • insufficient_evidence then blocked are the scarce outcomes — 3 of 14 and 5 of 15 sourced. Merged history is the wrong place to look for blocked.
  • n8n can fail this deliverable: one sourced slot of eight, never mined.
  • The pool was swept by a build that no longer exists. langgraph, modelcontextprotocol/servers and pydantic-ai were each swept 40 PRs deep and triggered zero times on a pre-Epic: account for every narrowing decision — exclusion ledger, conservation invariant, and skip-requires-evidence #403 build. Those zeroes are not empty cells.
  • The pinning convention is unsettled. The sweeps pin the merge commit and its parent; the GitHub API returns the base branch tip today and the PR branch head. For stripe/ai#232 these are entirely different commits. The sweep form is correct here and must not be "corrected" from the API.

What the review changed

  1. [P1] Exposure. Six candidates are engine-development inputs, including one the reviewer did not list: openai-agents-python#3392 produced test_capability_change_schema_hash_parity.py and the fix behind it.
  2. [P2] Holdout floor computed as ceil(count × fraction) from the policy.
  3. [P2] Profile + state bound to the register for all 26 sourced candidates. Sample profiles are additionally checked against the declared manifest type for the five source-type profiles; coding_agent_trust_roots and multi_agent_handoffs are scenario profiles with nothing mechanical to check against, which the register states.
  4. [P1] awslabs/mcp#4489 was misdescribed — I had registered the repository's registration idiom from a walk, not the PR's diff. It adds two literal FastMCP entrypoints (budget-actions, budget-notifications), exactly what a grep-the-literal extraction reads, so it never supported insufficient_evidence. Moved to the reserve; the cell is now a gap for a genuinely non-enumerable surface.
  5. [P2] Origin is a fact about the PR. adk-python#6605 is closed-unmerged. The check for it also caught that adk-samples#1745 is still open — not history at all — so it leaves its slot for the reserve.

Two guard bugs found while building this: a per-candidate tree scan took 34s (now 0.56s, one indexed pass), and the guard file names every candidate by construction, so the detector matched its own docstrings.

One judgment is flagged, not settled. benchmark_scored does not by itself block holdout, on the rule that being measured is not being tuned on. If any W24–W26 scoring drove engine changes not traceable to a test or a walk, those subjects need reclassifying — a question about this project's history that only the owner can settle.

Not in this PR, and why

  • Labels/receipts/the signed artifact need two independent agent rater sessions on different model families plus owner adjudication. The runbook is explicit that this repository does not ship fabricated human labels, and a self-produced label would violate Amendment 1's independence condition outright.
  • Gate 2 outreach messages real PR authors and reviewers — the owner's call.
  • SAFETY_QUALIFICATION_* variables and the Release Rehearsal dispatch need repository-admin and CI-dispatch authority.
  • The benign-escalation re-measurement is Cut D's rider.

The first Done-when box — the labeling protocol recorded before any label exists — is satisfied: Amendment 1 landed in #476 and this change creates no label.

Refs #456

🤖 Generated with Claude Code

pengfei-threemoonslab and others added 4 commits August 31, 2026 11:31
…rpus (#456)

Maps the known candidate pool onto all 28 profile x decision cells the
`pre_1_0` policy requires, so Cut B mines the empty cells instead of
re-finding the full ones. 56 slots, two per cell: each is a pinned
candidate, an identified-but-unpinned one, or a gap carrying the lead
that would close it.

It is a sourcing plan, not evidence. No label, no verdict, no receipt;
nothing in it reaches the qualification runner. It is also not an
admissible rater input -- it names a target decision for every slot, so
Amendment 1's blindness condition makes a session that has read it
unable to produce an admissible label. `target_basis` is a closed
vocabulary with no verifier-derived member, because a corpus assembled
to match the engine's own verdicts cannot measure the engine.

`tests/test_strata_inventory.py` derives the grid from
`pre_release_safety_requirements()` rather than restating it, closing
the one definition site the decision document says no gate can detect.
It also re-reads every cited miner label and pinned SHA from its source,
and refuses the two escapes: restating a labeled subject's basis as
`diff_substance`, or dropping the pins off a subject an existing sweep
already resolved.

The plan's shape is the finding: 28 of 56 slots have a candidate, the
origin floor (23 qualifying cases) binds well before the case count,
`n8n` has one sourced slot of eight, and the sweeps behind the pool ran
before #403 -- so their zero-trigger repositories are unexplored rather
than empty.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>
…ate notes (#456)

Review of the inventory guards found four issues, all reproduced by
perturbing the CSV and confirmed fixed the same way.

1. Nothing bound an in-tree candidate to `synthetic`, so relabeling the
   twelve `samples/` slots `real_history` reported 41 of 56 qualifying
   origins against a floor of 23 while adding no real evidence -- the
   origin floor satisfied by renaming. What a candidate *is* now decides
   both its origin and its split, checked against the path, and the
   holdout test consumes that decision instead of restating it.
2. `sample_design` was the one basis whose evidence was never
   cross-checked: a row could cite any directory that happened to exist.
   It must now cite the sample it is about.
3. Slot numbering was validated in file order, so re-sorting the CSV to
   review origin coverage failed a test on byte-identical content. Cell
   membership and 1..n numbering are now checked as sets; naming another
   cell, or numbering 1,4, still fails.
4. The `.template.csv` skip in the label reader was unreachable -- the
   `*.labels.csv` glob never yields one. Replaced with a comment saying
   why the pattern, not the branch, is what excludes them.

Also regrounds all fifteen `human_label` notes in the miner rationales
they cite. Several paraphrases were wrong in a way that matters for a
sourcing artifact: #3451 is a least-privilege tightening release, not a
"runtime handling fix"; #232 removes the toolkit's action and permission
bounds rather than merely migrating to MCP; #312 rewires the skill supply
chain from an authenticated endpoint to an unauthenticated fetch, which
is why it is a trust-root case at all.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>
#456)

Second review round. `strata-inventory.md` restates the CSV in three
tables -- 28 of 56 sourced, the per-profile shortfall, the per-outcome
scarcity -- and those numbers are what a corpus owner reads to decide
where to mine. Nothing kept them in sync, so the first added candidate
would leave the reading behind and send the next person to a cell the
plan calls empty and the file calls full.

Every number on the page is now recomputed from the CSV. The per-outcome
figures move from prose into a table so one parser covers all three, and
a duplicate numeric row label fails rather than silently shadowing the
table nobody then checks.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>
Third review round, two narrow fixes.

The `diff_substance` check searched the whole document, so a candidate
listed only under **Reserve** -- the section for candidates deliberately
*not* placed in a cell -- satisfied a row claiming a register entry. It
now reads only the register section, and a per-profile row missing from
the reading fails with a message instead of a KeyError.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>

@pengfei-threemoonslab pengfei-threemoonslab left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert review outcome: changes requested.

The PR head and all CI checks are green. I reproduced the 15 new tests, ran the 222 inventory and qualification or release tests, and ran the full PR verifier (control_state=complete, merge_verdict=mergeable). Those checks do not cover the benchmark-design failures below. The holdout and provenance issue is release-relevant and should be fixed before Cut A is treated as complete.

I left five inline findings. I also recommend revisiting target_basis=human_label: the W24-W26 labeling workflow exposed verifier verdicts to labelers, so using those labels to select cells is not as independent from verifier output as this PR claims.

Comment thread tests/test_strata_inventory.py Outdated
assert row["origin_class"] in QUALIFYING_ORIGINS, (
f"{row['slot_id']} names an upstream PR, which is not synthetic material"
)
assert row["split_eligibility"] == "either", row["slot_id"]

@pengfei-threemoonslab pengfei-threemoonslab Aug 31, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Track exposure before declaring a holdout candidate

The policy defines holdout as evidence the engine was never tuned on, but this branch forces every external PR to either. Several current rows are known development inputs: github-mcp-server#3020 is named in test_surface_exclusions.py as the case that showed the defect; #3076 drives trigger regressions; awslabs/mcp#4489 drives init regression coverage; adk-samples#1745 drives declaration ranking and ADK tests; and stripe/ai#232 is a validation anchor. The mcp_openapi_declared_binding.passed cell then counts #3020 as its only holdout-capable slot although both current cases are tuning material. This can yield an invalid qualification corpus. Store reviewed exposure provenance explicitly, allow external rows to be tuning_only, and add new holdout candidates or gaps for starved cells.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 958a907 — this was the right call and the class was wider than the five you named.

A new exposure column records engine-development exposure per slot, and split_eligibility is derived from it rather than from the candidate's path. A detector re-reads tests/ and src/ on every run as a floor: a row may declare more exposure than is findable, never less. It found a sixth case you did not list — openai/openai-agents-python#3392 produced tests/test_capability_change_schema_hash_parity.py and the fix behind it, so a miner_label row was tuning material too.

maintainer_walk is the hard part: grafana/mcp-grafana#1080 appears nowhere in this repository and still drove tool_sources[].binding. Nothing can detect it, so diff_substance now implies it — the only way this project read that diff was by walking the repository. Dropping it from a walked row fails.

Consequence, and it is the headline of the plan now: all four walked vendor MCP servers are disqualified from holdout use, so three mcp_openapi_declared_binding cells carry a third slot whose only job is to be holdout-eligible. The best-understood material and the admissible material are close to disjoint.

Comment thread tests/test_strata_inventory.py Outdated
starved = sorted(
(stratum.profile, stratum.expected_decision)
for stratum in requirements.required_strata
if eligible[(stratum.profile, stratum.expected_decision)] < 1

@pengfei-threemoonslab pengfei-threemoonslab Aug 31, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Compute the holdout floor from the policy

This check hard-codes one eligible case instead of ceil(stratum.count * minimum_holdout_fraction_per_stratum). With the policy mocked to 0.60, this test still passes while 12 cells have one eligible slot and require two. Use the policy-derived ceiling per stratum so the inventory actually fails when the holdout rule moves.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 958a907. The floor is now math.ceil(stratum.count * requirements.minimum_holdout_fraction_per_stratum), taken per stratum from the policy object.

Verified the way you described: with the fraction moved to 0.60 the inventory fails in 14 cells. Confirmed the policy edit actually reached the test — a same-length restore inside the same second leaves __pycache__ valid and silently keeps the perturbed bytecode, which cost me three confusing failures earlier in this PR.

if row["target_basis"] != "diff_substance":
continue
assert row["evidence_ref"].split("#", 1)[0] == REGISTER.relative_to(REPO_ROOT).as_posix()
assert f"`{row['candidate_ref']}`" in register, (

@pengfei-threemoonslab pengfei-threemoonslab Aug 31, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Bind each candidate to its declared profile

The register already has a structured Profile column, but this test only searches for the candidate string. Swapping google/adk-samples#1745 and github/github-mcp-server#3020 between the handoff and MCP rows passes all row guards even though the register assigns the opposite profiles. Because qualification counts the corpus-declared profile, this can falsely satisfy per-profile coverage. Parse a unique candidate-to-profile mapping and assert it exactly; add equivalent machine-checkable profile provenance for sample and label-backed rows.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 958a907, and extended to every sourced row rather than the diff_substance ones.

The register now carries a Profile and a State column for all 26 sourced candidates, parsed into a unique candidate → (profile, state) mapping and asserted exactly. Swapping two candidates' register profiles fails.

For the sample and label-backed rows you asked about, machine-checkable provenance is uneven and I did not paper over it: for the five source-type profiles a sample's declared manifest type: is checked against the profile, but coding_agent_trust_roots and multi_agent_handoffs are scenario profiles — samples/agent_weakens_gate declares type: mcp and belongs to neither MCP cell — so for those the register entry is the justification and there is nothing to compare it against. samples/n8n_workflow_agent ships workflow JSON with no manifest at all; it is named in an explicit unchecked list rather than silently skipped, so a newly manifest-less sample has to be considered.

| `github.com/github/github-mcp-server#3020` | `mcp_openapi_declared_binding` | Adds `find_duplicate`, a read-only tool gated behind a `duplicate_detection` feature flag. The repository checks in per-tool MCP schemas under `pkg/github/__toolsnaps__/`; the count goes 115 → 116. The flag is invisible in the schema — the filename encodes it, the JSON does not. |
| `github.com/github/github-mcp-server#3076` | `mcp_openapi_declared_binding` | Adds a confirmed repository-deletion tool to the same 117-schema surface. Walk notes record base `bfb59bb7…` and head `5ea9a0e8…` as abbreviations only; both must be resolved to full SHAs. |
| `github.com/grafana/mcp-grafana#1080` | `mcp_openapi_declared_binding` | Adds `update_incident` to a Go MCP server registering tools as `mcpgrafana.MustTool("…", …)`. The published surface goes 99 → 100. |
| `github.com/awslabs/mcp#4489` | `mcp_openapi_declared_binding` | Python/FastMCP monorepo registering tools as `@budget_server.tool(name='budgets', …)`. No checked-in export exists, so the surface has to be read from code. |

@pengfei-threemoonslab pengfei-threemoonslab Aug 31, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Reassess this candidate from the actual diff

awslabs/mcp#4489 does not add the existing budgets entrypoint cited here. Its patch adds two literal FastMCP decorators, budget-actions and budget-notifications, and documents both as read-only. Their names and entrypoints are statically visible in Python, so the absence of a checked-in export does not by itself support an insufficient_evidence hypothesis. Update the register and CSV to the actual 1-to-3 surface and reassess placement, or use a genuinely dynamic or non-enumerable candidate for this cell.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, and I had not read the diff — the register described the repository's registration idiom observed during a walk, not this PR. Fixed in 958a907.

#4489 adds two literal @budget_server.tool entrypoints, budget-actions and budget-notifications, taking the budget tool surface from 1 to 3; both documented read-only, both requiring new budgets:DescribeBudgetActionsFor* permissions.

On placement I took your second option. Static literals are exactly what a grep-the-literal extraction reads, so this never supported an insufficient_evidence hypothesis — and all four walked vendor servers register with literals, so none of them can supply that cell. #4489 moves to the reserve as a mcp_openapi_declared_binding × passed alternate (with its engine_tests exposure recorded), and the cell becomes a gap whose lead asks for a server that assembles its tool list at runtime.

google_adk.review_required.1,google_adk,review_required,real_history,pinned,either,github.com/google/adk-samples#1975,ca6776f841dc19f8c0ec1400b46f34a7e7cc41cd,4f8c74236398c5255180de8ae6bef935d4840409,human_label,benchmark/miner/results/2026-W25-mined.labels.csv,,"Adds a travel agent with an McpToolset against the Google Maps MCP endpoint, exposing search_places, lookup_weather and compute_routes: net-new external tool surface, read-only and domain-scoped but authority-bearing."
google_adk.review_required.2,google_adk,review_required,synthetic,gap,either,,,,unsourced,,Construct from samples/google_adk_agent: add a domain-scoped external write tool - the adk-samples#1975 shape - with no approval policy.,
google_adk.insufficient_evidence.1,google_adk,insufficient_evidence,synthetic,pinned,tuning_only,samples/google_adk_cold_start_agent,,,sample_design,samples/google_adk_cold_start_agent,,ADK agent with no declarations at all; the cold-start shape whose extraction confidence is capped.
google_adk.insufficient_evidence.2,google_adk,insufficient_evidence,real_history,gap,either,,,,unsourced,,"google/adk-python#6605 was identified during the #385 sub-agent work but its diff has not been read. Read it before placing it in a cell, and pin base and head.",

@pengfei-threemoonslab pengfei-threemoonslab Aug 31, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Match the planned origin to the named lead

google/adk-python#6605 was closed without merge on 2026-08-13, but this row plans real_history. If #6605 is the intended candidate, its origin is rejected_or_reverted; otherwise the named lead cannot fill this planned origin. Correct the origin or the lead and add an origin-provenance check for specifically named gaps.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 958a907. #6605 is closed-unmerged, so the gap now plans rejected_or_reverted, and the register records its state and title.

The origin-provenance check you asked for is in, and it caught a third error I had not seen: google/adk-samples#1745 is still open. It was filling multi_agent_handoffs.blocked.1 as real_history. An open PR is not history — there is no landed change for the counterfactual to be about, and Gate 2's "reviewers outrank authors" rule presupposes a decision that resolved it — so it moves to the reserve as the shape to match, and the cell is a gap.

The check is offline: merged → real_history/design_partner, closed → rejected_or_reverted, open → cannot fill a slot, read from the register's State column for both sourced rows and any gap whose lead names a PR.

One related thing while pinning: the sweep pins and the GitHub API disagree. For stripe/ai#232 the sweep records 5af4bcd1… → cd8cee57…; the API reports dd624f51… → 4d201d8e…, which is the base branch tip today and the PR branch head. The sweep convention is the correct one and the pins must not be "corrected" from the API — documented, with the guard reading them back from the sweep.

… to the PR (#456)

Addresses the five review findings. Each was reproduced before the fix and
re-checked after; the guard sweep is nine perturbations plus two policy moves.

**[P1] Exposure is now recorded, and it decides the split.** Forcing every
external PR to `either` was wrong: holdout means the engine was never tuned
on the case, which is a fact about this project's history, not about where
the bytes live. Six candidates are engine-development inputs -- #3020,
#3076, #4489 and #1745 through `test_surface_exclusions`, `triggers.py`,
`test_init_scaffold_disclosure` and `test_declaration_questionnaire`;
stripe/ai#232 through a committed fixture tree; and
openai-agents-python#3392, which produced
`test_capability_change_schema_hash_parity.py` and the fix behind it. A new
`exposure` column carries them, `split_eligibility` is derived from it, and
a detector re-reads the tree as a *floor*, so a row may declare more than is
findable but never less. `maintainer_walk` has no detector -- mcp-grafana#1080
appears nowhere in this repository and still drove `tool_sources[].binding` --
so `diff_substance` implies it, closing the one exposure anybody could drop.
Three `mcp_openapi_declared_binding` cells gain a third slot whose only job
is to be holdout-eligible.

**[P2] The holdout floor is computed.** `ceil(count * fraction)` from the
policy, not a hard-coded one. At 0.60 the inventory now fails in 14 cells.

**[P2] Profile and merge state are bound to the register.** Every sourced
candidate has a register entry; swapping two candidates' profiles now fails.
For the five source-type profiles a sample's declared manifest type is
checked too; `coding_agent_trust_roots` and `multi_agent_handoffs` are
scenario profiles with nothing mechanical to check against, which the
register says.

**[P1] awslabs/mcp#4489 was misdescribed and misplaced.** It adds two
literal FastMCP entrypoints, `budget-actions` and `budget-notifications`,
taking the budget surface 1 -> 3; both read-only. Static literals are exactly
what a grep-the-literal extraction reads, so this never supported an
`insufficient_evidence` hypothesis. Moved to the reserve as a `passed`
alternate; the cell is now a gap leading to a genuinely non-enumerable
surface.

**[P2] An origin is a fact about the PR.** adk-python#6605 was closed
without merge, so the gap naming it plans `rejected_or_reverted`. Checking
this also caught adk-samples#1745, which is still **open** -- not history at
all -- so it leaves its slot for the reserve.

**Recommendation taken.** `human_label` is renamed `miner_label` and
disclosed as *not verifier-independent*: the labeling worksheet carries
`head_decision`, `verify_verdict` and `verify_can_merge`, and LABELING.md
tells the labeler those are enough to label without opening the diff. That
biases cell targeting; it does not reach the corpus label, which Amendment 1
raters produce blind. A test fails if the worksheet ever stops exposing
verdicts, so the disclosure cannot outlive its cause.

One judgment is flagged rather than settled: `benchmark_scored` does not by
itself block holdout, on the rule that being measured is not being tuned on.
If any W24-W26 scoring drove engine changes not traceable to a test or a
walk, those subjects need reclassifying, and only the owner can say.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>
@pengfei-threemoonslab

Copy link
Copy Markdown
Contributor Author

All five findings fixed in 958a907, replies inline. Answering the recommendation here, since it was the most consequential part of the review.

On target_basis=human_label: you were right, and I had the claim backwards. I checked *.labels.template.csv — the worksheet carries head_decision, verify_verdict, verify_can_merge and verify_trust_root_touched, and LABELING.md tells the labeler those are "enough PR context … to label most rows without opening the diff." So cell targeting from a miner label was made with the engine's verdict in view, and this PR's claim of a verifier-independent target_basis was false as written.

What I did rather than soften the wording:

  • Renamed the basis human_label → miner_label. The old name also collided with Amendment 1's IndependentHumanLabelV1, which is the thing it is not.
  • The claim is now the narrower true one: no basis is taken from verifier output, and miner_label is explicitly not verifier-independent. It biases which cell a candidate is aimed at; it does not reach the corpus label, which Amendment 1 raters produce blind and without this file.
  • The disclosure is bound to its cause: test_the_miner_label_basis_is_disclosed_as_verifier_exposed fails if the worksheet ever stops exposing verdicts, so it cannot outlive the thing that makes it necessary.
  • Every affected row carries miner_label in exposure, so the bias is visible per row instead of argued in prose.

Two of your requested checks caught errors beyond the ones you named, which I think validates asking for the check rather than the correction:

  • The exposure detector found a sixth engine-development input — openai/openai-agents-python#3392 produced tests/test_capability_change_schema_hash_parity.py and the fix behind it. It was a passed slot marked holdout-capable.
  • The origin-provenance check found that google/adk-samples#1745 is still open. It was filling multi_agent_handoffs.blocked.1 as real_history. An open PR is not history — there is no landed change for the counterfactual, and Gate 2's "reviewers outrank authors" rule presupposes a decision that resolved it. It moves to the reserve and the cell is a gap.

One judgment I flagged rather than settled. benchmark_scored does not by itself block holdout, on the rule that being measured is not being tuned on. Two subjects already crossed that line and earned engine_tests instead. But if any of the W24–W26 scoring drove engine changes that are not traceable to a test or a walk, those subjects need reclassifying — that is a question about this project's own history, and I did not want to answer it silently.

Net effect on the plan. Sourced slots drop 28 → 26 of 59, and all four walked vendor MCP servers are now disqualified from holdout use, so three mcp_openapi_declared_binding cells carry a third slot whose only job is to be holdout-eligible. The best-understood material in this pool and the admissible material are close to disjoint — which is a worse position than the PR previously described, and the right one to plan Cut B from.

…s too (#456)

Two loose spots in the guards added for the review, found on a self-review
pass and each reproduced by perturbation.

The register parser read the Candidate register and the Reserve as one
table. They share a `State` column, but the Reserve's second column is an
*origin*, not a profile — so a reserved candidate was filed with
`real_history` as its profile name, and a slot filled from the Reserve
would have been "registered" against a profile that is not one. The two
sections are now read apart, a Reserve entry carries no profile, and a
sourced row that resolves to one fails.

The exposure detector applied `benchmark_scored` only to external
candidates, so the constructed sweep's scoring of `fixture://` samples went
undeclared on four rows. Applied uniformly. It changes no eligibility —
those rows are `shipped_sample` already — but a detector that skips a whole
class of participation is a detector nobody can reason about.

Refs #456

Co-Authored-By: Claude Opus 5 <[email protected]>
@pengfei-threemoonslab
pengfei-threemoonslab merged commit 08fc598 into main Aug 31, 2026
6 checks passed
pengfei-threemoonslab added a commit that referenced this pull request Aug 31, 2026
One conflict, and not a semantic one: #477 (the Cut A strata inventory, #456)
and this branch both inserted an entry at the top of `## Unreleased`, so git
had no basis to order them. Both are kept — main's entry holds the position it
has on main, and this branch's follows it. Verified byte-for-byte on both
sides rather than by eye, because a hand-resolved changelog is where a
paragraph goes missing quietly.

Everything else main brought in — `benchmark/safety-qualification/*` and
`tests/test_strata_inventory.py` — is disjoint from the adapters, the
generator, and the report surfaces this branch touches. Confirmed by re-running
`generate_schemas.py --check` on the merged tree: nothing upstream moves a
generated artifact. Full suite, lint, and the perf and adapter-static legs are
green on the merge itself.

Co-Authored-By: Claude Opus 5 <[email protected]>
pengfei-threemoonslab added a commit that referenced this pull request Sep 1, 2026
Two commits landed on main while this was in review: #477 (the pre-1.0 corpus
strata inventory) and #478 (the published determinism boundary).

Only `CHANGELOG.md` conflicted, and only because both sides prepended a new
entry under `## Unreleased` — no line either side wrote was edited by the other.
Resolved by keeping all three entries, with main's two left where main put them
so this merge adds a block rather than reordering one; the resolution is 66
insertions and no deletions against main.

`scripts/generate_schemas.py` and `docs/INDEX.md` auto-merged. Both needed
checking rather than trusting, because #478 also registers generated artifacts
in the same `BUILDERS` table and consolidates `extraction_is_complete()` in the
semantic resolver — which feeds the capability facts the payload projects, and
therefore the committed examples. Verified after the merge:

- `generate_schemas.py --check` is clean, and the merged `BUILDERS` table
  carries both this branch's three capability-payload entries and #478's two.
- `docs/capability-payload-schema.v1.json` and both worked examples are
  byte-identical to before the merge, so the resolver consolidation did not move
  the facts they derive from.
- The full non-perf suite, `ruff`, `compileall` and `git diff --check` all pass
  on the merged tree.

Co-Authored-By: Claude Opus 5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant