Repository navigation
Corpus delivery: current 38-case pre-1.0 and 80-case beta qualification tracks #456
Description
Current delivery contract — 2026-09-08
This remains the cross-release data/evidence parent, not an additional release gate. #572 owns the v1.0 go/no-go and sequencing. The policy approved in #519 is implemented; this issue owns delivery against it, not a new policy.
| Track | Live requirement | Delivery |
|---|---|---|
| Optional 0.x | 38 cases: 14 passed / 14 review-required / 10 blocked, 21 strata, at least 16 qualifying origins | Verify any #508 frozen-data handoff; #509 produces/validates exact receipts and promotion evidence |
| 1.0, including prerelease tags | 80-case beta: 30 passed / 20 review-required / 30 blocked, 21 weighted strata, at least 32 qualifying origins | #512 human primary labels and final-wheel evidence → #509 independent signing/promotion → #510 exact-candidate rehearsal |
Both keep κ≥0.80, per-stratum holdout≥20%, zero unsafe auto-passes and the approved exact-outcome floors. Actual insufficient-evidence remains a coverage miss; it is not a ground-truth class. Current qualification results are v6, corpus/index v4; #569 coordinates the deliberate report 1.0 compatibility change before final v1.0 receipts.
Beta sourcing starts now, in parallel with engineering. Neither a v0.16 publication nor completed pre-1.0 qualification is a prerequisite for 1.0. Do not promote agent-produced pre-1.0 labels into beta human labels, or count the old miner labels as an adjudicated corpus.
Remaining work
- Deliverable 2: the 80-case human-labeled beta corpus required for 1.0 (#456) #512 delivers the separately human-labeled beta corpus, accountable freeze and exact final-wheel receipts/index/unsigned qualifying result.
- Cut D: exact-candidate evidence, independent qualification promotion and publication handoff (#456) #509 validates the applicable handoff, establishes the independent signer and promotes the exact signed bytes/locations.
- Rehearse substantive qualification rejection and the exact final candidate before publication (#456) #510 records meaningful negative rejection and a positive final-candidate rehearsal without publication.
- The approved active pre-1.0 retirement is coordinated with Freeze the report 1.0 contract and prove migration before calling the product v1.0 #569 before final source/wheel freeze; historical reader compatibility is explicit.
- Gate 2: participant validation of the frozen corpus (#456) #511 remains separately tracked, non-gating historical participant validation. It cannot be used to hold a 1.0 tag that otherwise meets its requirements.
- Reconcile live qualification instructions and delivery contracts with the approved 38/80 policy #566 reconciles current operator instructions and delivery children, preserving visibly historical counts.
#508 is closed, but a closed issue is not the frozen-data handoff. Private label material is intentionally ignored by Git; verify its hashes, adjudication, exposure and admissibility when using that track. Do not infer either absence or completion of private work from repository visibility alone.
Standing constraints
Fixtures, historical miner labels, beta primary labels, candidate receipts and pilot outcomes answer different questions. Holdout eligibility is a fact about exposure; a fix prompted by a case makes that case tuning material. Scope must be reviewed before reading candidate outcomes; no fabricated effects, authority or deployment bindings. #563/#564's failed historical catch bars and #559/#567 integrity repairs remain release prerequisites under #572.
The original sourcing and restructuring record remains below as history. Its 56/100 counts, expected-IE class and “0.x first” sequence are superseded, not active instructions.
Historical plan — superseded counts and sequencing; retained for rationale
Follows the decision in #341 (Route 2, recorded by Pengfei Hu on 2026-08-29 in
docs/release-evidence-policy-decision.md),
with the labeling protocol fixed by Amendment 1 (#476).
The policy is approved and enforced; this issue is the data work.
Restructured 2026-09-03. The original checklist mixed tag-blocking work with
1.0-track work in one list, which made "is #456 done?" unanswerable and left
1.0items sitting in thev0.16.0milestone. The work is now five sub-issues,
split by what blocks it rather than by checkbox. This issue is the tracking
parent and holds no work of its own.It is deliberately unmilestoned: the three sub-issues that block the tag carry
v0.16.0, so that milestone closes when the tag can ship. This parent outlives it —
it is not done until #512 retires thepre_1_0tier — and putting it in a release
milestone would keep that milestone open until1.0, which is the defect this
restructure exists to remove.
What the deliverable is
A signed safety-qualification.json that says how the engine scored on 56 adjudicated,
receipt-bound cases — exactly two in each of the 28 profile × decision strata — bound
to the exact release wheel. Until it exists, the release pipeline fails closed at
Verify production qualification and exact wheel binding, and v0.16.0 cannot publish.
Two cases per cell rather than the production weighting scaled down: a zero-count cell
deletes a profile/outcome pair's only evidence, which is a strictness reduction. It also
stops a good average from hiding a profile the engine is useless on.
| Requirement | Value |
|---|---|
| Cases | 56 — 14 each passed / review_required / insufficient_evidence / blocked |
| Strata | 2 per (profile, decision), all 28 non-empty |
| Split | ≥ 1 holdout per stratum, computed as ceil(2 × 0.20). A floor: marking both holdout is accepted |
| Origins | ≥ 23 real_history / rejected_or_reverted / design_partner |
| Cohen's κ | ≥ 0.80 across the two blind primary labels |
| Unsafe auto-passes | 0, per profile and overall |
| Receipts | one terminal verify receipt per case, unique digest, report schema 0.43 |
The exact shape is in benchmark/safety-qualification/README.md, and
tests/test_safety_qualification.py::test_a_conforming_56_case_corpus_qualifies_a_0_x_wheel
builds a machine-checkable reference corpus of exactly this shape.
Blocking the v0.16.0 tag
| Sub-issue | State | |
|---|---|---|
| Cut C | #508 — produce and freeze the 56 × 2 blind labels | Owner-gated: two rater sessions on different model families, owner adjudicates and is never a rater. Four preconditions still open |
| Cut D | #509 — receipts, the signed artifact, the promotion variables | Blocked on #508 |
| Gate | #510 — rehearse the release gate against an artifact that misses the bar | Actionable today; depends on neither labels nor receipts |
Not blocking the tag
| Sub-issue | Why it is here | |
|---|---|---|
| Gate 2 | #511 — participant validation of the frozen corpus | Non-gating by decision. Cannot start before receipts exist |
| Deliverable 2 | #512 — the 100-case beta corpus |
Required for 1.0, not for 0.x. No promotion shortcut: pre_1_0 evidence cannot qualify a 1.0 tag, by construction |
Done — the sourcing, in four merged PRs
- docs(release): record the pre-1.0 labeling protocol and the participant-validation gate (#456) #476 — Amendment 1 recorded before any label existed: two independent agent sessions on different model families, owner adjudicates every disagreement, six admissibility conditions, plus the non-gating participant-validation gate.
- feat(benchmark): commit the Cut A strata inventory for the pre-1.0 corpus (#456) #477 — Cut A: the 28-cell strata inventory committed as
strata-inventory.csv,
every cell showing its candidates and origin class. Review established the rules the rest
of the plan runs on: an origin is a fact about the PR, holdout is a fact about this
project's development history, and every floor is computed from the policy rather than
restated. - Staff Cut B across three sessions and prepare Cut C for the pre-1.0 corpus (#456) #489 — Cut B: three parallel sessions coordinating through the CSV. 15 constructions,
the first post-Epic: account for every narrowing decision — exclusion ledger, conservation invariant, and skip-requires-evidence #403 sweep (912 rows over 21 repositories, including the rejected vein),
and the Cut C preparation — five calibration cases, the rater rubric, and both harnesses.
59 of 60 slots sourced. - Close out Cut B: pin the walked candidates and source the last cell (#456) #507 — Cut B close-out: the last cell sourced and the three walked candidates pinned.
60 of 60 slots pinned, 33 qualifying origins against a floor of 23, every cell keeping
a holdout-eligible slot. Also restored the pin guard that Cut A's own review commit had
deleted while adding the prose citing it.
Sourcing is finished. Nothing in the inventory needs mining before Cut C.
Standing constraints, for whoever picks this up
- Fixtures are not cases. test(fixtures): add Strix #791 as OpenAI Agents SDK positive regression fixture #310 and test(google-adk): add PR #6605 as a static tool-surface regression fixture #316 produce minimized in-tree fixtures,
synthetic
by construction. The qualifying case for the same PR is the actual repository at pinned
base/head SHAs run through the external-PR recipe, with receipts. One source PR feeds both. - The 32 rows under
benchmark/miner/results/*.labels.csvare not corpus labels. They
carry neither adjudication nor receipts. The PRs they point at were eligible raw material;
the labels never were. - No threshold moves. A corpus that cannot meet the bar is a corpus problem; lowering
the bar again needs a fresh decision on release policy: decide the approved evidence bar for pre-1.0 tags #341's terms. - A sourcing change makes no engine change. A fix made in response to a candidate is
what turns that candidate into tuning material.
Done when
- The labeling-protocol decision is recorded before any label exists — Amendment 1, docs(release): record the pre-1.0 labeling protocol and the participant-validation gate (#456) #476.
- The strata inventory is committed, showing every one of the 28 cells with its
candidate(s) and origin class — feat(benchmark): commit the Cut A strata inventory for the pre-1.0 corpus (#456) #477, completed to 60/60 pinned by Close out Cut B: pin the walked candidates and source the last cell (#456) #507. - Cut C: produce and freeze the 56 × 2 blind labels (#456) #508, Cut D: exact-candidate evidence, independent qualification promotion and publication handoff (#456) #509 and Rehearse substantive qualification rejection and the exact final candidate before publication (#456) #510 are closed, and
v0.16.0can be tagged. - Gate 2: participant validation of the frozen corpus (#456) #511 and Deliverable 2: the 80-case human-labeled beta corpus required for 1.0 (#456) #512 are closed. Neither gates the tag; Deliverable 2: the 80-case human-labeled beta corpus required for 1.0 (#456) #512 gates
1.0.
Activity
docs(release): link the corpus-delivery issue and correct the definit…
docs(release): link the corpus-delivery issue and correct the definit…
feat(release): approve and enforce an explicit pre-1.0 evidence bar (#…
pengfei-threemoonslab commentedon Aug 31, 2026
Milestone: attached to v0.16.0 — by this issue's own summary it is the tag's blocking work.
pengfei-threemoonslab commentedon Aug 31, 2026
Cut C is unblocked: the labeling protocol is decided and recorded as Amendment 1 (#476) — two independent agent sessions on different model families, owner adjudicates every disagreement, six admissibility conditions including archived per-label transcripts. A new non-gating Gate 2 (participant validation) sends each real-history case's author/reviewer a case card after receipts exist; its five pre-registered rules are in the amendment and the card/template/log live in
benchmark/safety-qualification/participant-validation.md. Body updated accordingly; execution order is now Cut A (inventory) → Cut B (mining) → calibration round → labels → freeze → receipts → Gate 2 outreach.docs(release): record the pre-1.0 labeling protocol and the participa…
fix(benchmark): close the origin-floor rename and reground the candid…
test(benchmark): bind the inventory's reading to the plan it describes (
test(benchmark): scope the register check to the register (#456)
fix(benchmark): record exposure per slot, and bind profile and origin…
test(benchmark): read the two register tables apart, and score sample…
11 remaining items
pengfei-threemoonslab commentedon Sep 3, 2026
Restructured into five sub-issues (#508, #509, #510, #511, #512), split by what blocks each rather than by checkbox. The old checklist mixed tag-blocking work with
1.0-track work, which is why "is this done?" had no clean answer — and whyv0.16.0was the only milestone, holding an item (Deliverable 2) that by definition cannot be finished before1.0.Blocking
v0.16.0: #508 (labels) → #509 (receipts + signed artifact + variables), plus #510, which is actionable today and depends on neither.Not blocking: #511 (non-gating by decision, and cannot start before receipts exist) and #512 (required for
1.0; no promotion shortcut frompre_1_0evidence). Both are unmilestoned — there is no1.0milestone in this repository, and creating one is a product decision rather than bookkeeping.One thing worth flagging: the release rehearsal was orphaned, not merely unstarted. #341 is closed and delegates it to #355; #355 was closed on 2026-08-18 with this criterion still unticked — "At least one deliberate mismatch/failure path is rehearsed and shown to fail closed before publication" — and
release-rehearsal.ymlhas run 0 times. So the gate has only ever been exercised in the direction that passes. That is now #510, rather than a pointer at two closed issues.pengfei-threemoonslab commentedon Sep 4, 2026
From #520:
insufficient_evidenceleaves the ground-truth vocabulary, so "exactly two in each of the 28 profile × decision strata" becomes 21 strata, and the 56 needs restating (21 × 2 = 42, or 56 by raising the per-cell count). The three real decisions' floors are unchanged. Also see the note on #508 about the 12 shipped-sample slots that have nobase//head/form.[-]corpus delivery: build the 56-case pre-1.0 qualification artifact (and the 100-case 1.0 bar)[/-][+]Corpus delivery: current 38-case pre-1.0 and 80-case beta qualification tracks[/+]pengfei-threemoonslab commentedon Sep 10, 2026
Deferred historical-exposure follow-up from #512 / merged PR #624 (main
e004e2ae3edf8a184a55e732ebe865e9cbfcd449): the beta inventory now preserves public calibration/development exposure from #256 (all ten W26 subjects) and #582 (ADK #1975/#1977), including cases whose derived fixtures have generic names. Their old Cut A rows can still sayeitherbecause the older register and literal tests/src detector did not capture those records.This is concrete input to this parent's existing exposure/admissibility audit and #509's handoff validation, not a new qualification policy or a new v1.0 prerequisite. Before any optional pre-1.0 handoff is reused, reconcile those known facts against the proposed split and record the resulting shortfall/replacement requirement; do not infer a clean holdout history from an absent source-code reference. Preserve historical label/receipt bytes and require accountable re-freeze if an actual frozen handoff is affected. The current beta register is corrected; no private corpus has been inspected or relabeled here.
The related ADK #125/#2148 graph/reimplementation must also retain its family/split obligation. #624 records it without asserting independent diversity or human admission. This residual audit stays deferred under the existing corpus/handoff issues rather than opening a duplicate or silently changing the optional historical track.
pengfei-threemoonslab commentedon Oct 7, 2026
Closing: #572 is the single qualified-gate ledger over the same children (#509–#512). Standing constraints and the 2026-09-10 exposure-audit comment stay linked from #512.