Skip to content

Verdict integrity round 2: bind rows to sealed waves, strict single-lane authorization, immutable waves, sealed refutations, retained packets - #124

Merged
seathatflowsinourveins merged 7 commits into
mainfrom
claude/verdict-integrity-2-20260923
Sep 23, 2026
Merged

seathatflowsinourveins merged 7 commits into
mainfrom
claude/verdict-integrity-2-20260923

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator

Summary

Closes the #122 review findings (agent-lab-e9, review122/findings.md) before the 20260923 re-record of all 32 rows. No ledger row changes. The regression tests fail on 97a56e2: 26 failures and 4 errors.

  1. Rows bound to their sealed returns. For new-wave rows, landscape.py recomputes the agreement from both sealed returns' winner_keys, resolved through the retained packet. The winners must equal the chosen lane's (same_winner or codex_absent: claude; disagree: the adjudication's winner_lane). A row that isn't recorded carries no winners. Relabelling agreement or swapping in the losing lane's winners fails.
  2. Strict single-lane authorization. The record must live under docs/decisions/ and contain the exact line single-lane-authorization: <catalog>/<layer_id>. Its sha256 is stored as lanes.single_lane_decision_sha256 and re-checked.
  3. Immutable waves.
    • --write refuses once run-manifest.json exists, unless --append-rows names only absent rows.
    • It never overwrites a sealed file with different bytes.
    • Rows store lanes.run_manifest_sha256 and, for disagree rows, lanes.adjudication_sha256. The run manifest also lists adjudications.
    • Files in a sealed folder that nothing references are rejected, and so is a malformed wave folder.
  4. Sealed refutations.
    • A Claude return is rejected unless its refutation summary is unrefuted and both lens votes on the sealed round are false. This is checked at record time and in CI.
    • The lane (agent-lab Adopt native Claude profile with pinned community practice and review evidence #33, e6953e5) is re-vendored byte for byte (sha256 2eb9c10d…7fbb).
    • claude_lane.py copies the refutation summary and lists layers without a final in failures.json.
  5. Retained packets.
    • Each wave's packets and SHA256SUMS are sealed under packets/ and listed in the run manifest.
    • CI checks every hash and recursively rejects any withheld key: stars, forks, watchers, *_at, latest, prerelease, pin_behind_upstream, newcomer.
    • Every adjudication's stripped_packet_sha256 must name a retained packet.
  6. Also:
    • a failed lane outcome with a reason (finding 7);
    • the kept adopted marker is documented and candidate-only note is stripped (finding 9);
    • findings 4 and 8 are left for a later PR.

Review

  • Opus review found one medium bypass: withheld keys at unchecked depths. It also found two lows: adjudications bound only by the row, and stray wave folders skipped. All three are fixed.
  • The re-review accepts. Its remaining low finding: the recursive check is a denylist, so it cannot prove that every possible disposition or recency signal is absent.
  • The verdict-review gate (github-automation-gap-closure) independently enforces the same names and hashes, and merges after this PR.

Checks (merged with current main)

  • python3 -m unittest: 2,960 tests OK, 339 skipped.
  • validate.py, evidence_manifest.py --check, landscape.py, build_verdicts.py --check, component_matrix.py --check and trading_gates.py --check: all pass.
  • Guarded gitleaks: no leaks.

🤖 Generated with Claude Code

…o their sealed wave

Independent review of catalog #122 (head 2f19193), findings 1, 2, 3, 5 and 6, plus 7 and 9.
Each has a regression test that fails on 97a56e2.

1. landscape.py recomputes a new-wave row's agreement from both sealed returns' winner_keys,
   resolved through the retained packet, and requires a recorded row's winners to equal the
   chosen lane's set (same_winner/codex_absent: claude; disagree: the adjudication's
   winner_lane); an unrecorded new-wave row carries no winners.
2. --allow-single-lane needs a dated docs/decisions/ record carrying the line
   "single-lane-authorization: <catalog>/<layer_id>"; the record's sha256 is stored on the row
   (lanes.single_lane_decision_sha256) and re-checked in CI.
3. A new wave's --write refuses once its run-manifest.json exists unless --append-rows names only
   absent rows, and never overwrites a sealed file with other bytes. Rows store
   lanes.run_manifest_sha256 and lanes.adjudication_sha256; landscape.py verifies both and rejects
   any file in a new wave's sealed folder that no row or the run manifest references.
5. Re-vendor layer-verdict-lane.js from agent-lab 6aab001 (sha256 2eb9c10d...), registered in
   lane-provenance.json; the return schema gains the refutation summary, claude_lane.py copies it,
   codex_lane.py drops it from the strict copy, and record_verdicts.py and landscape.py reject a
   Claude return whose final was refuted or unknown or lacks a lens vote.
6. --write retains the wave's packets and SHA256SUMS under <sealed_base>/packets/ (manifest
   retained_packets) and refuses withheld keys; CI re-checks hashes and withheld keys with the
   policy lane_packets.py now imports from landscape.py.
7. claude_lane.py and codex_lane.py write <lane>/failures.json; the run manifest records a
   "failed" outcome with its reason.
9. The kept "adopted" marker is documented; the candidate-only note is withheld.

Findings 4 and 8 are left for a later PR. GRANDFATHERED_RUN_IDS and every 20260922 artifact are
unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…d adjudications, stray wave folders

Independent review of fcee72b:

- MEDIUM: withheld_packet_keys checked only the positions withhold_popularity
  strips, so nested release/popularity keys, a top-level newcomers list, a
  non-null review_status and decisions[].selection passed CI and --write.
  scripts/landscape.py now walks the whole packet (is_withheld_packet_key,
  shared with lane_packets.withhold_popularity, which strips at any depth),
  rejects the disposition labels on candidate/component copies, and requires
  the always-listed policy labels in withheld[] (proof of --withhold-labels).
  Exemptions: requirement-gated archived/license and the packet's checked_at.
  Real 2026-09-23 packets leaked upstream.latest_flag (tag release/2025-11-28);
  it is now withheld. Default 2026-09-22 packets are byte-identical.
- LOW: the run manifest entry lists a sealed adjudication's sha256; landscape.py
  requires it to equal the row's lanes.adjudication_sha256 and re-hashes the file.
- LOW: a layer-verdicts-<id> folder with a non-alphanumeric id is rejected, not skipped.
- INFO (lane policy edge): kept deliberately and recorded in
  docs/decisions/2026-09-23-verdict-integrity.md with its overturn condition.

Regression tests (fail on fcee72b): test_lane_packets RecursiveWithheldKeyTests;
test_landscape test_retained_packets_are_checked_at_every_depth,
test_a_sealed_adjudication_is_bound_by_the_run_manifest,
test_a_layer_verdicts_folder_no_row_can_name_is_rejected; test_record_verdicts
ReviewOf122Tests test_a_packet_carrying_a_disposition_label_or_a_nested_withheld_key_is_refused,
test_the_run_manifest_lists_each_sealed_adjudication.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@seathatflowsinourveins
seathatflowsinourveins enabled auto-merge (squash) September 23, 2026 15:54
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-23T16:06:20.833310Z 238c754 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 238c754ef2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/sota-convergence/claude_lane.py Resolved
Comment thread tools/sota-convergence/codex_lane.py Resolved
Comment thread scripts/landscape.py Resolved
Comment thread scripts/landscape.py Resolved
… at e6953e5

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- claude_lane: delete a layer's previous return when its layer now fails
  or its packet is lost, before failures.json is written.
- codex_lane: delete the old output for a layer whose attempts all fail.
- record_verdicts: --append-rows fails before writing when a named row has
  no packet or lane return in the work dir.
- landscape: a retained SHA256SUMS must list every retained packet exactly
  once with its hash; malformed or duplicate lines are rejected.
- landscape: a sealed winner key must name an adopted candidate.

Each fix has a regression test that fails on the previous code. The new
codex_lane.py hash is appended to lane-provenance.json.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
record_verdicts now applies landscape.py's retained-SHA256SUMS rule before
any write: exactly the retained packets, each once, with its actual sha256.
A --write can no longer seal a wave CI would reject, since a sealed wave is
never re-recorded. The test PacketWriter keeps one line per packet, as
lane_packets.py does, and the three SHA256SUMS tests now expect the refusal
and still check landscape.py against a wave tampered after sealing.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@seathatflowsinourveins
seathatflowsinourveins merged commit 0f35e58 into main Sep 23, 2026
21 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/verdict-integrity-2-20260923 branch September 23, 2026 18:00
seathatflowsinourveins added a commit that referenced this pull request Sep 23, 2026
…r every changed verdict row (#135)

* Add the required verdict-review-gate check for changed layer-verdict rows

With one maintainer and zero required approvals, a PR changing a verdict row
could merge unreviewed. scripts/verdict_review_gate.py compares ledger rows at
the merge base with the head, keyed by (catalog, layer_id, run_id), and requires
for every changed non-grandfathered row: a registered wave document and run
manifest listing the row; registered, hash-matching sealed returns from two
distinct families; the agreement recomputed from the returns (review finding
1); winners resolved through the sealed packet, failing closed without it
(finding 6); a both-family, both-order, unrefuted adjudication for disagree
rows; an explicit docs/decisions single-lane-authorization line (finding 2);
receipt-derived platform_status-only changes; and frozen earlier wave entries
(finding 4). It then runs landscape.py and build_verdicts.py --check.

validate.yml gains the verdict-review-gate job (every pull_request, no path
filter; contents: read; harden-runner first; no persisted credentials; base
sha via env). The target main ruleset adds the context; the coordinator
applies it after merge.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Bind a changed row's winner fields and platform values to the sealed evidence

A changed non-grandfathered row now also needs each winner's evidence_class and
why_selected to be the chosen lane's and its pin to be the sealed packet's; a
pending row may carry no winners; every changed platform_status value must be
the one scripts/platform_status.py derives. Changes under the sealed verdict
artifacts or the landscape/sota-convergence catalogs run the repository
validators even when no row changed, so a deleted sealed lane file fails.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Close the verdict-review-gate review findings: decoy ledger, base-owned rules, published fields

- Read rows from build_verdicts.LEDGER_FILES and fail when the head's landscape manifest names
  other ledgers (a decoy could be validated in place of the published ledger).
- Run the base commit's gate against the PR checkout in CI, and fail a verdict change that also
  changes the gate's trust base in the same comparison.
- Tie a changed winner's repository and recipe_ref to the sealed packet candidate, and the
  published alternatives, verdict_overturn_when and overturn_protocol to what record_verdicts
  derives from the sealed returns; open_gaps changes are re-checked.
- Fail closed (exit 2) when a base or head ledger, manifest or wave registry exists but cannot be
  read or parsed.
- Add a negative-control test for each closed bypass and record the review in the decision.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Close the second verdict-review-gate review: row continuity, semantic compare, #124 bindings

F1: a base row cannot disappear, roll back to an older or grandfathered wave, or move to any
wave but the newest registered one (then it needs the full new-wave evidence); one row per layer.
F2: the PR-editable job definition is recorded as an accepted residual with its rejected
alternatives, mitigations and overturn; a test asserts the push-to-main run executes the gate.
F3: rows, wave documents and the registry compare on parsed values, so a pure reformat passes
and is not a verdict change for the trust-base rule; documented that a changed row value in a
non-grandfathered wave needs a new recorded wave.
F4: TRUST_PATHS adds scripts/validate.py, lane-provenance.json and the host-receipt schema; a
test derives the transitive imports and rule inputs from the modules.
F5: on pull_request the base is the merge commit's first parent, cross-checked against the
payload base.sha; push-to-main uses github.event.before and fails closed on all-zero.
F6: packets resolve through the run manifest (packet_sha256, retained_packets,
packets_sha256sums) and carry no withheld key at any depth; run_manifest_sha256,
adjudication_sha256 and single_lane_decision_sha256 are required and bound, with #124's names.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Record the second-review measurements and tolerate #124's adjudication message

The measured entry lists the new controls, the four real-checkout F1 mutations (each exit 1),
the real-ledger reformat (exit 0 with both validators) and the #124 overlay run. Two
adjudication tests accept the message #124's run_manifest_row_issue reports first; the
no-new-wave move message no longer prints None.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Derive a changed row's verdict_status from its sealed evidence; fail closed on git errors and duplicate keys

Third review of the verdict-review-gate: a recorded newest-wave row could be
relabelled pending_lanes or no_selection with its winners cleared and still
pass. The gate now derives the status record_verdicts.py writes for the sealed
evidence (agreement, adjudication, single-lane decision, remaining
alternatives) and rejects no_selection on a new-wave row. A failing git
diff/ls-files raises a read error (exit 2); every parse rejects duplicate JSON
keys; #124's constant values are pinned so the alignment test asserts before
#124 merges. The free-form newest-wave fields are documented.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Bind winner refs and pin to the sealed evidence; require base-present single-lane authorizations

Fourth review of the verdict-review-gate:
- G1: every winner field except platform_status must equal record_verdicts.build_winners output
  (evidence_refs and pin included, no extra key), and platform_status derives from receipts bound
  to the sealed pin and refs; a pin the packet lacks binds only when the base's row candidates
  carry it. The canonical repository index must derive the same alternatives at the base.
- G2: a single-lane decision record must exist byte-identical at the base.
- G3: the trust-path test derives head-side reads with an audit hook; adoption/manifest.json
  platform_profiles is bound as a rule input; base tree reads use ls-tree and fail closed.
- G4: the workflow asserts HEAD is the merge commit whose second parent is the PR head.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Record the fourth verdict-review-gate review, its fixes and real-checkout mutations

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Refresh the evidence manifest hashes of the gate, its docs, record and workflow

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Hold raised platform_status to base-trusted evidence; freeze registered SOTA manifests

Fifth review of the verdict-review-gate. A changed platform_status value may
rank no higher than platform_status() derives from only the cited evidence
files and host receipts already at the base with the same bytes and
registered there with that sha256, so a sealed citation made to resolve, or
a receipt added, in the same PR cannot raise a status. The SOTA manifest
every base wave registers, the newest included, keeps its pointer and value.
merge_base fails closed on git errors other than no common history. The
decision record adds the self-attested sealed-returns residual and rewords
the overturn line and workflow comment as a consistency check.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Record the fifth verdict-review-gate review measurements and real-checkout mutations

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Re-derive an unchanged platform_status whose pin or evidence changed; fall back to the base row's candidate pin

Sixth review: a candidates-only pin change with a same-PR receipt could keep a
declared status the sealed pin does not support. The no-packet-pin fallback now
comes from the base row, and an unchanged value is skipped only when pin,
evidence_refs and evidence_class are unchanged. Three negative controls, each
mutation-checked; two low findings recorded as residuals with proposed fixes.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Close the review of #135: retarget re-run, superseded-wave freeze, strict bootstrap and type-strict values

- H1: the pull_request trigger adds the edited type (a base-branch change) and the
  verdict-review-gate job fails closed on a pull_request whose base branch (PR_BASE_REF from
  github.base_ref, through env) is not main, so a retargeted PR cannot keep a stale green run.
- M1: when the head registers a wave newer than the base's newest, that newest wave's registry
  entry (sha256 included) and document are frozen byte for byte.
- L4: the head's own gate runs only when base...HEAD adds scripts/verdict_review_gate.py (A);
  any other base without the gate fails closed.
- L5: frozen values compare as canonical json.dumps(sort_keys=True), so 1, 1.0 and true differ.
- Hardening: changed paths are listed with git diff -z / git ls-files -z and split on NUL.
- L3: the decision record states that the pinning tests are head-defined too, and corrects the
  "visible workflow edit" claim with the retarget path and its fix.

Each new negative control is mutation-checked (reverting its fix fails its test).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Close round 2 of the #135 review: one new current wave per change, base on origin/main

A newly registered wave must be the only new run id, the head's newest and
newer than every base wave, so build_verdicts.py --check always regenerates
it; two new waves or a backdated wave fail. The PR gate step also requires
the merge commit's first parent to be a commit on origin/main, so a merge
commit built on a stacked branch fails closed. Fix the record's H1
cross-reference and document both checks.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Type-strict derivation checks and no unregistering the newest wave (#135 round 2 lows)

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Record the live retarget test of the #135 H1 fix and close the round-2 review lows

A throwaway PR (#143) showed that the edited run after a retarget checks out the merge commit
built on the old base; the round-2 origin/main ancestry check failed it closed, and close and
reopen rebuilt it on main, where the gate passed. The error message now names that recovery.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: seathatflowsinourveins <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Sep 23, 2026
… policy (#144)

* Make the gate apply the landscape validator's withheld-key policy instead of a drifted copy

The gate's own withheld_packet_keys predated #124 and lacked the top-level, copy-disposition and
withheld[] label checks; importing scripts/landscape.withheld_packet_keys from the base makes a key
the tooling owner adds (gap_receipts, blind-lanes round 2) apply here with no gate change.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Record the independent review of the withheld-policy import and correct the record's wording

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: seathatflowsinourveins <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant