Repository navigation
PR-5: record the 32 layer verdicts of the grand catalog (Claude lane; Codex lane absent) - #71
Merged
Merged
Conversation
…ct lane run The saved Claude workflow sota-convergence (run wf_2388074a-e82: foundation, trading and beyond review lanes, two Opus refuters per proposal, one critic; 110 Opus 5/high children, 908 requests, 1,064,906 output tokens) reviewed the 20-layer foundation taxonomy and the 12-layer trading taxonomy against GitHub metadata for 469 repositories fetched at 14:57 UTC. build_manifest.py now emits 20 foundation and 12 trading layers, 109 confirmed components, 17 unique pins behind upstream (information only), 16 candidate dispositions (none promoted), per-component why_selected / comparison_that_would_overturn on 140 of 242 rows, and the critic's gaps. No pin or upstream field changed for any of the 242 rows relative to the first refresh, and no pin string changed for the 225 components already listed; 17 rows were added for the four new foundation layers. The Codex lane could not run (account usage limit until 2026-09-28); the ledger rows stay pending for the PR-5 owner to record. Tooling (tools/sota-convergence/build_manifest.py), each with regression tests: - review-lane grouping ids outside the taxonomy (the beyond lane's awesome-list-convergence, star-audit-targeted-candidates, unmaintained-reference-material) are carried verbatim into a new lane_groupings section instead of being appended as trading rows, which broke scripts/landscape.py coverage; - every lane's entry for a (layer, repository) is kept: the row takes one by a documented precedence (adversarially verified beats unverified, the owning lane beats beyond, then lexical) and the rest are published under other_lane_reviews; a lane entry naming another card is never folded in; - a refuted demotion/unmaintained proposal resolves to the component's own baseline class (confirmed_default / confirmed_conditional / confirmed_selected) instead of an unconditional confirmed_default; - an adversarial verdict is shared across layers for the same lane, repository and proposal kind, with a provenance evidence line; - lane entries key by catalog_id when set, so two cards for one repository in one layer (alpaca-py execution and data) keep distinct rationales; - sanitization keeps the file basename and JSON pointer of redacted citations (<work-dir>/file#pointer, <host-path>/file#pointer), never republishes a home directory name, scrubs session-scoped /tmp paths and bare session UUIDs, and assert_no_leak refuses the validate.py session-identifier pattern; - a --citation-review overlay stamps an independent review's findings onto the rows they name (citation_review) and keeps the rest under citation_review.general with counts; generated text is never hand-edited. 66 new tests (124 in tests.test_sota_convergence). Independent review: evidence/artifacts/sota-convergence-review-20260922/ manifest-citation-review.json records the four-reviewer pass over the first refresh (36 findings: 5 generator defects fixed here, 31 citation-quality findings now attached to 27 rows or listed as general). Regenerated: layer-verdicts-20260922.json (join only; rows still pending), docs/ecosystem/index.html, manifests/evidence.json hashes (+1 listed file). Checks (after rebase on d093a4c): tests.test_sota_convergence 124 OK; python3 -m unittest 1438 OK (204 skipped, system python); validate.py 69 components / 1746 files / 139 receipts; landscape.py, validate_catalogs.py, validate_convergence.py --all-recorded, validate_foundation.py, catalog_decisions.py --check, build_verdicts.py --check, build_ecosystem.py --check all passed; git diff --check clean. Co-Authored-By: Claude Fable 5.1 <[email protected]> (cherry picked from commit 0a4972a)
…layer narrative) tools/sota-convergence/lane_packets.py builds stripped per-layer packets (incumbent rationale withheld, adopted kept, seeded candidate order, catalog-wide manifest join by repository slug) with SHA256SUMS; lane-return.schema.json and lane-prompt.md are the shared lane contract; record_verdicts.py validates each lane return against the layer-verdict schema v2 rules, seals it under evidence/artifacts/layer-verdicts-20260922/<lane>/<run_id>.json, derives pin/component_id/recipe_ref/platform_status in code, computes agreement (same_winner/disagree/codex_absent/pending) and writes the ledger rows; build_verdicts.py renders the per-layer narrative in the handbook's generated block; codex_lane.py runs the Codex lane per packet (resumable, usage captured). 100 tests cover the tools on synthetic fixtures. Handbook block, evidence manifest and explorer regenerated. Co-Authored-By: Claude Fable 5.1 <[email protected]>
… generated indexes Found by rehearsing the record step on a full 32-layer lane run: the dated quality comparison mirrors each layer's v1 overturn_when, so a recorded verdict now carries its own verdict_overturn_when (schema v2 field; the recorded-verdict marker rule moves to it) and no v1 field is rewritten. Lane citations carry line anchors and commentary; rows keep the bare canonical repository path and the sealed lane return keeps the full citation. Citations that resolve to no repository file or name a generated index (manifests/evidence.json, docs/ecosystem/index.html, which made the explorer embed its own hash) are counted in one open gap. Rehearsal: record --write/--check, build_verdicts, landscape, validate.py, convergence and the explorer fixed point all pass on 32 recorded rows; python3 -m unittest Ran 1513 OK. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…strict lane schema, winner/alternative exclusion) Findings from the Opus evidence review of fc478d1..53545eb, each with a regression test: - a recorded row never lists its winner among its alternatives (adjudicated disagreement case); a row left with no indexed alternative stays pending_lanes with the gap; scripts/landscape.py rejects a winner repository among alternatives; - record --check compares every recomputed sealed file with the file on disk, and scripts/landscape.py verifies lanes.<lane>.sealed_sha256 against the sealed file's bytes; - lane returns are checked against lane-return.schema.json properties at every level, an https challenger repository, and a why_selected that names one of its cited evidence paths; - the packet file must hash to its SHA256SUMS entry (a missing entry gets its own message); a leak refusal while sealing rejects that lane instead of aborting the run; a malformed adjudication is reported, not ignored; - sources_read absolute paths are rewritten to repository-relative paths before sealing; - codex_lane.py decodes the partial output of a timed-out attempt instead of discarding it (mutation-checked); - a no_selection row renders as "no selection" in the handbook narrative. The packet withholding limit (adopted flag and foundation decision `selection` values reveal the incumbent default) is documented with the comparison that would test it. python3 -m unittest Ran 1582 OK (skipped=204). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
… source paths, adjudication leak) The independent verification of cf68d27 confirmed seven findings resolved and three partly resolved, and raised three new minors, all fixed here with tests: - why_selected must cite an evidence path, as the lane contract says: one of its own cited paths or any repository-style path in the text (a lane may leave winner_evidence_refs empty rather than guess); - sources_read is rewritten to repository-relative form only under a checkout passed as --lane-repo-root (repeatable; the same values reproduce the sealed text under --check), so a same-named file from another checkout is never attributed to this repository; every other absolute path is redacted by sanitize_value; - a leak marker in an adjudication file is reported and leaves the row pending instead of aborting the run; - tests now cover a recorded row reduced to no indexed alternative, landscape.py rejecting a winner listed among its alternatives, and the no_selection narrative label. python3 -m unittest Ran 1587 OK (skipped=204). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…bsent, gap named) Claude lane run wf_bf9a4521-4a9 (32 packets from the refreshed manifest; per layer an Opus proposer, two Opus refuters with evidence and challenger lenses, one revision) plus the one-layer re-run wf_f9391ce2-0a2 for us-equities/security-supply-chain after its first return failed the single-repository challenger rule. Every completed child resolved to claude-opus-5-5; the coordinator set model.name from that measured provenance. record_verdicts.py --write with --lane-repo-root: 32 recorded, 0 rejections; each accepted return sealed under evidence/artifacts/layer-verdicts-20260922/claude/ with its sha256 in lanes.claude.sealed_sha256; lanes.agreement is codex_absent on every row (Codex account usage limit until 2026-09-28) and each row names that gap. v1 fields unchanged; verdict conditions live in verdict_overturn_when. Handbook block, join, evidence manifest (32 sealed files hash-listed) and explorer regenerated. validate.py, landscape, record --check, build_verdicts --check, convergence --all-recorded and explorer --check pass; python3 -m unittest Ran 1587 OK. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This was referenced Sep 22, 2026
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Fifth PR of the 2026-09-22 grand-catalog finalization. All 32 layers of the frozen taxonomy now carry a recorded verdict: 20 foundation-harness layers and 12 US-equities north-star layers. Each verdict names its winners, the alternatives, why the winners were chosen over them, the condition that would overturn it and the gaps it leaves open. The branch includes the refreshed SOTA manifest from #66 as its first commit, cherry-picked with original authorship. #66 can be closed with a reference once this merges.
How the verdicts were produced
tools/sota-convergence/lane_packets.py). One stripped packet per layer. The incumbentcurrent_choice,decisionand rationales are withheld.adoptedstays, because the never-promote rule needs it. Candidate order is seeded identically for both lanes. Candidates join manifest components by repository slug.layer-verdict-lane). Per layer: an Opus proposer, then two Opus refuters (an evidence lens and a challenger lens), then one revision when either refuter finds a problem. Recording runwf_bf9a4521-4a9, plus a one-layer re-runwf_f9391ce2-0a2forus-equities/security-supply-chain, whose first return put four repositories in the single challenger field. Every completed child resolved toclaude-opus-5-5.tools/sota-convergence/record_verdicts.py). Validates each return against the lane-return schema and contract rules. Seals the return underevidence/artifacts/layer-verdicts-20260922/claude/, andscripts/landscape.pyverifies the sealed sha256. Pin, component id, recipe ref and platform status are derived in code. Agreement is computed from both lanes. v1 fields are never rewritten; a verdict's own overturn condition is the new schema-v2 fieldverdict_overturn_when.lanes.agreement = codex_absentand names that gap.tools/sota-convergence/codex_lane.pyis ready and resumable for the re-run. Lane agreement never promotes a candidate.Verdicts
Robustness
An earlier full run (
wf_77e72cb4-fe3,claude-opus-5, packets from the pre-refresh manifest) is kept as a comparison. Against the recording run, the winner sets are identical in 20 of 32 layers and overlap in the other 12. None is disjoint.Review
isolated-builder, reviewed by an Opusevidence-reviewerand given one fix round.overturn_whenis mirrored by the quality comparison. Lane citations carried line anchors. A citation ofmanifests/evidence.jsonmade the explorer hash itself.Evidence
Local integration only:
validate.py,landscape.py,record_verdicts.py --check,build_verdicts.py --check,validate_convergence.py --all-recorded,build_ecosystem.py --checkandtrading_gates.py --checkall pass, andpython3 -m unittestran 1,587 tests OK (204 skipped for pinned runtimes absent from system Python). A bounded gitleaks scan overorigin/main..HEADfound no leaks; it used--max-target-megabytes 2, so the explorer HTML is excluded.Limits carried in the rows and records
native_provenorlocal_integrationwithin their recorded scope.git-github-automationandexecution-brokerrest on source review.adoptedand the foundation decisionselectionvalues still reveal the incumbent default. This is documented, with the comparison that would test it.🤖 Generated with Claude Code