Skip to content

PR-5: record the 32 layer verdicts of the grand catalog (Claude lane; Codex lane absent) - #71

Merged
seathatflowsinourveins merged 6 commits into
mainfrom
claude/grand-catalog-pr5
Sep 22, 2026
Merged

seathatflowsinourveins merged 6 commits into
mainfrom
claude/grand-catalog-pr5

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator

Scope

Fifth PR of the 2026-09-22 grand-catalog finalization. All 32 layers of the frozen taxonomy now carry a recorded verdict: 20 foundation-harness layers and 12 US-equities north-star layers. Each verdict names its winners, the alternatives, why the winners were chosen over them, the condition that would overturn it and the gaps it leaves open. The branch includes the refreshed SOTA manifest from #66 as its first commit, cherry-picked with original authorship. #66 can be closed with a reference once this merges.

How the verdicts were produced

  • Packets (tools/sota-convergence/lane_packets.py). One stripped packet per layer. The incumbent current_choice, decision and rationales are withheld. adopted stays, because the never-promote rule needs it. Candidate order is seeded identically for both lanes. Candidates join manifest components by repository slug.
  • Claude lane (agent-lab saved workflow layer-verdict-lane). Per layer: an Opus proposer, then two Opus refuters (an evidence lens and a challenger lens), then one revision when either refuter finds a problem. Recording run wf_bf9a4521-4a9, plus a one-layer re-run wf_f9391ce2-0a2 for us-equities/security-supply-chain, whose first return put four repositories in the single challenger field. Every completed child resolved to claude-opus-5-5.
  • Record (tools/sota-convergence/record_verdicts.py). Validates each return against the lane-return schema and contract rules. Seals the return under evidence/artifacts/layer-verdicts-20260922/claude/, and scripts/landscape.py verifies the sealed sha256. Pin, component id, recipe ref and platform status are derived in code. Agreement is computed from both lanes. v1 fields are never rewritten; a verdict's own overturn condition is the new schema-v2 field verdict_overturn_when.
  • Codex lane. Absent: the Codex account is at its usage limit until 2026-09-28, and no credits were bought. Every row records lanes.agreement = codex_absent and names that gap. tools/sota-convergence/codex_lane.py is ready and resumable for the re-run. Lane agreement never promotes a candidate.

Verdicts

Catalog Layer Winner(s) @ pin Evidence class Alternatives Open gaps
foundation native-clients claude-code @ 2.1.278; codex @ 0.155.1 native_proven 3 8
foundation instructions-skills candidate:typesafe-ai-skills @ unpinned; affaan-m/ECC @ dd6ee538aee0f548d4a6b520118f875431fd749e; candidate:openai-skills @ unpinned local_integration 7 11
foundation workers claude-code @ 2.1.278; worktrunk @ 0.79.0; codex @ 0.155.1 native_proven 9 11
foundation isolation worktrunk @ 0.79.0; sandbox-runtime @ 0.0.77 native_proven 5 12
foundation code-navigation serena @ 2.0.0.dev0 @ c6fbd1c5932df2494ffa0020af5a9fbe80b82143; jcodemunch-mcp @ 1.108.319; codebase-memory-mcp @ 0.11.0 native_proven 2 11
foundation document-retrieval qmd @ 2.8.3; poppler @ 26.09.0 native_proven 4 7
foundation semantic-rag socraticode @ 1.14.0; qdrant @ 1.19.1; vllm @ 0.25.0 native_proven 5 11
foundation durable-memory ai-memory @ 2.3.2 native_proven 9 10
foundation web-research tavily-cli @ 0.1.8; agent-browser @ 0.38.1; openresearch @ 0.2.7 native_proven 5 10
foundation token-efficiency rtk @ 0.49.0; headroom @ 0.37.0; ccusage @ 20.0.24 local_integration 4 10
foundation quality-evaluation promptfoo @ 0.123.1; playwright-test @ 1.63.0 local_integration 7 12
foundation ci-supply-chain zizmor @ 1.30.1; syft @ 1.52.0; candidate:actions-attest @ unpinned native_proven 5 11
foundation scheduling-supervision systemd @ 255.4-1ubuntu8.17; dagu @ 2.16.6 native_proven 4 8
foundation hosting-services fastapi @ 0.141.1; postgresql @ 18.6; nextjs @ 16.3.5 local_integration 8 9
foundation recovery-portability restic @ 0.19.1 native_proven 6 8
foundation observation-inference opentelemetry-collector-contrib @ 0.161.0; prometheus @ 3.14.0; grafana @ 13.2.2 native_proven 10 8
foundation agent-sdks codex @ 0.155.1 native_proven 4 11
foundation mcp-surfaces mcporter @ 0.13.13; mcp-inspector @ 2.7.0 native_proven 1 9
foundation secrets-credentials gitleaks @ 8.30.1 native_proven 1 8
foundation git-github-automation worktrunk @ 0.79.0; difftastic @ 0.71.0; candidate:cli-cli @ unpinned source_review 1 11
us-equities market-data-reference data-alpaca-py @ v0.44.0; data-duckdb @ v1.5.5; data-edgartools @ v5.58.0 native_proven 13 10
us-equities identity-provenance alpaca-py @ 0.44.0; data-duckdb @ v1.5.5; data-edgartools @ v5.58.0 native_proven 12 11
us-equities storage-compute data-duckdb @ v1.5.5; alpaca-py @ 0.44.0 native_proven 12 9
us-equities data-quality-orchestration candidate:anthropics-claude-code @ unpinned; dagu @ v2.16.6; loki @ v3.7.8 native_proven 18 12
us-equities research-factors-ml skfolio @ 1.2.9; nautilustrader @ 2.0.0rc5; lean @ 985ef30ad3ac774218c5ac516b4cb0aa2655730f native_proven 9 15
us-equities backtesting-engine nautilustrader @ 2.0.0rc5; lean @ 985ef30ad3ac774218c5ac516b4cb0aa2655730f local_integration 15 9
us-equities execution-broker nautilus-ibkr-adapter @ 2.0.0rc5; alpaca-py @ 0.44.0; lean @ 985ef30ad3ac774218c5ac516b4cb0aa2655730f source_review 11 14
us-equities portfolio-risk nautilustrader @ 2.0.0rc5; lean @ 985ef30ad3ac774218c5ac516b4cb0aa2655730f; alpaca-py @ 0.44.0 local_integration 9 9
us-equities evaluation-experiments codex-native-sdk @ CLI rust-v0.155.1; candidate:anthropics-claude-code @ unpinned; dagu @ v2.16.6 native_proven 17 11
us-equities agents-models-workers foundation-ai-memory @ v2.3.1; foundation-serena @ v1.7.0; foundation-qmd @ v2.8.3 native_proven 18 10
us-equities observability-hosting codex-native-sdk @ CLI rust-v0.155.1; dagu @ v2.16.6; opentelemetry-collector-contrib @ v0.161.0 native_proven 16 12
us-equities security-supply-chain codex-native-sdk @ CLI rust-v0.155.1; dagu @ v2.16.6; opentelemetry-collector-contrib @ v0.161.0 native_proven 15 11

Robustness

An earlier full run (wf_77e72cb4-fe3, claude-opus-5, packets from the pre-refresh manifest) is kept as a comparison. Against the recording run, the winner sets are identical in 20 of 32 layers and overlap in the other 12. None is disjoint.

Review

  • Unit reviews. Each tooling unit was built by an isolated-builder, reviewed by an Opus evidence-reviewer and given one fix round.
  • Independent diff review. An Opus review of the integrated tooling found 3 majors and 7 minors, all resolved with regression tests. The majors were: winner listed among alternatives, sealed-file integrity, and strict schema checking.
  • Fix verification. An independent verification confirmed the fixes and raised 3 more minors, also resolved.
  • Record rehearsal. Rehearsing the record step on a full run found three defects before any row was recorded, all fixed. v1 overturn_when is mirrored by the quality comparison. Lane citations carried line anchors. A citation of manifests/evidence.json made the explorer hash itself.

Evidence

Local integration only: validate.py, landscape.py, record_verdicts.py --check, build_verdicts.py --check, validate_convergence.py --all-recorded, build_ecosystem.py --check and trading_gates.py --check all pass, and python3 -m unittest ran 1,587 tests OK (204 skipped for pinned runtimes absent from system Python). A bounded gitleaks scan over origin/main..HEAD found no leaks; it used --max-target-megabytes 2, so the explorer HTML is excluded.

Limits carried in the rows and records

  • Second lane. The Codex lane has not run.
  • Evidence classes. Most winners are native_proven or local_integration within their recorded scope. git-github-automation and execution-broker rest on source review.
  • Packet withholding. adopted and the foundation decision selection values still reveal the incumbent default. This is documented, with the comparison that would test it.
  • Trading candidate sets. The 12 trading rows share four group-wide candidate sets. The security row's challenger (syft) and its open gaps (gitleaks, grype, sandbox-runtime) show that layer-specific curation of the trading ledger is the next ledger step.
  • Cross-catalog pins. The trading agents row cites the trading cards' own older pins (ai-memory v2.3.1, serena v1.7.0), while the foundation catalog pins newer versions.

🤖 Generated with Claude Code

seathatflowsinourveins and others added 6 commits September 22, 2026 16:25
…ct lane run

The saved Claude workflow sota-convergence (run wf_2388074a-e82: foundation,
trading and beyond review lanes, two Opus refuters per proposal, one critic;
110 Opus 5/high children, 908 requests, 1,064,906 output tokens) reviewed the
20-layer foundation taxonomy and the 12-layer trading taxonomy against GitHub
metadata for 469 repositories fetched at 14:57 UTC. build_manifest.py now
emits 20 foundation and 12 trading layers, 109 confirmed components, 17 unique
pins behind upstream (information only), 16 candidate dispositions (none
promoted), per-component why_selected / comparison_that_would_overturn on 140
of 242 rows, and the critic's gaps. No pin or upstream field changed for any of
the 242 rows relative to the first refresh, and no pin string changed for the
225 components already listed; 17 rows were added for the four new foundation
layers. The Codex lane could not run (account usage limit until 2026-09-28);
the ledger rows stay pending for the PR-5 owner to record.

Tooling (tools/sota-convergence/build_manifest.py), each with regression tests:
- review-lane grouping ids outside the taxonomy (the beyond lane's
  awesome-list-convergence, star-audit-targeted-candidates,
  unmaintained-reference-material) are carried verbatim into a new
  lane_groupings section instead of being appended as trading rows, which
  broke scripts/landscape.py coverage;
- every lane's entry for a (layer, repository) is kept: the row takes one by a
  documented precedence (adversarially verified beats unverified, the owning
  lane beats beyond, then lexical) and the rest are published under
  other_lane_reviews; a lane entry naming another card is never folded in;
- a refuted demotion/unmaintained proposal resolves to the component's own
  baseline class (confirmed_default / confirmed_conditional / confirmed_selected)
  instead of an unconditional confirmed_default;
- an adversarial verdict is shared across layers for the same lane, repository
  and proposal kind, with a provenance evidence line;
- lane entries key by catalog_id when set, so two cards for one repository in
  one layer (alpaca-py execution and data) keep distinct rationales;
- sanitization keeps the file basename and JSON pointer of redacted citations
  (<work-dir>/file#pointer, <host-path>/file#pointer), never republishes a home
  directory name, scrubs session-scoped /tmp paths and bare session UUIDs, and
  assert_no_leak refuses the validate.py session-identifier pattern;
- a --citation-review overlay stamps an independent review's findings onto the
  rows they name (citation_review) and keeps the rest under
  citation_review.general with counts; generated text is never hand-edited.
66 new tests (124 in tests.test_sota_convergence).

Independent review: evidence/artifacts/sota-convergence-review-20260922/
manifest-citation-review.json records the four-reviewer pass over the first
refresh (36 findings: 5 generator defects fixed here, 31 citation-quality
findings now attached to 27 rows or listed as general).

Regenerated: layer-verdicts-20260922.json (join only; rows still pending),
docs/ecosystem/index.html, manifests/evidence.json hashes (+1 listed file).

Checks (after rebase on d093a4c): tests.test_sota_convergence 124 OK;
python3 -m unittest 1438 OK (204 skipped, system python); validate.py 69
components / 1746 files / 139 receipts; landscape.py, validate_catalogs.py,
validate_convergence.py --all-recorded, validate_foundation.py,
catalog_decisions.py --check, build_verdicts.py --check, build_ecosystem.py
--check all passed; git diff --check clean.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
(cherry picked from commit 0a4972a)
…layer narrative)

tools/sota-convergence/lane_packets.py builds stripped per-layer packets (incumbent rationale withheld, adopted kept,
seeded candidate order, catalog-wide manifest join by repository slug) with SHA256SUMS; lane-return.schema.json and
lane-prompt.md are the shared lane contract; record_verdicts.py validates each lane return against the layer-verdict
schema v2 rules, seals it under evidence/artifacts/layer-verdicts-20260922/<lane>/<run_id>.json, derives
pin/component_id/recipe_ref/platform_status in code, computes agreement (same_winner/disagree/codex_absent/pending)
and writes the ledger rows; build_verdicts.py renders the per-layer narrative in the handbook's generated block;
codex_lane.py runs the Codex lane per packet (resumable, usage captured). 100 tests cover the tools on synthetic
fixtures. Handbook block, evidence manifest and explorer regenerated.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
… generated indexes

Found by rehearsing the record step on a full 32-layer lane run: the dated quality comparison mirrors each
layer's v1 overturn_when, so a recorded verdict now carries its own verdict_overturn_when (schema v2 field;
the recorded-verdict marker rule moves to it) and no v1 field is rewritten. Lane citations carry line
anchors and commentary; rows keep the bare canonical repository path and the sealed lane return keeps the
full citation. Citations that resolve to no repository file or name a generated index
(manifests/evidence.json, docs/ecosystem/index.html, which made the explorer embed its own hash) are
counted in one open gap. Rehearsal: record --write/--check, build_verdicts, landscape, validate.py,
convergence and the explorer fixed point all pass on 32 recorded rows; python3 -m unittest Ran 1513 OK.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…strict lane schema, winner/alternative exclusion)

Findings from the Opus evidence review of fc478d1..53545eb, each with a regression test:
- a recorded row never lists its winner among its alternatives (adjudicated disagreement case); a row left with
  no indexed alternative stays pending_lanes with the gap; scripts/landscape.py rejects a winner repository
  among alternatives;
- record --check compares every recomputed sealed file with the file on disk, and scripts/landscape.py verifies
  lanes.<lane>.sealed_sha256 against the sealed file's bytes;
- lane returns are checked against lane-return.schema.json properties at every level, an https challenger
  repository, and a why_selected that names one of its cited evidence paths;
- the packet file must hash to its SHA256SUMS entry (a missing entry gets its own message); a leak refusal while
  sealing rejects that lane instead of aborting the run; a malformed adjudication is reported, not ignored;
- sources_read absolute paths are rewritten to repository-relative paths before sealing;
- codex_lane.py decodes the partial output of a timed-out attempt instead of discarding it (mutation-checked);
- a no_selection row renders as "no selection" in the handbook narrative.
The packet withholding limit (adopted flag and foundation decision `selection` values reveal the incumbent
default) is documented with the comparison that would test it. python3 -m unittest Ran 1582 OK (skipped=204).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
… source paths, adjudication leak)

The independent verification of cf68d27 confirmed seven findings resolved and three partly resolved, and raised
three new minors, all fixed here with tests:
- why_selected must cite an evidence path, as the lane contract says: one of its own cited paths or any
  repository-style path in the text (a lane may leave winner_evidence_refs empty rather than guess);
- sources_read is rewritten to repository-relative form only under a checkout passed as --lane-repo-root
  (repeatable; the same values reproduce the sealed text under --check), so a same-named file from another
  checkout is never attributed to this repository; every other absolute path is redacted by sanitize_value;
- a leak marker in an adjudication file is reported and leaves the row pending instead of aborting the run;
- tests now cover a recorded row reduced to no indexed alternative, landscape.py rejecting a winner listed among
  its alternatives, and the no_selection narrative label.
python3 -m unittest Ran 1587 OK (skipped=204).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…bsent, gap named)

Claude lane run wf_bf9a4521-4a9 (32 packets from the refreshed manifest; per layer an Opus proposer, two Opus
refuters with evidence and challenger lenses, one revision) plus the one-layer re-run wf_f9391ce2-0a2 for
us-equities/security-supply-chain after its first return failed the single-repository challenger rule. Every
completed child resolved to claude-opus-5-5; the coordinator set model.name from that measured provenance.
record_verdicts.py --write with --lane-repo-root: 32 recorded, 0 rejections; each accepted return sealed under
evidence/artifacts/layer-verdicts-20260922/claude/ with its sha256 in lanes.claude.sealed_sha256;
lanes.agreement is codex_absent on every row (Codex account usage limit until 2026-09-28) and each row names
that gap. v1 fields unchanged; verdict conditions live in verdict_overturn_when. Handbook block, join, evidence
manifest (32 sealed files hash-listed) and explorer regenerated. validate.py, landscape, record --check,
build_verdicts --check, convergence --all-recorded and explorer --check pass; python3 -m unittest Ran 1587 OK.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant