Repository navigation
Refresh the SOTA convergence manifest from the 2026-09-22 layer-verdict lane run - #66
Closed
seathatflowsinourveins wants to merge 1 commit into
Closed
seathatflowsinourveins wants to merge 1 commit into
seathatflowsinourveins wants to merge 1 commit into
Conversation
…ct lane run The saved Claude workflow sota-convergence (run wf_2388074a-e82: foundation, trading and beyond review lanes, two Opus refuters per proposal, one critic; 110 Opus 5/high children, 908 requests, 1,064,906 output tokens) reviewed the 20-layer foundation taxonomy and the 12-layer trading taxonomy against GitHub metadata for 469 repositories fetched at 14:57 UTC. build_manifest.py now emits 20 foundation and 12 trading layers, 109 confirmed components, 17 unique pins behind upstream (information only), 16 candidate dispositions (none promoted), per-component why_selected / comparison_that_would_overturn on 140 of 242 rows, and the critic's gaps. No pin or upstream field changed for any of the 242 rows relative to the first refresh, and no pin string changed for the 225 components already listed; 17 rows were added for the four new foundation layers. The Codex lane could not run (account usage limit until 2026-09-28); the ledger rows stay pending for the PR-5 owner to record. Tooling (tools/sota-convergence/build_manifest.py), each with regression tests: - review-lane grouping ids outside the taxonomy (the beyond lane's awesome-list-convergence, star-audit-targeted-candidates, unmaintained-reference-material) are carried verbatim into a new lane_groupings section instead of being appended as trading rows, which broke scripts/landscape.py coverage; - every lane's entry for a (layer, repository) is kept: the row takes one by a documented precedence (adversarially verified beats unverified, the owning lane beats beyond, then lexical) and the rest are published under other_lane_reviews; a lane entry naming another card is never folded in; - a refuted demotion/unmaintained proposal resolves to the component's own baseline class (confirmed_default / confirmed_conditional / confirmed_selected) instead of an unconditional confirmed_default; - an adversarial verdict is shared across layers for the same lane, repository and proposal kind, with a provenance evidence line; - lane entries key by catalog_id when set, so two cards for one repository in one layer (alpaca-py execution and data) keep distinct rationales; - sanitization keeps the file basename and JSON pointer of redacted citations (<work-dir>/file#pointer, <host-path>/file#pointer), never republishes a home directory name, scrubs session-scoped /tmp paths and bare session UUIDs, and assert_no_leak refuses the validate.py session-identifier pattern; - a --citation-review overlay stamps an independent review's findings onto the rows they name (citation_review) and keeps the rest under citation_review.general with counts; generated text is never hand-edited. 66 new tests (124 in tests.test_sota_convergence). Independent review: evidence/artifacts/sota-convergence-review-20260922/ manifest-citation-review.json records the four-reviewer pass over the first refresh (36 findings: 5 generator defects fixed here, 31 citation-quality findings now attached to 27 rows or listed as general). Regenerated: layer-verdicts-20260922.json (join only; rows still pending), docs/ecosystem/index.html, manifests/evidence.json hashes (+1 listed file). Checks (after rebase on d093a4c): tests.test_sota_convergence 124 OK; python3 -m unittest 1438 OK (204 skipped, system python); validate.py 69 components / 1746 files / 139 receipts; landscape.py, validate_catalogs.py, validate_convergence.py --all-recorded, validate_foundation.py, catalog_decisions.py --check, build_verdicts.py --check, build_ecosystem.py --check all passed; git diff --check clean. Co-Authored-By: Claude Fable 5.1 <[email protected]>
seathatflowsinourveins
force-pushed
the
claude/gc-pr5-manifest
branch
from
September 22, 2026 17:24
e3c9470 to
0a4972a
Compare
seathatflowsinourveins
added a commit
that referenced
this pull request
Sep 22, 2026
…nd Codex (#68) Full-coverage sweep of the SDK, harness, runtime and worker layers beyond Claude Code and Codex, with preregistered selection. Adds `catalogs/sota-convergence/sdk-runtime-coverage-20260922.{json,md}`, six per-category evidence artifacts under `evidence/artifacts/sdk-runtime-coverage-20260922/` (every shortlisted row's evidence packet and both refutation votes) and `docs/tasks/2026-09-22-sdk-runtime-coverage.md`; registers the nine files in `manifests/evidence.json`. No landscape verdict row, handbook section or SOTA manifest entry is edited (PR-5 and #66 own those); the ledger is the input for the owners of `agent-sdks`, `workers`, `isolation` and `mcp-surfaces`. | Stage | Agents | Result | | --- | --- | --- | | Discover (six categories x web and GitHub-search angles, critic, gap round) | 19 | 356 unique repositories, 297 not yet in the catalog | | Enrich and gate (`gh api`, no model) | 0 | 290 pass, 64 license flagged, 2 excluded (archived, stale) | | Shortlist (three Opus judges per category, majority) | 18 | 38 of 372 category-rows; 334 rejections retained with reasons | | Evidence packet + accuracy and fit refutation | 38 + 76 | one packet per repository from primary sources | | Selection (three Opus judges per category, criteria C1-C6) + critic | 19 | majority disposition; conservative on no majority | | Re-collection of three placeholder packets | 18 | rebuilt, refuted and re-judged | Dispositions after six coordinator resolutions (all listed in the ledger): **integrate_now 1** (`modelcontextprotocol/inspector`), **targeted_candidate 6** (`awslabs/cli-agent-orchestrator`, `swe-agent/mini-swe-agent`, `anthropics/claude-code-action`, `untrivial-ai/agent-orchestrator`, `trailofbits/coop`, `snyk/agent-scan`), keep_but_compare 18, not_adopted 7, refuted 6. Agreement never promoted anything; every row keeps its named comparison or acceptance command. Dispositions are per (category, repository); the one cross-category override (`dagger/container-use`) follows a named canonical-record precedence rule. Evidence class: source review, plus unretained spot probes some judges ran (named in the ledger), plus one retained local_integration execution for the integrate_now row: agent-lab `tools/mcp-conformance/check.sh` (serena 22, ai-memory 23, socraticode 26 tools, `--strict` clean; `--self-test` observes the failing path with a fixture server whose unportable schema exits 6). No macOS execution for any row; incumbents' capabilities were not re-exercised. Gate: `validate.py` passed (69 components, 1,763 hashed files, 140 receipts); `build_ecosystem.py --check`, `validate_catalogs.py`, `validate_foundation.py`, `validate_convergence.py --all-recorded` passed; `python3 -m unittest` 1,428 OK (204 skipped); guarded Gitleaks over both commits: no leaks (2 MB target cap, which now excludes nothing, and decode depth 0). Session identifiers, task handles, local paths, the local user name and groups, credential file locations and token scopes were redacted from worker output. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Collaborator
Author
|
Carried into main by PR-5 (#71): commit 86bc4be applies this PR's change with its original authorship. Every source file this PR changed (manifest-20260922.json, build_manifest.py, tests/test_sota_convergence.py, tools/sota-convergence/README.md, layer-verdicts-20260922.json, the citation review) is byte-identical at 86bc4be; only regenerated files (docs/ecosystem/index.html, manifests/evidence.json) differ because they were rebuilt on the newer base. Closed without a separate merge. |
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Refreshes
catalogs/sota-convergence/manifest-20260922.jsonfrom the saved Claude workflowsota-convergence(runwf_2388074a-e82: foundation, trading and beyond review lanes, two Opus refuters per proposal, one completeness critic; 110 Opus 5/high children, 908 requests, 1,064,906 output tokens, 46.5 min) over GitHub metadata for 469 repositories fetched 2026-09-22 14:57 UTC.why_selected/comparison_that_would_overturnlane_groupingsThe Codex lane could not run: the ChatGPT account's usage limit is exhausted until 2026-09-28 (thread
01a0c9a3-102b…, no purchase). Cross-family review of this PR is therefore not established; the ledger rows staypending_lanesfor the PR-5 owner to record withcodex_absent. The PR-5 owner cherry-picks this commit into PR-5; this PR stays open unmerged until PR-5 merges and then closes with the reference.Tooling (
tools/sota-convergence/build_manifest.py), each with regression testslane_groupingsinstead of fake trading rows (which brokescripts/landscape.pycoverage).other_lane_reviews; an entry naming another card is never folded in.confirmed_default/confirmed_conditional/confirmed_selected), never an unconditionalconfirmed_default.catalog_idwhen set, so two cards for one repository in one layer (alpaca-py execution and data) keep distinct rationales.<work-dir>/file#pointer,<host-path>/file#pointer), never republishes a home-directory name, scrubs session-scoped/tmppaths and bare session UUIDs;assert_no_leakrefuses thescripts/validate.pysession-identifier pattern.--citation-reviewoverlay stamps an independent review's findings onto the rows they name (citation_review) and keeps the rest undercitation_review.general, with counts; generated text is never hand-edited.tests/test_sota_convergence.py).Independent review
evidence/artifacts/sota-convergence-review-20260922/manifest-citation-review.json(hash-listed) records the four-reviewer Opus pass over the first refresh: one reviewer on the tooling diff, three opening every cited item behindwhy_selected/comparison_that_would_overturnon 157 component rows. Of its 36 findings, 5 were generator defects (fixed above, then re-reviewed in a build → review → fix → recheck round,wf_299a1543-f6e, with the two recheck residuals resolved by the coordinator and covered by tests) and 31 are citation-quality findings on lane text (line-number drift, execution claims without receipt paths, unfalsifiable or prescriptive overturn statements, uncited star-audit claims) now attached to their rows. Pins and upstream blocks of all reviewed rows matchedmanifests/stack.json, the trading cards andgithub-freshness.json.Evidence classes
Metadata and source review by the lanes; adversarial verification inside the workflow; local integration checks below. No install, run, benchmark or provider trial. The beyond lane's 9 proposals beyond the per-lane cap of 20 are recorded as unverified, not as refuted.
Checks (worktree, after rebase on
d093a4c)Private inputs, sealed
lanes.json(sha256315946d7…) and per-child usage are under the private state directory named indocs/tasks/2026-09-22-sota-convergence.md(agent-lab).🤖 Generated with Claude Code