Skip to content

Refresh the SOTA convergence manifest from the 2026-09-22 layer-verdict lane run - #66

Closed
seathatflowsinourveins wants to merge 1 commit into
mainfrom
claude/gc-pr5-manifest
Closed

seathatflowsinourveins wants to merge 1 commit into
mainfrom
claude/gc-pr5-manifest

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

What

Refreshes catalogs/sota-convergence/manifest-20260922.json from the saved Claude workflow sota-convergence (run wf_2388074a-e82: foundation, trading and beyond review lanes, two Opus refuters per proposal, one completeness critic; 110 Opus 5/high children, 908 requests, 1,064,906 output tokens, 46.5 min) over GitHub metadata for 469 repositories fetched 2026-09-22 14:57 UTC.

Item Value
Foundation / trading layers 20 / 12 (taxonomy frozen by PR-2)
Components confirmed 109
Unique pins behind upstream 17 (information only; never-upgrade rule kept)
Candidate dispositions 16 in taxonomy rows (1 refuted targeted, 8 refuted keep-but-compare, 3 refuted not-adopted, 4 not-adopted confirmed) + 2 in lane groupings; nothing promoted
why_selected / comparison_that_would_overturn 140 of 242 component rows
Pin strings changed 0 of the 225 previously listed components; 17 rows added for the four new foundation layers
Lane groupings 3 (the beyond lane's non-taxonomy groupings), carried verbatim in lane_groupings
Citation review overlay 36 findings in the artifact: 28 attached to 27 rows, 3 general, 5 tooling findings out of scope

The Codex lane could not run: the ChatGPT account's usage limit is exhausted until 2026-09-28 (thread 01a0c9a3-102b…, no purchase). Cross-family review of this PR is therefore not established; the ledger rows stay pending_lanes for the PR-5 owner to record with codex_absent. The PR-5 owner cherry-picks this commit into PR-5; this PR stays open unmerged until PR-5 merges and then closes with the reference.

Tooling (tools/sota-convergence/build_manifest.py), each with regression tests

  • Review-lane grouping ids outside the taxonomy go to lane_groupings instead of fake trading rows (which broke scripts/landscape.py coverage).
  • Every lane's entry for a (layer, repository) is kept: the row takes one by a documented precedence (adversarially verified beats unverified, the owning lane beats beyond, then lexical), the rest are published under other_lane_reviews; an entry naming another card is never folded in.
  • A refuted demotion/unmaintained proposal resolves to the component's own baseline class (confirmed_default / confirmed_conditional / confirmed_selected), never an unconditional confirmed_default.
  • Adversarial verdicts are shared across layers for the same lane, repository and proposal kind, with a provenance evidence line.
  • Lane entries key by catalog_id when set, so two cards for one repository in one layer (alpaca-py execution and data) keep distinct rationales.
  • Sanitization keeps the file basename and JSON pointer of redacted citations (<work-dir>/file#pointer, <host-path>/file#pointer), never republishes a home-directory name, scrubs session-scoped /tmp paths and bare session UUIDs; assert_no_leak refuses the scripts/validate.py session-identifier pattern.
  • --citation-review overlay stamps an independent review's findings onto the rows they name (citation_review) and keeps the rest under citation_review.general, with counts; generated text is never hand-edited.
  • 66 new tests (124 in tests/test_sota_convergence.py).

Independent review

evidence/artifacts/sota-convergence-review-20260922/manifest-citation-review.json (hash-listed) records the four-reviewer Opus pass over the first refresh: one reviewer on the tooling diff, three opening every cited item behind why_selected / comparison_that_would_overturn on 157 component rows. Of its 36 findings, 5 were generator defects (fixed above, then re-reviewed in a build → review → fix → recheck round, wf_299a1543-f6e, with the two recheck residuals resolved by the coordinator and covered by tests) and 31 are citation-quality findings on lane text (line-number drift, execution claims without receipt paths, unfalsifiable or prescriptive overturn statements, uncited star-audit claims) now attached to their rows. Pins and upstream blocks of all reviewed rows matched manifests/stack.json, the trading cards and github-freshness.json.

Evidence classes

Metadata and source review by the lanes; adversarial verification inside the workflow; local integration checks below. No install, run, benchmark or provider trial. The beyond lane's 9 proposals beyond the per-lane cap of 20 are recorded as unverified, not as refuted.

Checks (worktree, after rebase on d093a4c)

python3 -m unittest tests.test_sota_convergence   # Ran 124 tests OK
python3 -m unittest                               # Ran 1438 tests OK (skipped=204, system python)
python3 scripts/validate.py                       # 69 components / 1746 hashed files / 139 receipts
python3 scripts/landscape.py --root .             # passed
python3 scripts/validate_catalogs.py              # passed
python3 scripts/validate_convergence.py --all-recorded --root . --json   # valid
python3 scripts/validate_foundation.py --root . --json                  # ok, 20 layers / 54 decisions
python3 scripts/catalog_decisions.py --check      # passed
python3 tools/sota-convergence/build_verdicts.py --check --root .       # checked
python3 scripts/build_ecosystem.py --check        # passed
git diff --check                                  # clean
gitleaks git . --config .gitleaks.toml --max-target-megabytes 2 --log-opts=d093a4c..HEAD   # first head: no leaks (explorer HTML skipped by size: incomplete coverage); amended head: local guarded scan blocked by another session's scan lock, hosted secret-scan is the check of record

Private inputs, sealed lanes.json (sha256 315946d7…) and per-child usage are under the private state directory named in docs/tasks/2026-09-22-sota-convergence.md (agent-lab).

🤖 Generated with Claude Code

…ct lane run

The saved Claude workflow sota-convergence (run wf_2388074a-e82: foundation,
trading and beyond review lanes, two Opus refuters per proposal, one critic;
110 Opus 5/high children, 908 requests, 1,064,906 output tokens) reviewed the
20-layer foundation taxonomy and the 12-layer trading taxonomy against GitHub
metadata for 469 repositories fetched at 14:57 UTC. build_manifest.py now
emits 20 foundation and 12 trading layers, 109 confirmed components, 17 unique
pins behind upstream (information only), 16 candidate dispositions (none
promoted), per-component why_selected / comparison_that_would_overturn on 140
of 242 rows, and the critic's gaps. No pin or upstream field changed for any of
the 242 rows relative to the first refresh, and no pin string changed for the
225 components already listed; 17 rows were added for the four new foundation
layers. The Codex lane could not run (account usage limit until 2026-09-28);
the ledger rows stay pending for the PR-5 owner to record.

Tooling (tools/sota-convergence/build_manifest.py), each with regression tests:
- review-lane grouping ids outside the taxonomy (the beyond lane's
  awesome-list-convergence, star-audit-targeted-candidates,
  unmaintained-reference-material) are carried verbatim into a new
  lane_groupings section instead of being appended as trading rows, which
  broke scripts/landscape.py coverage;
- every lane's entry for a (layer, repository) is kept: the row takes one by a
  documented precedence (adversarially verified beats unverified, the owning
  lane beats beyond, then lexical) and the rest are published under
  other_lane_reviews; a lane entry naming another card is never folded in;
- a refuted demotion/unmaintained proposal resolves to the component's own
  baseline class (confirmed_default / confirmed_conditional / confirmed_selected)
  instead of an unconditional confirmed_default;
- an adversarial verdict is shared across layers for the same lane, repository
  and proposal kind, with a provenance evidence line;
- lane entries key by catalog_id when set, so two cards for one repository in
  one layer (alpaca-py execution and data) keep distinct rationales;
- sanitization keeps the file basename and JSON pointer of redacted citations
  (<work-dir>/file#pointer, <host-path>/file#pointer), never republishes a home
  directory name, scrubs session-scoped /tmp paths and bare session UUIDs, and
  assert_no_leak refuses the validate.py session-identifier pattern;
- a --citation-review overlay stamps an independent review's findings onto the
  rows they name (citation_review) and keeps the rest under
  citation_review.general with counts; generated text is never hand-edited.
66 new tests (124 in tests.test_sota_convergence).

Independent review: evidence/artifacts/sota-convergence-review-20260922/
manifest-citation-review.json records the four-reviewer pass over the first
refresh (36 findings: 5 generator defects fixed here, 31 citation-quality
findings now attached to 27 rows or listed as general).

Regenerated: layer-verdicts-20260922.json (join only; rows still pending),
docs/ecosystem/index.html, manifests/evidence.json hashes (+1 listed file).

Checks (after rebase on d093a4c): tests.test_sota_convergence 124 OK;
python3 -m unittest 1438 OK (204 skipped, system python); validate.py 69
components / 1746 files / 139 receipts; landscape.py, validate_catalogs.py,
validate_convergence.py --all-recorded, validate_foundation.py,
catalog_decisions.py --check, build_verdicts.py --check, build_ecosystem.py
--check all passed; git diff --check clean.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Sep 22, 2026
…nd Codex (#68)

Full-coverage sweep of the SDK, harness, runtime and worker layers beyond Claude Code and Codex, with preregistered selection. Adds `catalogs/sota-convergence/sdk-runtime-coverage-20260922.{json,md}`, six per-category evidence artifacts under `evidence/artifacts/sdk-runtime-coverage-20260922/` (every shortlisted row's evidence packet and both refutation votes) and `docs/tasks/2026-09-22-sdk-runtime-coverage.md`; registers the nine files in `manifests/evidence.json`. No landscape verdict row, handbook section or SOTA manifest entry is edited (PR-5 and #66 own those); the ledger is the input for the owners of `agent-sdks`, `workers`, `isolation` and `mcp-surfaces`.

| Stage | Agents | Result |
| --- | --- | --- |
| Discover (six categories x web and GitHub-search angles, critic, gap round) | 19 | 356 unique repositories, 297 not yet in the catalog |
| Enrich and gate (`gh api`, no model) | 0 | 290 pass, 64 license flagged, 2 excluded (archived, stale) |
| Shortlist (three Opus judges per category, majority) | 18 | 38 of 372 category-rows; 334 rejections retained with reasons |
| Evidence packet + accuracy and fit refutation | 38 + 76 | one packet per repository from primary sources |
| Selection (three Opus judges per category, criteria C1-C6) + critic | 19 | majority disposition; conservative on no majority |
| Re-collection of three placeholder packets | 18 | rebuilt, refuted and re-judged |

Dispositions after six coordinator resolutions (all listed in the ledger): **integrate_now 1** (`modelcontextprotocol/inspector`), **targeted_candidate 6** (`awslabs/cli-agent-orchestrator`, `swe-agent/mini-swe-agent`, `anthropics/claude-code-action`, `untrivial-ai/agent-orchestrator`, `trailofbits/coop`, `snyk/agent-scan`), keep_but_compare 18, not_adopted 7, refuted 6. Agreement never promoted anything; every row keeps its named comparison or acceptance command. Dispositions are per (category, repository); the one cross-category override (`dagger/container-use`) follows a named canonical-record precedence rule.

Evidence class: source review, plus unretained spot probes some judges ran (named in the ledger), plus one retained local_integration execution for the integrate_now row: agent-lab `tools/mcp-conformance/check.sh` (serena 22, ai-memory 23, socraticode 26 tools, `--strict` clean; `--self-test` observes the failing path with a fixture server whose unportable schema exits 6). No macOS execution for any row; incumbents' capabilities were not re-exercised.

Gate: `validate.py` passed (69 components, 1,763 hashed files, 140 receipts); `build_ecosystem.py --check`, `validate_catalogs.py`, `validate_foundation.py`, `validate_convergence.py --all-recorded` passed; `python3 -m unittest` 1,428 OK (204 skipped); guarded Gitleaks over both commits: no leaks (2 MB target cap, which now excludes nothing, and decode depth 0). Session identifiers, task handles, local paths, the local user name and groups, credential file locations and token scopes were redacted from worker output.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator Author

Carried into main by PR-5 (#71): commit 86bc4be applies this PR's change with its original authorship. Every source file this PR changed (manifest-20260922.json, build_manifest.py, tests/test_sota_convergence.py, tools/sota-convergence/README.md, layer-verdicts-20260922.json, the citation review) is byte-identical at 86bc4be; only regenerated files (docs/ecosystem/index.html, manifests/evidence.json) differ because they were rebuilt on the newer base. Closed without a separate merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant