Landscape-sweep harness: two-family (Claude + GPT-6) saturation sweep as a reproducible repo tool by seathatflowsinourveins · Pull Request #324 · native-agent-stack/native-agent-stack

seathatflowsinourveins · 2026-09-26T04:23:02Z

Scope

Lane: lane:foundation.
What: the saturation-sweep lane (recipes/saturation-sweep.md) becomes a reproducible repository tool, so any future session on any host (Linux/WSL2 or macOS) can rerun it without the agent-lab-only path. It lives in tools/sota-convergence/landscape-sweep/, and the recipe gets a dated "Run the lane in this repository (2026-09-26)" section.

What it does

Per layer:
- discover:<layer> (Claude Opus, max effort) runs alongside gpt6-discover:<layer> (GPT-6-Astra, max effort, codex --search exec, read-only sandbox, stdin closed, never ultra).
- The two lists are merged, capped at 8 per layer, and anything dropped is logged.
- Then three refuters: refute-facts:<layer> (Sonnet, max effort), refute-fit:<layer> (Opus, max effort) and gpt6-refute-fit:<layer>.
- The fit vote is two-family: refuted if either family refutes. When GPT-6 is unavailable the vote is recorded as single-family, never passed off as agreement.
Across layers: a completeness critic, then at most 8 follow-up layers.
Prompts: every worker carries the merit-not-incumbency rule and role-matched installed skills, and records skills_used.
GPT-6 runner (codex_call.sh/codex_job.py):
- a flock semaphore, 3 slots by default;
- the usage limit is detected in stderr and in --json error/turn.failed events. That was a real miss in the live run, fixed there too.
- refused jobs finish immediately instead of blocking their wrappers.
Evidence pipeline:
- convert.py produces returns, lanes and ledger layers; lost or degraded layers are never counted as clean.
- make_result.py produces RESULT.json, and usage_record.py wraps child-usage.mjs.
- source_reviews.py uses gh api only.
- build_args.py produces compact args, plus an embedded-args copy for scriptPath launches so large args never enter the coordinator's context.
Seeds: kept in seeds-20260926.json.

SOTA sources

The repo's own contract: scripts/saturation_ledger.py (labels, returns.json shapes, workflow_run equal to the transcript_dir segment) and recipes/saturation-sweep.md.
Reference implementation: agent-lab sota-convergence.js, cited, not copied.
openai/codex rust-v0.155.1: exec_events.rs and event_processor_with_jsonl_output.rs, which report fatal errors as stdout error/turn.failed events.
Claude Code Workflow tool semantics (agent/parallel/pipeline) and child-usage.mjs in examples/claude-native/workflows/.

Evidence-class table

Claim	Class
Behaviour parity with the 2026-09-26 live prototype: first-round GPT-6 prompts byte-identical; `convert.py` on the real smoke run reproduces the prototype	local_integration against scratchpad copies of that run
Limit detection, semaphore, argv, stdin, empty working directory, timeout	synthetic, with a fake codex on PATH
`sweep.js` control flow: labels, models, effort, cap, follow-ups, lost rounds	synthetic, run under node with stubbed agents
convert → make_result → `saturation_ledger.append` with 0 errors	synthetic checkout
A real Workflow launch or real Codex call with the packaged files	not yet run; the live run used the prototype

Commands

uv run --no-project --with jsonschema --with pyyaml python -m unittest tests.test_landscape_sweep_harness -v   # 66 OK (2 skip without shellcheck/bash3.2)
uv run ... -m unittest tests.test_saturation_ledger ...                                                        # OK
python3 scripts/validate.py                                                                                    # passed (6072 hashed)
bash -n codex_call.sh; shellcheck codex_call.sh                                                                 # clean

Review

A two-family review (Claude Opus at max and GPT-6-Astra at max), then one repair round. It had 12 plus 5 findings, with 4 overlapping, and all were verified and fixed:
- a failed or degraded lane never counts as clean;
- a proposal and all its votes stay grouped by round;
- a failed follow-up is recorded as lost with a reopen trigger;
- generated paths are shell-quoted;
- slugs are canonicalized for URLs versus bare owner/repo;
- source-review file names cannot collide;
- RUST_LOG is scrubbed so cited text on stderr cannot trigger the limit check.
The workstation coordinator offered a further GPT-6 review before merge; that request is pending.

🤖 Generated with Claude Code

… and macOS) tools/sota-convergence/landscape-sweep/ runs the landscape-sweep lane of recipes/saturation-sweep.md from this repository. Before this, the lane ran only from agent-lab, which remains an alternative. The lane has two model families: - Discovery: Claude Opus plus GPT-6-Astra through the Codex CLI. - Refutation: a Sonnet facts refuter, and fit refutation by one refuter from each family. - A completeness critic, followed by one bounded follow-up round. The harness then converts the run into the evidence saturation_ledger.py binds: returns.json, lanes.json, ledger layers, source reviews, a sanitized child-usage record and RESULT.json. Changes from the 2026-09-26 prototype: - No hardcoded paths: every tool takes --work-dir or SWEEP_WORK_DIR, and build_args.py stages the runtime into the work dir. - The date, layer count and skills date are template placeholders. Filled with the 2026-09-26 values, the templates reproduce that run's prompts_sha256 3adfbed7...18d4. - The GPT-6 runner uses only the Python standard library for locking, detaching and the timeout, so it runs on macOS bash 3.2 without flock, setsid or timeout. - Usage-limit detection also reads the error and turn.failed events that Codex prints on stdout in --json mode (openai/codex rust-v0.155.1, codex-rs/exec/src/exec_events.rs). It never reads model content. - Known-repository slugs are no longer cut by rstrip(".git"), which had shortened qdrant/qdrant to qdrant/qdran in 30 of 32 layers. - A layer none of whose rounds returned is excluded rather than recorded as a clean layer. - convert.py sanitizes host paths, scans its output for PRIVATE_CONTENT, and checks each wrapper's copy of a GPT-6 output against the file Codex wrote. tests/test_landscape_sweep_harness.py adds 50 stdlib tests on synthetic fixtures. They use a fake codex and gh on PATH, run sweep.js under node with stubbed agents, and append convert.py and make_result.py output with saturation_ledger.append in a synthetic checkout. recipes/saturation-sweep.md gains a dated section, and tools/sota-convergence/README.md gains a pointer. That README is a registered file, so its manifests/evidence.json entry needs re-registration. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…ever counts as clean A Claude Opus and GPT-6-Astra review of the first package reported nine findings. Each was checked against the source; all were real, and the reviewers' reproductions now show the repaired behaviour. - Failed lanes (high). convert.py wrote votes: retained for a layer whose GPT-6 fit jobs, facts refuter or one discovery family had failed. With every GPT-6 fit job failed, both layers derived as clean (count 1). convert.py now lists each layer's retained failures under returns.json failures/<layer>: - lost rounds - missing discovery families - missing votes - a lost critic - a critic-flagged layer beyond the follow-up cap - GPT-6 copy problems Each such layer gets a retained_failure reopen entry pointing there, so it resets (the same sweep now derives 0). make_result.py refuses a layer whose failures lack that entry, and merges --reopen entries instead of replacing them. convert.py exits 4 on file_only (an exit-0 Codex output that never reached the workflow), no_file and file_unparseable, as well as mismatch. - Re-proposals. A repository proposed again in the follow-up kept its first-round row while its votes were replaced one by one, and a vote missing in the later round fell back to the earlier round. The latest round's proposal and all three of its votes now travel together, and a missing vote in that round counts as refuted. A vote refutes unless it says refuted: false, and a conflicting double vote refutes. - Effort drift. usage_record.py exits 1 on a non-zero child-usage.mjs exit or on effort_mismatches. make_result.py refuses a non-zero measurement exit code, listed mismatches, and any child not at effort max alone. Each Claude vote names its own worker's measured model and effort (null when not measured) instead of a hardcoded max. The lane limits name the GPT-6 model the jobs reported. - Lost follow-ups. sweep.js keeps a null follow-up round as {layer_id, catalog, round: followup, lost: true} and returns lost_followups. A layer the critic names twice gets one round, since a repeat would reuse the first round's GPT-6 job ids. convert.py attributes a null follow-up from an older run by the critic's plan, and refuses follow-up rounds that do not match that plan. - Shell quoting. Every argument and redirection target in the wrapper commands is POSIX-quoted (shlex.quote rules). Layer ids must match a job-id-safe pattern. Work directories with control characters are refused by sweep.js and by build_args.py. - canon(). URLs are told from bare slugs before the slug is taken, so httpie/cli and branch URLs are canonical. slug() also handles bare owner/repo with .git, and sweep.js slug() is kept identical (tested under node). - Source reviews. They keep the 2026-09-23 <owner>-<repo>.json names. Names two repositories share get a sha256-of-slug suffix, and a file reviewing another repository is never overwritten. Tests: 66 stdlib tests on synthetic fixtures, 16 of them new. Every new test of a repaired behaviour fails on the previous commit. The slug-parity check and the clean-layer contrast are guards that hold on both commits. All 66 pass with BASH32_BINARY and shellcheck, and 64 pass with 2 skipped without them. bash -n and shellcheck 0.11.0 pass on codex_call.sh. scripts/validate.py reports only the registered tools/sota-convergence/README.md hash, from the first commit's pointer line. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

chatgpt-codex-connector · 2026-09-26T04:23:06Z

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

seathatflowsinourveins · 2026-09-26T04:35:52Z

GPT-6 cross-family review (Codex gpt-6-astra, effort max, read-only) at head f5aaf83d, run from the workstation coordinator at the author's request.

DEFECTS FOUND

high | tools/sota-convergence/landscape-sweep/convert.py:313 | Reproposals overwrite earlier votes before missing-vote failures are collected. A failed first-round GPT-6 fit followed by successful refutations produces failures: {}, no reopen entry, and a clean count of 1; the real manifest builder and ledger accept it. | Fix: Collect failures from every round before selecting the final proposal and votes.
high | tools/sota-convergence/landscape-sweep/codex_job.py:212 | Successful jobs are reused solely by job ID, without reading the new prompt or schema. On Workflow resume, regenerated proposals can therefore receive cached GPT-6 votes for different claims; the output-copy check still passes. | Fix: Bind cached jobs to prompt, schema, model and effort hashes; reject or rerun mismatched inputs.
medium | tools/sota-convergence/landscape-sweep/codex_job.py:184 | Arbitrary stderr text triggers LIMIT, while RUST_LOG remains inherited despite the author’s claimed repair. Pinned Codex source logs model response data at trace level. A quoted usage-limit message in that data reproduced a false limit. | Fix: Remove inherited tracing configuration from the Codex subprocess and distinguish structured provider errors from diagnostic/model text.
medium | tools/sota-convergence/landscape-sweep/codex_job.py:227 | Retrying a failed job deletes its events, error, timestamps and exit status, then overwrites its inputs. This destroys failed-attempt evidence and removes its usage from subsequent results. | Fix: Preserve immutable attempt directories and account for every attempt, marking unavailable usage explicitly.
medium | tools/sota-convergence/landscape-sweep/codex_job.py:289 | Timeout cleanup returns when the parent exits after SIGTERM, without ensuring its process group is gone. A child ignoring SIGTERM remained running in reproduction. | Fix: Escalate surviving group members to SIGKILL after the grace period even when the parent has exited.
medium | tools/sota-convergence/landscape-sweep/README.md:242 | Exact prototype cost figures and the parity claims at line 303 lack retained measurement/comparison evidence. The checked-in expected-hash literal does not independently establish historical parity. | Fix: Retain sanitized commands, returned counters and comparison results with their evidence classes, or remove the unsupported claims.

Every finding comes from the reviewer's own reproduction or its pinned source reading. The author verifies each one before repairing it.

- convert.py: collect missing-vote failures per round, so a failed round-1 GPT-6 fit stays a retained failure (and reopens the layer) after a follow-up re-proposal. - codex_job.py: bind a done job to its inputs (prompt/schema sha256, model, effort); a resumed Workflow with a regenerated prompt reruns instead of reusing a cached vote, and a running job with different inputs is refused. - codex_job.py: keep every earlier attempt unchanged under attempts/<n>/ and report each attempt's exit and usage (unavailable when none was reported); convert.py adds them to gpt6_usage.earlier_attempts. - codex_job.py: run Codex without RUST_LOG (trace level logs model response data, codex-rs/codex-api/src/sse/responses.rs at rust-v0.155.1); stderr counts for the usage limit only as an ERROR/Error line when no turn completed. - codex_job.py: timeout cleanup KILLs the whole process group after the grace period even when Codex itself already exited. - README: drop the unretained smoke cost figures and prototype-parity assertions; each run's retained record is its cost reference, parity stays unverified until compared. Eight regression tests fail on f5aaf83 and pass here; suite 71 tests OK (2 skipped). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…IMIT refusal GPT-6 re-check of a0988d1 (findings 1, 3, 4, 5 resolved, no new defects) left two: - codex_job.py: a start whose inputs changed but that LIMIT refuses now archives the finished attempt, so result no longer returns the old claim's exit-0 output (the wrapper would have accepted it as ok). Reproduced by the extended test before the fix. - README: the prompts_sha256 note says what was checked: the value computed on 2026-09-26 from that run's staged templates.json, a local check; the run's registered record is the evidence. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

seathatflowsinourveins · 2026-09-26T04:54:40Z

Repair of the GPT-6 review of `f5aaf83` (2 high, 4 medium)

#	Finding	Repair	Re-check (GPT-6-Astra, max, read-only, diff `f5aaf83`..`a0988d1`)
1	high: re-proposal overwrote round-1 missing votes	`convert.py` collects missing-vote failures per round; a failed round-1 GPT-6 fit stays a retained failure and reopens the layer	RESOLVED
2	high: done jobs reused by job id alone	`codex_job.py` binds a job to `inputs.json` (prompt/schema sha256, model, effort); changed inputs rerun, a running job with other inputs is refused	NOT_RESOLVED: under `LIMIT`, changed inputs still served the old exit-0 output → fixed in `04c4809` (the refused start archives the old attempt; test reproduced the stale `ok` before the fix)
3	medium: stderr text and inherited `RUST_LOG`	Codex runs without `RUST_LOG*`; stderr counts only as an `ERROR`/`Error` line when no turn completed	RESOLVED
4	medium: retry deleted the failed attempt	earlier attempts move unchanged to `attempts/<n>/`; `result` and `gpt6_usage.earlier_attempts` account every attempt (`usage_unavailable` when none reported)	RESOLVED
5	medium: TERM-ignoring child survived timeout	`stop_group` KILLs the whole group after `kill_grace_s` even when Codex exited	RESOLVED
6	medium: unretained cost/parity claims	cost figures removed (each run's retained record is its cost reference); parity marked unverified	NOT_RESOLVED: line 115 still said "reproduce that run's" → reworded in `04c4809` to the local check it is (value computed 2026-09-26 from the run's staged `templates.json`; the registered record is the evidence)

Re-check: "no new defects". Every finding has a regression test that fails on the pre-repair code (eight in 6a68feb, one extended in 04c4809). Suite: 71 tests OK (2 skipped), validate.py passed.

Evidence class: local tests and a cross-family source review; no live GPT-6 sweep run used this packaged runner yet (the running 2026-09-26 sweep uses the prototype).

🤖 Generated with Claude Code

Scout and others added 3 commits September 26, 2026 00:22

Register the landscape-sweep harness files (hot-file commit)

f5aaf83

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Sep 26, 2026

Scout and others added 4 commits September 26, 2026 00:44

Re-register the repaired landscape-sweep harness files (hot-file commit)

a0988d1

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Re-register the repaired landscape-sweep harness files (hot-file commit)

f2c3326

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

seathatflowsinourveins enabled auto-merge (squash) September 26, 2026 04:54

seathatflowsinourveins merged commit 1b0e459 into main Sep 26, 2026
24 checks passed

seathatflowsinourveins deleted the claude/landscape-sweep-harness-20260926 branch September 26, 2026 05:08

seathatflowsinourveins mentioned this pull request Sep 26, 2026

Landscape sweep 2026-09-26 record: all 32 layers, 56 survivors, manifest-20260926 for the verdict wave (11 lane limits) #357

Merged

seathatflowsinourveins mentioned this pull request Oct 3, 2026

Client configuration for the new distribution: wire only what the definitive manifest installs (map, tool, F9) #608

Merged

5 tasks done

Navigation Menu

Repository navigation

Landscape-sweep harness: two-family (Claude + GPT-6) saturation sweep as a reproducible repo tool - #324

Landscape-sweep harness: two-family (Claude + GPT-6) saturation sweep as a reproducible repo tool#324
seathatflowsinourveins merged 7 commits into
mainfrom
claude/landscape-sweep-harness-20260926

seathatflowsinourveins commented Sep 26, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 26, 2026

Uh oh!

seathatflowsinourveins commented Sep 26, 2026

Uh oh!

seathatflowsinourveins commented Sep 26, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

seathatflowsinourveins commented Sep 26, 2026

Scope

What it does

SOTA sources

Evidence-class table

Commands

Review

Uh oh!

chatgpt-codex-connector Bot commented Sep 26, 2026

Uh oh!

seathatflowsinourveins commented Sep 26, 2026

Uh oh!

seathatflowsinourveins commented Sep 26, 2026

Repair of the GPT-6 review of f5aaf83 (2 high, 4 medium)

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Repair of the GPT-6 review of `f5aaf83` (2 high, 4 medium)