GPT-6 lane: capacity retry, idle watchdog that ignores reconnect notices, completed-turn success, per-lane effort (ultra disclosed as xhigh + delegation) and web search - #549

Scope

• What this PR changes: it hardens the landscape-sweep GPT-6 lane harness (codex_job.py) against the failures seen on 2026-09-30, and adds per-lane effort and web search:
  • Codex's terminal "Selected model is at capacity." error is retried twice with capped, jittered backoff.
  • An idle watchdog ignores reconnect notices.
  • Success requires a completed turn.
  • Effort and web search are set per lane, following the user's 2026-09-30 GPT standard.
• Base commit: 11227bfd
• Lane: lane:foundation
• Owned paths touched:
  • tools/sota-convergence/landscape-sweep/{codex_job.py,codex_call.sh,README.md}
  • tests/test_landscape_sweep_harness.py
  • docs/decisions/2026-09-29-codex-lane-default-effort.md (dated amendment)
  • manifests/evidence.json (hash re-registration only)

Why. These are the failures observed on 2026-09-30, all retained in the lane work dirs:

• Two jobs ended at once with turn.failed "Selected model is at capacity." (exit 1, no output).
• One native job printed its last event (item.completed web_search) and then nothing for hours until it was killed by hand.
• A research job was still working when the 3000 s budget ended it.
• A job whose codex process group got SIGTERM was recorded as done, exit 0, empty output. A rerun with the same inputs would then have answered "already done".

SOTA sources

Every rule was checked against openai/codex at tag rust-v0.159.2 (commit ff6aec96948b), which is the installed client (codex-cli 0.159.2):

• Capacity is terminal upstream. codex-rs/protocol/src/error.rs:384-412: retry_delay() returns None for ServerOverloaded. The retry therefore lives in the harness.
• SIGTERM exit 0. codex-cli/bin/codex.js:270-293: the npm wrapper forwards the signal. When the child dies by signal, the wrapper re-raises it on itself while its own listener is still registered, so the listener catches it and the wrapper exits 0. The native binary handles only ctrl_c (codex-rs/exec/src/lib.rs:1126-1132). exec prints turn.completed and then writes -o (codex-rs/exec/src/event_processor_with_jsonl_output.rs:525-536,631-636). Success therefore requires exit 0, turn.completed and a non-empty -o file; anything else is exit 126, incomplete.
• Reconnect notices are JSONL error events. The chain is core/src/session/mod.rs:5030 → app-server/src/bespoke_event_handling.rs:1054-1070 (will_retry: true) → exec/src/event_processor_with_jsonl_output.rs:447-458. So:
  • the watchdog does not count error events as progress;
  • the silent hang was not a reconnect loop, and its cause stays unknown;
  • the upstream default unbounded_connection_retries (Stable, on: features/src/lib.rs:1292-1297; backoff 5→60 s: core/src/responses_retry.rs:71-96) is kept. Codex rides out short outages, and the watchdog stops a loop longer than the idle budget.
• What ultra means at 0.159.2.
  • For gpt-6.1-sol and gpt-6-astra, ultra resolves to the model's multi_agent_reasoning_effort "xhigh" on every root request (models-manager/models.json:22,196; protocol/src/openai_models/reasoning_effort.rs:12-35; core/src/client.rs:863-872), and it enables proactive delegation (core/src/session/multi_agents.rs:77-103). The live catalog cache (fetched 2026-09-30T18:54Z, client 0.159.2) agrees.
  • A root multi-agent wait may block up to 3600 s (core/src/config/mod.rs:257; tools/handlers/multi_agents_v2/wait.rs:53-64), and exec drops sub-agent items (exec/src/lib.rs:1624-1629). Ultra lanes therefore default to idle 4200 s and total 14400 s, and a staged ultra idle of 3600 s or less is refused.
  • exec reports primary-thread usage only (exec/src/lib.rs:1636-1638), so ultra usage is labelled primary_thread_only.
  • A model outside the catalog gets fallback metadata (models-manager/src/model_info.rs:98-106), where ultra would resolve to medium, so it is refused.
• Web search modes: protocol/src/config_types.rs:371-382.
• Backoff shape: capped exponential backoff with jitter (AWS Architecture Blog, "Exponential Backoff And Jitter").

Evidence-class table

Local commands run

$ TMPDIR=<outside /tmp> python3 -m unittest tests.test_landscape_sweep_harness
Ran 166 tests ... OK (skipped=3)
$ python3 scripts/validate.py
{"components": 69, "hashed_files": 8515, "profiles": 4, "receipts": 176, "status": "passed"}
$ git diff --check   -> exit 0

Decision record

docs/decisions/2026-09-29-codex-lane-default-effort.md, "Amendment (2026-09-30)":

• the user's decision (verbatim): "yes no limit, for highest quality resolution, gpt6.1sol at ultra and astra when tasks truly needed it";
• the harness default stays max;
• a lane may stage ultra;
• blind or isolated review lanes stay at max, because ultra delegates;
• the resolved-effort fact and the ultra idle rule.

Overturn conditions:

• a Codex release makes ServerOverloaded retryable, or fixes the wrapper's exit status: drop the matching harness rule;
• the catalog changes ultra's resolved effort: update the map.

Review history

1. Draft cc00dbd9, from a gpt-6.1-sol research lane.

2. Finalization 9aff1344, by a native gpt-6.1-sol builder at ultra: the completion rule, the error-event rule, per-lane effort and web search.

3. Claude Opus cross-family review: repair. Findings:

   • blocking: a 7200 s default outlasted the sweep wrapper's 4320 s wait;
   • ultra shows "ultra" but sends xhigh;
   • ultra delegation can outlast the idle budget;
   • ultra usage leaves out sub-agents;
   • the reconnect override cost outage ride-out;
   • the named provider had become a no-op;
   • eight low-severity items.

4. Repair round db75ddca, by the same native builder with coordinator decisions. It fixes all of the above in the allowed paths, and is verified by the tests and the live checks above.

5. Hot-file protocol: rebuilt as b019b900 (content, no hot file) + ed818625 (last commit: manifests/evidence.json only) on main 8fc86119; tree byte-identical to the reviewed head 9f68193c; Gate A owner (6d): no objection, merge after c5's train.

Residuals

These are outside this PR. All three files are also touched by the open #541, and I've notified its owner.

• sweep.js:92: the 4320 s wrapper wait. Queued jobs, and ultra lanes, can still outlast it.
• convert.py:200-208,291: it drops request_effort and failure, and would treat primary_thread_only usage as complete.
• build_args.py:508: it always stages max, which is correct for the sweep's blind lanes, and has no flags for effort, web search or budgets.

Checklist

• No new or changed GitHub Actions.
• No new or changed workflows.
• No secrets printed, logged or committed; the gitleaks pre-commit hook passed on every commit.
• No new paid hosting, subscription or billing surface.
• Peer-owned untracked files and worktrees preserved.

🤖 Generated with Claude Code