Choose ns-run's Anthropic slot in fill-first order and move on when credit is exhausted - #1103

Scope

• What this PR changes, in one or two sentences: ns-run (tools/adoption/runtime_profiles.py) now chooses its Anthropic Console slot in the fill-first order the review worker uses. When the first call is refused as credit-exhausted, before any billed output, the run moves to the next usable slot instead of stopping.
• Base commit: ef2fea18 (main). Two commits: 0de55d7f (the change) and ef50b1a2 (the review round's blockers). The head merges cleanly with main 000ae0e3 (git merge-tree rc 0). The merged-tree checks below ran at main 04926b5d; the one commit since, Retain exact seq2 replay sources for a git-bound inventory #1102, touches only blueprints/us-equities/paper-open-e2e/seq2-replay/.
• Lane: lane:foundation
• Owned paths touched:
  • tools/adoption/runtime_profiles.py: fill_first_order, choose_slot(after=), first_call_refusal, credit_refused, cut_short, and the print-run retry;
  • tools/claude-review-worker/worker.py: credit_exhausted(status, text), named and used by analyze() (no behavior change);
  • tests/test_runtime_profiles.py (FillFirstSlotTests) and tests/test_claude_review_worker.py (a classifier test);
  • manifests/evidence.json: re-registration.

The incident. The command center's ns-run pi researcher run, on the pi-claude profile at 2026-10-14T01:01:52Z, was refused with Anthropic's 400 invalid_request_error "Your credit balance is too low to access the Anthropic API". It billed 0 tokens and stopped. The cause was choose_slot():

• it walked a fixed api-4/3/2 list, stale against the fill-first order;
• it took the first slot whose no-token check (GET /v1/models) read "valid". A key whose prepaid credit is exhausted still answers 200 there, so it chose anthropic-api-4, which was out of credit;
• the run never moved on.

The api-actions lane's ledgered one-token probes at 01:04:52-01:04:57Z confirmed the state: anthropic-api-4 and anthropic-api-2 refused for credit, while anthropic-api-7, -5, anthropic-api and -6 had credit. credential_run.py injects one named slot by design and never switches; switching belongs to its caller.

The change.

• Order: choose_slot() walks fill_first_order(), which is the review worker's DEFAULT_KEYS, so every Console consumer follows one order. This PR does not change that order; it changes only through Take up the command center's two remaining notes on the key slots #1050, on its own review, and ns-run then follows with no further change.
• Screening: the no-token check now removes only a slot that is not stored, invalid (401) or forbidden (403). "valid" no longer decides; a rate-limited or unreachable answer does not remove a slot either.
• Retry: a print run checks its first model call:
  • Pi records a refusal as an assistant message_end with stopReason: "error" and errorMessage "<status> <body>", with zero usage, and exits 0. That is the incident's own run record.
  • OpenCode emits an error event.
  • When the refusal is credit-exhausted by the worker's classifier (402, or 400 with "credit balance is too low"), the refused slot is recorded in the receipt's credit_refused_slots, and the run starts again on choose_slot(after=<that slot>).
  • Any other refusal, or a first call that billed usage, ends the run on its slot.
  • With no slot left, the last refused run's receipt stands, and its own slot is in the list too.
  • A timed-out or signalled attempt never moves to another slot, whatever its log holds. It keeps its own receipt and exit status (cut_short: timeout's 124, 125, 126, 127 and 137, the credential runner's 128+N, a negative subprocess status).
• Classifier: the worker's inline expression is now credit_exhausted(status, text). analyze() uses it, and runtime_profiles.py imports it rather than copying it.

Review round at 0de55d7f. The command center's Claude micro passed. The local review (pr1103-0de55d7f/review_output.json) and the Astra read (full1-pr1103-0de55d7f4/GPT-VERDICT.md) asked for changes, with three distinct P2 blockers and no P1. Commit ef50b1a2 takes all three:

• A cut-short attempt stops failover (Astra): the new cut_short check runs before the credit check.
• The last refused slot is recorded (both reads): the slot is appended before a successor is looked for.
• Pinned refusal-event sources (local review): first_call_refusal cites them, as listed under SOTA sources.

Delta round at ef50b1a2. The command center's Claude micro passed, and the Astra delta read (fix1-pr1103-ef50b1a23/GPT-VERDICT.md) acknowledged the head with no blocker. The local delta review (pr1103-ef50b1a2/review_output.json) raised one P2, which the command center ruled a follow-up. It is the first item below.

Follow-ups (not in this change):

• OpenCode partial output before a credit refusal (the local delta review's P2): OpenCode retries a failed stream itself (anomalyco/opencode@53d1eabb, packages/opencode/src/session/processor.ts:649-689), so a credit refusal can follow text, reasoning or a tool call. The parser stops only at a finished step, so such a run would start again on another slot. The fix stops at a text, reasoning or tool_use event too. It is prepared with a fail-first test and goes up as its own PR once this one lands.
• tools/adoption/runtime_acceptance.py also calls choose_slot(). It gets the order and the screening, but not the retry. Its loader_health_ready now means a stored slot that was not rejected outright, not shown credit.
• The interactive (TUI) path chooses the first usable slot. A refusal there is visible to the operator, but there is no automatic retry.
• The parser decides on the first qualifying refusal. A log holding a zero-token refusal and then a billed completion would still move on. The generated Pi profile disables native retries (runtime_profiles.py:482), so this is not a reproduced billing case; whole-run usage and recovery-order tests remain to add (Astra's P3).
• A slot-selection error at startup happens after the worktree is created and before the print run's cleanup, and an all-exhausted Pi run still returns the native exit status 0 (Astra's P3).
• The OpenCode refusal path is tested on constructed events shaped as the pinned source emits them. It is not yet run against a live refusal on the qualified OpenCode 1.18.35 binary.
• The incident's run events, the probe ledger and the errors-page capture are private lane records, cited here by name and not committed.
• The installed ~/.local/bin/ns-run runs runtime_profiles.py from the runtime-profiles lane's worktree, not main. After landing it must be re-pointed to main's file, by that lane's install or by api-actions. The record of which says so.

SOTA sources

• The review worker's own fill-first implementation and classifier: tools/claude-review-worker/worker.py (DEFAULT_KEYS, analyze()), which this change reuses instead of duplicating.
• Pi 1.1.0, earendil-works/pi@04566631 (the qualified release):
  • packages/ai/src/api/anthropic-messages.ts:893-895 sets stopReason: "error" and errorMessage from the SDK error; :589-596 starts usage at zero;
  • packages/agent/src/agent-loop.ts:443-455 emits that message as message_end;
  • packages/coding-agent/src/modes/print-mode.ts:108-112 writes each event as a JSON line;
  • @anthropic-ai/sdk 0.129.0 (pinned at packages/ai/package.json:70), src/core/error.ts:40-50: the error message is "<status> <body>".
• OpenCode 1.18.35, anomalyco/opencode@53d1eabb (the qualified release):
  • packages/opencode/src/cli/cmd/run.ts:750 emits step_finish, :789 emits the error event, and :842 sets exit status 1;
  • packages/opencode/src/session/message-v2.ts:707-717 and packages/core/src/util/error.ts:55-60: a provider refusal is {name, data: {statusCode, ...}}.
• The installed timeout --help, "Exit status" (124, 125, 126, 127, 137), and tools/credentials/credential_run.py, "Exit status" (the command's own; 128+N for a signal death).
• Anthropic API errors (https://docs.anthropic.com/en/api/errors.md, capture of a public page, read 2026-10-12): 402 billing_error. The 400 "credit balance is too low" form is the one the incident's own run recorded (request id in the command center's incident note, cited by name).

Evidence-class table

Local commands run

$ python3 -m unittest tests.test_runtime_profiles.FillFirstSlotTests   # with the 0de55d7f runtime_profiles.py
FAILED (failures=6)
$ python3 -m unittest tests.test_runtime_profiles.FillFirstSlotTests tests.test_claude_review_worker.ReviewRunTest.test_the_credit_classifier_is_one_function_every_consumer_imports
Ran 9 tests in 0.577s
OK
# the next five on the merge of ef50b1a2 with main 04926b5d
$ python3 -m unittest tests.test_runtime_boot tests.test_runtime_database_isolation tests.test_runtime_opencode_native tests.test_runtime_profile_readiness tests.test_runtime_profiles tests.test_claude_review_worker tests.test_upstream_surface_watch tests.test_validate tests.test_evidence_manifest tests.test_task_model_routing tests.test_runtime_publication tests.test_registry_rows
Ran 387 tests in 28.231s
OK (skipped=12)
$ python3 -m unittest tests.test_catalog_freshness_propose
Ran 87 tests in 97.841s
OK
$ python3 -m unittest tests.test_runtime_worker_openhands_push_gate
OK
$ python3 scripts/evidence_manifest.py --check
{"files": 11943, "status": "passed"}
$ python3 scripts/validate.py
{"components": 71, "hashed_files": 11943, "profiles": 4, "receipts": 240, "status": "passed"}
$ gitleaks git --log-opts="origin/main..HEAD" --redact .
no leaks found
$ git merge-tree --write-tree origin/main HEAD   # main 04926b5d, before the push
rc 0

CI at 0de55d7f failed validate, validate-shard-0 and validate-shard-5 on one line, the date-based model-currency rule for claude-fable-5-1. #1101 fixed that on main (04926b5d). CI at ef50b1a2 passes: 26 checks, 5 skipped.

Decision record

None: this implements the fill-first rule already recorded in docs/decisions/2026-10-10-anthropic-key-slots-and-practice.md and docs/secret-storage.md (item 10, "Fill-first failover") for the ns-run profiles.

Host evidence

Not applicable (no evidence/hosts/ change).

Checklist

• New/changed GitHub Actions are pinned to a full commit SHA with a
  version comment (no floating tags). (none changed)
• New/changed workflows declare top-level permissions: {} and grant each job
  only what it needs (contents: read, or an explicitly justified addition). (none changed)
• No secrets are printed, logged or committed; no new required secret was
  added without a documented owner.
• No new paid hosting, subscription or billing surface was introduced.
• Peer-owned untracked files and worktrees were preserved (not deleted,
  moved or overwritten).

🤖 Generated with Claude Code