Add Anthropic key slots api-5 to api-10, the key practice and a no-token health check - #969

Scope

• What this PR changes: it adds the Anthropic API key slots anthropic-api-5 to anthropic-api-10 (the command center's local commit, cherry-picked), records the 2026-10-10 key practice in docs/secret-storage.md, and adds tools/credentials/anthropic_key_health.py, a no-token health check per slot. It follows Add two optional direct Anthropic Console key stores #930's pattern.
• Base commit: 8b1ef0c37, with 8 commits. Three rebases:
  • 2026-10-11, from e1c88ff3d onto 8c95a50d1 (the command center's triage): only manifests/evidence.json conflicted; main's copy was taken and each commit re-registered.
  • 2026-10-12 at 02:12Z, onto 86fa38442: besides manifests/evidence.json, two conflicts, each resolved as the command center approved ("go 969"). tests/test_credential_run.py's INJECTABLE_IDS is the union of this PR's ANTHROPIC_IDS form (all ten) and main's alpaca-paper-c (Credential inventory: alpaca-paper-c slot for paper account C; the guard reads a one-letter pointer suffix #1014). The catalog's five docs/secret-storage.md citations are re-derived from the rebased document as the line that names each key; tests.test_upstream_surface_watch passes at each of the 8 rebased commits. This PR's own changed lines in the inventory, docs/harness-defaults.md, docs/secret-storage.md and the health tool are unchanged by the rebase.
  • 2026-10-12 at 04:13Z, onto 8b1ef0c37 (the command center's ask before landing): only manifests/evidence.json conflicted; main's side was taken and each commit re-registered. Without the manifest, the patch-ids of all 8 commits and of the whole range are identical to 295c3e51's. The two "go 969" resolutions are unchanged: tests/test_credential_run.py and docs/secret-storage.md are identical to 295c3e51's, and this PR's changed lines in the catalog are identical too (main's Resolve the upstream-surface backlog to Claude Code 2.1.296 and Codex rust-v0.162.1 #950 changed other lines of it).
• Lane: lane:foundation.
• Owned paths touched: adoption/credential-inventory.json, docs/secret-storage.md, docs/decisions/2026-10-10-anthropic-key-slots-and-practice.md (new), docs/harness-defaults.md, catalogs/foundation/upstream-surface-dispositions.json (citation lines moved), tools/credentials/anthropic_key_health.py (new), tests/test_anthropic_key_health.py (new), tests/test_credential_run.py, manifests/evidence.json.

Key practice, from the owner's directions of 2026-10-10 (relayed by the command center, paraphrased):

• One purpose per slot. Organizations are named by id prefix only.
• Keys are added and rotated only through open_credential_terminal.sh's hidden-input window, and injected per command by credential_run.py.
• CI keys live only in main-restricted environment secrets.
• The fill-first order is api-4 → api-3 → api-5 → anthropic-api → api-2 → api-6..10. api-3 serves CI review first, then local quality work. Keys are never used round-robin, because a prompt cache never crosses organizations.
• There are no admin keys: every account behind these keys is individual, and the Admin API is unavailable for individual accounts.

Inventory:

• The six new slots mirror anthropic-api-4.
• anthropic-api-3 gets its purpose (CI review first, then local quality work) and its rotation now includes re-setting the claude-review secret.
• The "no Anthropic key is stored in GitHub" notes on anthropic-api-3 and anthropic-api-4 are marked superseded by the API-key ruling (Authenticate CI Claude review with an API key in a main-only environment #968).

Health tool:

• Stdlib only. It calls GET /v1/models and GET /v1/organizations/me, which spend no tokens.
• The key reaches only a child started through credential_run.py. Request headers go to the child on stdin from an allowlist, and the child follows no redirects and reads no response body.
• The report gives stored, validity, status class and the first eight characters of the org id (from the anthropic-organization-id response header). It never shows a key or an org name.
• It is not yet run against a real key. No timer is added; the command center adds one after landing.

After the GPT read at 445c61c1 (CHANGES_REQUESTED, two P2s), one commit, 5c5bff1e at the head (written as 5ca54b8c, then rebased as 295c3e51 at 02:12Z and as 5c5bff1e at 04:13Z; the tool and its tests are byte-identical):

• An invalid inventory gave an empty, passing report: entries: [], or an anthropic-api entry with no variables, exited 0. The tool now applies scripts/credential_status.py's inventory_errors first, as the credential runner's load_inventory does. An anthropic-api entry in another shape, or no anthropic-api entry, is refused too. Each exits 1 before any slot is checked, naming only counts and inventory ids.
• A failed probe process counted as healthy: a child that printed a 200 line and then exited 7, was killed, or reported an error passed with exit 0. A nonzero return code now fails the probe before its output is read (probe_exit_<n>, or probe_signal_<n> for a negative code: CPython 3.13.16 Lib/subprocess.py:484, "The exit code of the process, negative for signals"). A line reporting an error, at the top level or for GET /v1/models, is probe_failed whatever status it printed.
• docs/secret-storage.md and the decision record describe the change. The catalog's five citations of later secret-storage.md lines move by the four lines added.

Follow-ups (not in this change):

• From the command center's Claude delta at 295c3e51 (PASS on the code; its P3s):
  • A real no-response is now probe_failed, not unreachable: the child prints a request error with no status, and any models error fails the probe. docs/secret-storage.md:158-159 and validity()'s docstring still list unreachable (now produced only by a line with neither a status nor an error). Either fail only a status that comes with an error, keeping unreachable, or update the vocabulary. The exit status is unchanged.
  • The child's own reason code is lost: it exits 1 whenever it reports a top-level error, and the runner returns that code, so the report says probe_exit_1 where it said unexpected_RuntimeError. The probe still fails.
  • A killed probe child shows probe_exit_137, not probe_signal_9: the credential runner returns 128+N for a signalled command, so probe_signal_<n> appears only when the runner itself is signalled.
  • The fail-first account at 445c61c1 in the record: state which assertions fail first.
  • docs/secret-storage.md:2767 cites tests/test_credential_run.py:67,280-309; the two-line ANTHROPIC_IDS insertion moved those targets to :69 and :282-311.
  • SOTA sources: add CPython 3.13.16 Lib/subprocess.py:484 (it is cited in the fix section only; added below).
  • Not caused by this PR: blueprints/runtime-workers/openhands/resolver/patch_policy.py:137 cites docs/secret-storage.md:199-200 for core.hooksPath, already wrong at the base.
• From the command center's Claude read at 28c99ca9 (PASS_WITH_P3); the local follow-up branch claude/api-actions-key-slots-followup-20261010 (ea7f3638) carries the first, second and fourth, to open as its own PR after this lands:
  • anthropic-api-6 described as empty in docs/secret-storage.md and rule 11 (the record was fixed at 16:04Z);
  • the docs omit the planned CI fallback to anthropic-api-4 on credit exhaustion;
  • the port drops the admin-key probe (stated in its docstring);
  • a 429 makes the tool exit 1, so a daily timer would alarm on rate limiting;
  • the redirect test mocks build_opener.
• From the GPT read at 445c61c1:
  • Retain the primary sources the decision record cites by hash (four Anthropic documentation captures, the private key-practice record and the command center's script), or cite retained sources instead.
  • Fill-first switching, recovery and the CI secret's rotation belong to the separately scoped launcher and timer work. Test them there against the stated order; this PR adds no key selector.
  • Existing machine-name labels in the catalog, docs/harness-defaults.md and the manifest: normalize them when those records are next maintained.
• From the GPT delta read at 295c3e51 (its P2, a stale base line in an earlier capture of this body, was already fixed by the 02:49Z edit; its P3s):
  • The cited provider captures and private inputs are still not retained (the first item above): keep the dependent provider and account claims marked unverified until a capture or a sanitized source is retained.
  • The check step still accepts a killed "missing" check: a slot check that prints missing (not found) and ends with a signal or an unexpected code becomes not stored, and the tool exits 0. Accept a missing store only with the runner's expected return code, and add a regression for a missing line followed by exit 7 or a signal.
  • Fill-first switching, recovery and CI rotation (the second item above): test exhaustion, revocation, restart, retry and recovery against the declared order in the launcher and timer work, including replacing the CI secret and checking health before a revocation.
  • Machine-name labels (the third item above): 56 occurrences on 54 existing lines, none in the patch.
  • Pin the inventory-validator correction in the record: cite scripts/credential_status.py:109 and tools/credentials/credential_run.py:167,172 at 86fa38442, next to the fail-first evidence.

Order with #968: this PR's docs describe the claude-review environment secret that #968 introduces. Land #968 first.

SOTA sources

• Anthropic Admin API, https://platform.claude.com/docs/en/manage-claude/admin-api (read 2026-10-10): "The Admin API is unavailable for individual accounts."; /v1/organizations/me under "Accessing organization info".
• Anthropic Usage and Cost API, https://platform.claude.com/docs/en/manage-claude/usage-cost-api (read 2026-10-10): the same availability sentence.
• Anthropic prompt caching, "Cache storage and sharing", https://platform.claude.com/docs/en/build-with-claude/prompt-caching#cache-storage-and-sharing (read 2026-10-10): "Caches are isolated between organizations."
• Claude API overview, "Response headers", https://platform.claude.com/docs/en/api/overview (read 2026-10-10): anthropic-organization-id.
• python/[email protected] Lib/urllib/request.py#L623-L656: a redirect copies the request headers to the new URL, which is why the probe follows none.
• CPython 3.13.16 Lib/subprocess.py:484 (python/cpython@cbc944f4, and the installed file, sha256 3418c49f…): "returncode: The exit code of the process, negative for signals." (the probe's return-code check).
• Add two optional direct Anthropic Console key stores #930 (80cc49d53), the slot pattern; tools/credentials/alpaca_rate_limit_probe.py (no redirect, no body read), the HTTP-probe pattern.

Evidence-class table

Local commands run

$ TMPDIR=/var/tmp/... python3 -m unittest tests.test_credential_run tests.test_anthropic_key_health tests.test_evidence_manifest tests.test_upstream_surface_watch tests.test_secret_path_guard tests.test_credential_tools tests.test_credential_status
Ran 403 tests, OK (skipped=8)        # at 28c99ca9, rerun by the coordinator
$ python3 scripts/validate.py
exit 0 (11407 hashed files)
$ gitleaks git --log-opts="e1c88ff3d..HEAD" --config .gitleaks.toml --redact
no leaks found

At 445c61c16 (the first rebase, on 8c95a50d1):

$ python3 scripts/validate.py
exit 0
$ python3 scripts/validate_shards.py run --shard N --shards 8   # N = 0..7, in parallel, as validate.yml runs them
5 shards OK; shards 1, 2 and 7 failed in three modules, each then rerun alone on this head and on unchanged main 8c95a50d:
  tests.test_new_wsl_mcp_conformance_lifecycle (2 errors under load): 9 OK on the head and on main
  tests.test_landscape_sweep_harness (1 failure under load): 289 OK (3 skips) on the head and on main
  tests.test_adaptive_paper_credential_race: the same 5 failures on the head and on main
$ gitleaks git --log-opts="$(git merge-base origin/main HEAD)..HEAD"
no leaks found

The builder also ran test_canary_proof, test_codex_worker_lane, test_credential_boot_receipt, test_host_requests, test_new_wsl_definitive_defaults and test_currency_due; all OK.

At 5ca54b8c (the fix commit before the 02:12Z rebase; the tool and its tests are byte-identical at 295c3e51):

$ python3 -m unittest tests.test_anthropic_key_health tests.test_credential_run
Ran 114 tests, OK
$ python3 -m unittest tests.test_anthropic_key_health   # the new test file on 445c61c1's tool
FAILED (failures=8)
$ python3 -m unittest tests.test_upstream_surface_watch
Ran 114 tests, OK (skipped=6)
$ python3 scripts/validate.py
exit 0
$ python3 scripts/validate_shards.py run --shard N --shards 8   # N = 0..7
7 shards OK; shard 4: one failure, test_secret_path_guard.test_host_profile_copy_is_verbatim, which compares this host's
installed guard (updated to main ccff8765) with the branch's copy; this PR changes neither
$ gitleaks git --log-opts="445c61c16..HEAD"
no leaks found

At the head 295c3e51 (2026-10-12, on 86fa38442):

$ python3 scripts/validate.py
exit 0 (11692 hashed files)
$ python3 -m unittest tests.test_anthropic_key_health tests.test_credential_run tests.test_credential_status tests.test_upstream_surface_watch tests.test_evidence_manifest tests.test_credential_tools
Ran 318 tests, OK (skipped=6)
$ python3 -m unittest tests.test_upstream_surface_watch   # at each of the 8 rebased commits
rc 0 at each
$ gitleaks git --log-opts="$(git merge-base origin/main HEAD)..HEAD"
no leaks found
GitHub CI at 295c3e51: validate, validate-shard-0 to -7 and sota-sources pass (02:33Z)

At the head 5c5bff1e (2026-10-12 04:13Z, on 8b1ef0c37; the registry-only rebase of 295c3e51):

$ git show <commit> -- . ':!manifests/evidence.json' | git patch-id --stable   # each of the 8 commits, and the range diff
identical to 295c3e51's (8 of 8, and the range)
$ python3 scripts/validate.py
exit 0 (11741 hashed files)
$ python3 scripts/evidence_manifest.py --check
exit 0 (11741 files); rows outside this PR's 8 files equal main's
$ python3 -m unittest tests.test_anthropic_key_health tests.test_credential_run tests.test_evidence_manifest tests.test_upstream_surface_watch tests.test_validate
Ran 321 tests, OK (skipped=6)
$ python3 -m unittest tests.test_credential_status tests.test_credential_tools
Ran 85 tests, OK
$ gitleaks git --log-opts="origin/main..HEAD"
no leaks found

Decision record

docs/decisions/2026-10-10-anthropic-key-slots-and-practice.md.

🤖 Generated with Claude Code