Skip to content

Add a pre-cue read by Anthropic's pr-review-toolkit agents in Actions - #909

Merged
seathatflowsinourveins merged 6 commits into
mainfrom
claude/api-actions-w4f-toolkit-review-20261009
Oct 9, 2026
Merged

seathatflowsinourveins merged 6 commits into
mainfrom
claude/api-actions-w4f-toolkit-review-20261009

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Scope

  • What this PR changes, in one or two sentences: a new workflow, claude-pr-toolkit-review.yml, runs two of Anthropic's pr-review-toolkit agents, pr-test-analyzer and silent-failure-hunter, on one pull request head through anthropics/claude-code-action v1.0.247. Both list every behaviour and test change the description does not declare. A maintainer dispatches it by hand; it is the command center's pre-cue step J8 (W4 draft f). The toolkit comes from anthropics/claude-code at a full commit, verified file by file and loaded with --plugin-dir; the action's unpinned plugins and plugin_marketplaces inputs are not used. It runs only while CLAUDE_PR_TOOLKIT_ENABLED is true, and only on the first attempt of a run, so merging this starts nothing.
  • Model, effort and caps (R5): --model claude-opus-5-5 at --effort max, by the owner's direction, relayed by the command center on 2026-10-09, that every Claude Action runs on Opus 5.5 at max. The command center's caps of the same day: --max-turns 12 for the coordinator (unchanged), --max-budget-usd 22, and a cost bound of $24.20 in the numbers step (the budget × 1.10). The expected cost of a run is about $6.6 to $10.9. The J8 runs this PR cites ran on Sonnet 5.5 at a $5 budget; no run on Opus 5.5 or at $22 exists yet.
  • R4, debug logging set as a repository secret or variable (2026-10-09; the cc-native-practice lane's L3 read against anthropics/claude-code-action v1.0.247 practice): the guard binds (secrets.ACTIONS_STEP_DEBUG || vars.ACTIONS_STEP_DEBUG) == 'true' and (secrets.ACTIONS_RUNNER_DEBUG || vars.ACTIONS_RUNNER_DEBUG) == 'true' as booleans and refuses either before any token is requested; a test pins both bindings, and six weakened copies each fail at least one test. This is hardening: the action step already pins ACTIONS_STEP_DEBUG: 'false', which is what the action's full-output switch reads. zizmor's auditor persona reports secrets-outside-env on the two reads; CI runs the regular persona with --no-ignores, which reports nothing. At R4 these lines were byte-identical in Bound the harness audit and move it to claude-code-action v1.0.247 #892, Add an on-demand, read-only pull request review by the Claude action #894, Add an on-demand Claude security review and a model-free flag on risky paths #895, Add a weekly lane triage: the model proposes, a model-free job applies #896 and Add a pre-cue read by Anthropic's pr-review-toolkit agents in Actions #909; R5 leaves Add a pre-cue read by Anthropic's pr-review-toolkit agents in Actions #909's unchanged.
  • The report notice at the cap (2026-10-09; the J8 micro read of Bound the harness audit and move it to claude-code-action v1.0.247 #892): the publish step wrote with jq -r, which adds a newline, so a 60,000-byte report was announced as cut. It writes with jq -j now; a test publishes exactly 60,000 and 60,001 bytes, and a copy that writes with jq -r fails it.
  • R5 (2026-10-09; from the command center's security read of b80a6eb8, two later audits, a GPT designated read of Add an on-demand Claude security review and a model-free flag on risky paths #895 and the caps decision):
    • Every result record. The numbers step (now id: numbers) reads every result record in the execution file, not only the last, so an idle success at the end cannot hide an earlier error. Each record must be a success without is_error, or a budget stop (error_max_budget_usd) after both agents handed back; any other error subtype fails. The cost is the highest any record reports, and models comes from that record.
    • Cost bound. $24.20, the $22 budget × 1.10. The factor is the largest settled overrun among J8's $5-budget runs (the command center's figure: Add an on-demand Claude security review and a model-free flag on risky paths #895, $5.46 on $5, 9.2%), rounded up, and it is re-derived after this workflow's first three runs. A run above the bound fails, budget stop or not. A budget stop at or under it, after both handbacks, passes and publishes both reports. The job summary names a budget stop and an overrun.
    • Publishing. The publish step is gated on steps.numbers.outcome == 'success' instead of success(), because the action fails its own step on a first result that is not a plain success.
    • Hosted execution file. The pinned action stops reading at the first result record. In J8 Bound the harness audit and move it to claude-code-action v1.0.247 #892 and Add an on-demand, read-only pull request review by the Claude action #894, read from the client directly, both handbacks came before that record, and the step passes on both streams cut there. Whether the action's SDK path also waits for the background agents is unmeasured. If it does not, the file ends before any handback, and the run fails closed with 0 of 2 agent reports and publishes nothing. The first hosted runs measure it.
    • No result record. A run with no usable result record keeps a lower bound of its assistant usage, each message id counted once, and fails. On J8 Add an on-demand, read-only pull request review by the Claude action #894's stream these sums equal the result's input and cache tokens and undercount output. Replayed on J8 docs(trading): revise R3 census to 2018-2023 #900's stream (no result record, one handback), the step took this path and failed.
    • Agents called. The coordinator must have called both toolkit agents, each at least once (a retry passes), and no other agent. Replayed on three J8 streams of 2026-10-09 (Bound the harness audit and move it to claude-code-action v1.0.247 #892, docs(trading): revise R3 census to 2018-2023 #900 and Add an on-demand, read-only pull request review by the Claude action #894), the bound passed on all three.
    • Tool list. An entry that is not a string is a forbidden tool, named non-string tool entry, instead of being dropped.
    • New usage.json fields: complete, agents_called, handbacks, result_subtypes and lower_bound_models.
    • Link removal. A new model-free step, Remove symbolic links from the pull request head, runs find pr-head -path pr-head/.git -prune -o -type l -exec rm -f {} + right after the head checkout.
    • Tests.
      • Exact pins on every claude_args line and the whole --settings JSON.
      • The toolkit test now reaches the hooks, MCP and scripts refusal, using the synthetic files' own hashes.
      • The cap test asserts the published prefix on the iconv path.
      • A five-record replay of J8 Add an on-demand, read-only pull request review by the Claude action #894's shape (synthetic text) in its measured order, plus the unmeasured early-result order as a separate fail-closed case.
      • The module goes from 41 to 56 tests, and 36 weakened copies of the workflow each fail at least one test.
    • Record.
      • It names every R5 change and realigns its old lines 161-162 with the publish step's comment.
      • It says what the subagents run on: model: inherit, so Opus 5.5; nested agents run on the model their own call names, and pinning that is follow-up F-08.
      • It adds the shared origin-rewrite sentence and the follow-ups line.
      • docs/github-automation.md now says Opus 5.5 and $22.
    • Not in R5. Filed as follow-ups before any enabling variable is set: transcript tool scanning, subagent model and Agent-tool pins, debug-value widening, tool-list shape beyond non-string entries, extra deny rules, and token-source and guard-step assertions.
  • Base commit: 72e1f943
  • Head at R5: 7aa2c128
  • Lane: lane:foundation
  • Owned paths touched: .github/workflows/claude-pr-toolkit-review.yml (new), tests/test_claude_pr_toolkit_review_workflow.py (new), tests/test_workflow_policy.py (one inventory entry, one exemption), tests/test_workflow_security_coverage.py (coverage set, offline zizmor test), docs/decisions/2026-10-09-claude-actions-pr-toolkit-review.md (new), docs/decisions/2026-10-04-ci-least-privilege.md (dated addendum), docs/github-automation.md (one entry), manifests/evidence.json

R6 (2026-10-09), at 8d5e7fe

  • Both handbacks required before anything publishes. This fixes the GPT designated read of 7aa2c12 (P2,
    reproduced). The numbers step fails with <n> of 2 agent reports handed back unless both called agents handed back:
    • a handback counts only as a SubagentHandback under the coordinator's Agent call for that agent, once per agent;
    • the bound that counted two headings in the relay is gone, so relay headings never count as completion;
    • a new test covers six partial runs, all of which must fail and publish nothing: headings alone, headings with
      pending text, two calls with no handback, one handback with a headed relay, one handback carrying the other
      agent's heading, and one agent's two handbacks after a retry.
  • What is published. The report step publishes the two agents' own handbacks, pr-test-analyzer's then
    silent-failure-hunter's, never the coordinator's relay. A relay is model text and can be a template or a paraphrase.
    • This reverses R3's relay preference, and R3's tests now enforce the new behaviour.
    • The coordinator prompt no longer asks for a relay. It ends with the single line "Both reports handed back.",
      which saves the relay's Opus output tokens on every run. The $24.20 cost bound stays as an upper bound.
  • Publish gate: !cancelled() && steps.numbers.outcome == 'success', matching the other W4 workflows, so a
    cancelled run publishes nothing.
  • Settings pin: the exact --settings pin compares canonical JSON, so 1 or 0 no longer passes where true or
    false is pinned (J8 R5 micro read of Add an on-demand, read-only pull request review by the Claude action #894, N1).
  • Cost factor: the record adopts the shared R6 paragraph. The bound is the budget × 1.10, from four measured J8 runs
    on a $5 budget ($5.46, $5.33, $5.16, $5.0007), re-derived after this workflow's first three hosted runs: $22 → $24.20.
  • Declared: the record's R6 section now declares the eight R5 behaviour and test changes that the J8 R5 micro read
    (b80a6eb..7aa2c12, Opus 5.5 at max) found undeclared.
  • Local checks at 8d5e7fe:
    • validate.py and evidence_manifest.py --check pass (10,904 files);
    • the six policy and hardening modules: 275 tests, OK, 2 skips outside this PR;
    • this module: 57 tests, with and without PyYAML;
    • actionlint and zizmor (pedantic, and regular with --no-ignores) are clean;
    • 49 of 49 weakened copies of the workflow fail at least one test.
  • Hosted checks at 8d5e7fe: not read yet.

SOTA sources

  • anthropics/claude-code-action v1.0.247 at 2dca132ff0e0c4094ce6048b422c6915a071210b:
    • action.yml: the federation inputs, claude_args, plugins and plugin_marketplaces (lines 160-167), display_report and track_progress;
    • base-action/src/parse-sdk-options.ts: --plugin-dir is passed through to the CLI.
    • base-action/src/run-claude-sdk.ts (R5), read at v1.0.246 (38c80c1), a file the v1.0.247 compare leaves unchanged:
      • the loop that stops reading at the first result record (lines 190-209);
      • the messages written on an SDK error (lines 212-216);
      • the step's conclusion (lines 236-298).
    • src/entrypoints/run.ts (R5): setExecutionFileOutputIfPresent in its catch.
    • src/github/operations/git-config.ts:129-134, called from src/modes/agent/index.ts:52-61 (R5): in agent mode the action rewrites the checkout's origin URL with the job token.
    • The releases API on 2026-10-09 lists v1.0.247 as the latest release.
  • anthropics/claude-code at 602df92bf481ed904533e95c09f740f40aab5aed, plugins/pr-review-toolkit. The two agent files and plugin.json match GitHub's git blob ids at that commit.
  • Claude Code 2.1.295 claude --help: --plugin-dir, --tools, --restricted, --permission-prompts, --effort and --max-budget-usd. The headless documentation: background subagents and CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS.
  • actions/checkout v7.0.1 (repository, ref, path and sparse-checkout), and step-level timeout-minutes with the steps context. The publish gate's steps.<id>.outcome == 'success' follows the pattern validate.yml and native-offhost-app-state.yml already use (R5).
  • openai/codex-action v1.13 at bdf19a4a223ec2549a3e2274a0cf61556bc07675, action.yml inputs. This backs the record's WATCH entry; the action is not adopted.

Evidence-class table

Claim Evidence class Command / receipt
On 2.1.295, these agents with these settings find undeclared changes before the designated reads native_proven (installed client, billed to a second key; not through the action) J8 runs on #892 (4d0a14d): 6 undeclared behaviour changes, $4.28. On #900 (f082159): 2 high-severity verifier defects (filed as R3-F2). Both on Sonnet 5.5 at a $5 budget; no run on Opus 5.5 or at $22
The workflow's guard, binding, link removal, diff, toolkit check, numbers and report steps behave as described synthetic tests/test_claude_pr_toolkit_review_workflow.py, 56 tests at 7aa2c12. Each step's shell is taken from the workflow file and run; the tests run without PyYAML too
The reports survive a coordinator that stops before relaying synthetic (from the measured J8 #894 stop) The numbers and report steps fall back to each agent's SubagentHandback message when the relay lacks a section; 3 tests, plus R5's budget-stop replays
Every result record is read. A budget stop passes only after both handbacks and at or under $24.20, and the summary names a stop and an overrun. models comes from the costliest record synthetic test_in_the_measured_order_a_budget_stop_after_both_handbacks_passes_wherever_the_file_ends, test_a_budget_stop_after_both_handbacks_publishes_up_to_the_cost_bound ($22.40 and $24.20 pass; $24.21 fails), test_a_budget_stop_without_both_handbacks_and_any_other_error_record_fail, test_the_cost_bound_is_the_budget_times_the_measured_overrun_factor, test_the_model_usage_is_the_costliest_result_records
The pinned action's execution file ends at the first result record source_review base-action/src/run-claude-sdk.ts lines 190-209 at v1.0.246, unchanged in the v1.0.247 compare
In J8 #892 and #894 both handbacks come before the first result record, and the step passes on both streams whole and cut there. The agents bound passes on all three streams. #900 (no result record) takes the lower-bound path and fails local_integration (the step's shell replayed on J8 streams recorded on 2026-10-09; the streams stay outside the repository) Counting records from zero: #894 handbacks 563 and 580, first result 598 (already at the run's $5.0007); #892 handbacks 303 and 323, first result 347
The assistant-usage sums are a lower bound local_integration On J8 #894's stream, input 148, cache read 8,370,121 and cache write 484,977 equal the result's model usage; output 448 against 323,641
The cost bound's factor, 1.10 native_proven (J8 receipts, as the command center reports them) The command center's decision of 2026-10-09: J8's settled $5-budget overruns were $5.46 (#895, 9.2%), $5.33, $5.16 and $5.0007. Only #894's $5.0007 was checked against its stream in R5
The tests detect a weakened workflow synthetic Before R5: 11 mutated copies, all 11 fail at least one test (handback fallback ignored, wrong handback tool name, handbacks preferred over a complete relay, report-sections bound removed, turns counting the agents, Bash allowed, no time scaling, no hash check, toolkit ref moved to main, runtime plugins input added, a $70 cost bound). At R5: 36 weakened copies, all 36 fail at least one test (mutant script kept outside the repository)
The workflow is well formed and has no audit finding local_integration actionlint 1.17.0 exits 0. zizmor 1.30.1, offline, pedantic and regular with --no-ignores: no findings (2 suppressed: the R4 guard's secrets-outside-env, auditor persona only, unchanged since R4)
The repository's policy tests accept the change local_integration 274 tests across six modules at 7aa2c12, below
A hosted run authenticates, stays in bounds and reads the cache; the action's SDK path waits for the background agents before its first result record not established No hosted run. Activation is an optional owner step (a federation rule in the paying organization); the same agents already run locally on the second key. If the SDK path does not wait, a run fails closed with 0 of 2 agent reports and publishes nothing

Local commands run

$ uv run --quiet --no-project --python 3.12 --with pyyaml==6.0.3 python -m unittest tests.test_claude_pr_toolkit_review_workflow tests.test_workflow_policy tests.test_workflow_security_coverage tests.test_workflow_hardening tests.test_github_automation_practice tests.test_adoption_docs_consistency
Ran 274 tests ... OK (skipped=2)   # at 7aa2c128; both skips are outside this module
$ uv run --quiet --no-project --python 3.12 python -m unittest tests.test_claude_pr_toolkit_review_workflow
Ran 56 tests ... OK   # without PyYAML, at 7aa2c128
$ actionlint .github/workflows/claude-pr-toolkit-review.yml
exit 0
$ zizmor --offline --no-config --persona=pedantic .github/workflows/claude-pr-toolkit-review.yml
No findings to report. Good job! (2 suppressed)
$ zizmor --offline --no-config --no-ignores --persona regular .github/workflows/claude-pr-toolkit-review.yml
No findings to report. Good job! (2 suppressed)
$ python3 scripts/evidence_manifest.py --check
{"files": 10904, "status": "passed"}   exit 0
$ python3 scripts/validate.py            # at 7aa2c128
{"components": 70, "hashed_files": 10904, "profiles": 4, "receipts": 239, "status": "passed"}   exit 0

Decision record

docs/decisions/2026-10-09-claude-actions-pr-toolkit-review.md: why the pinned checkout is used instead of the plugins input, the behaviour by name, the bounds, the cost of one run with the caps' derivation and the 1.10 factor, the alternatives (including openai/codex-action as WATCH with its overturn condition) and what would overturn it. Its dated sections name every change of R4, of the cap notice and of R5 ("Round 5: result records, caps, link removal and exact pins (2026-10-09)").

Host evidence

Not applicable: no file under evidence/hosts/ changes.

Checklist

  • New/changed GitHub Actions are pinned to a full commit SHA with a version comment (no floating tags). The toolkit is checked out at a full commit, and its files are verified by SHA-256.
  • New/changed workflows declare top-level permissions: {} and grant each job only what it needs (contents: read, pull-requests: read, id-token: write for federation, reviewed in the inventory).
  • No secrets are printed, logged or committed; no new required secret was added (federation, no key).
  • No new paid hosting, subscription or billing surface was introduced. The per-run cap rises from $5 to $22 within the same federated organization (command center caps, 2026-10-09).
  • Peer-owned untracked files and worktrees were preserved (not deleted, moved or overwritten).

🤖 Generated with Claude Code

@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Oct 9, 2026
@socket-security

socket-security Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Dependency limit exceeded — report not shown.

This pull request scan exceeded the 10,000-dependency limit applied to this scan, so the results are incomplete and may be inaccurate. To avoid reporting false positives, Socket has not posted a report.

Upgrade your plan to raise the dependency limit and get complete reports, or view the partial scan in the dashboard.

Socket is always free for open source. If this is a non-commercial open source project, contact us to request a free Team account.

seathatflowsinourveins added a commit that referenced this pull request Oct 9, 2026
…tries and raise the caps to 30 turns and $5

R5, quality and correctness only. A model-free step removes every symbolic
link from the pull request head right after its checkout, so a link to a
process environment file or to ../.git/config cannot carry credentials into
the job summary (GPT designated read of #909, P2; the command center's
security read). The numbers step counts every tool-list entry that is not a
string as a forbidden tool, named "non-string tool entry", instead of
dropping it (GPT designated read of this PR at f5f1fd2, P2). New tests pin
the whole claude_args list and the whole --settings JSON. The caps follow
the command center's measurement-based derivation: 30 turns and a $5 client
budget. The record gains a dated R5 section, corrects its evidence class,
withdraws the earlier dry cost estimate and states which tests run without
PyYAML.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Oct 9, 2026
…894)

### Scope

- What this PR changes, in one or two sentences: adds `claude-pr-review.yml`, an on-demand, read-only review of one same-repository pull request head by `anthropics/claude-code-action` v1.0.247. A maintainer dispatches it from `main` with the pull request number and the exact head commit; the review goes to the job summary and nothing is posted to the pull request. It runs only while the repository variable `CLAUDE_PR_REVIEW_ENABLED` is `true`, so merging this starts nothing.
- Effort (added 2026-10-08 at head a6d2379): `--effort max` in `claude_args`, because judgment work (designated reads, adjudication, pull request and security reviews, audits) runs at `max` under the command center's effort mapping, and every job records its level and reason. The level has to be in `claude_args`: `--restricted` ignores the settings files that would otherwise carry it, and Opus 5.5 runs at `medium` on the API when a request leaves effort unset. The bounds test now requires the flag; removing it fails the test. A loopback dry run of the installed 2.1.295 client with this workflow's `claude_args` sent `output_config.effort: "max"` on every request.
- Fixes from the pre-cue read of the sibling #892 (2026-10-09, head d6f8909): the record names every behaviour of this new workflow: the trigger; the slug, owner, first-attempt and variable guards; the timeout and grants; the guard and binding exits; the head-as-data checkout and diff limits; the explicit action inputs `display_report` and `track_progress` (defaults in the pinned `action.yml`); and the 14-day usage artifact. The bounds step now uses an allow-list, fails on a start record without lists or on an empty result, and names each unmet bound. The review cut is character-safe and says when it cut. The step tests no longer skip without PyYAML. The 2026-10-03 cooldown waiver is cited.
- Fixes from this PR's own pre-cue toolkit read (J8 on d6f8909, head 8e64120):
  - the empty paths array is now expanded safely on bash before 4.4;
  - a success without an execution file now fails the job;
  - `display_report` and `track_progress` are asserted;
  - the artifact name and the action-pin sentence are corrected, and the parity receipt's dependency on #892 is stated;
  - `claude-pr-review.yml:review` is now in both grant tables of ci-least-privilege;
  - the record states that a public run's summary is public.
- Pre-cue toolkit read of this PR (J8 on d6f8909, 2026-10-09; `pr-test-analyzer` and `silent-failure-hunter`): every finding at confidence 80 or above, and its status at head 41c9c15.

  | Finding | Status at 41c9c15 | Fixing commit and evidence |
  | --- | --- | --- |
  | `pr-test-analyzer` 1 (90): the record names the wrong artifact | fixed | 8e64120: the record names `claude-pr-review-usage-<run id>-<attempt>`, the upload's name; 41c9c15 adds `test_the_usage_artifact_is_named_for_the_run_and_its_attempt` |
  | `silent-failure-hunter` 2 (88): the record and the workflow disagree on the artifact | fixed | the same two commits |
  | `pr-test-analyzer` 2 (85): `docs/github-automation.md` claims the harness audit's action pin | fixed | 8e64120: the bullet says the two pins differ until #892 lands |
  | `silent-failure-hunter` 1 (92): the same false claim (pin and federation variables) | fixed | 8e64120, as above; the four federation variables are the same |
  | `pr-test-analyzer` 3 (85): the headline evidence file is not in this PR | fixed | 8e64120 states that #892 adds it and lands first; 41c9c15 cites its `LR` entry field by field |
  | `silent-failure-hunter` 3 (80): "#897" and "two blocking findings" have no source in that receipt | fixed | #892 adds `target_pull_request`, `target_head`, `report_blocking_findings` and `report_sha256` to the `LR` entry (a marked 2026-10-09 amendment from the run's harness receipt); 41c9c15 cites them, calls the verdict the model's own, and states that the prompt and `claude_args` are unchanged from the run's workflow commit a6d2379 to that head (R5 later raises the two caps and the prompt's turn sentence; see R5 below) |
- R4, debug logging set as a repository secret or variable (2026-10-09; the cc-native-practice lane's L3 read against `anthropics/claude-code-action` v1.0.247 practice): the guard binds `(secrets.ACTIONS_STEP_DEBUG || vars.ACTIONS_STEP_DEBUG) == 'true'` and `(secrets.ACTIONS_RUNNER_DEBUG || vars.ACTIONS_RUNNER_DEBUG) == 'true'` as booleans and refuses either before any token is requested; a test pins both bindings, and six weakened copies each fail at least one test. This is hardening: the action step already pins `ACTIONS_STEP_DEBUG: 'false'`, which is what the action's full-output switch reads. zizmor's auditor persona reports `secrets-outside-env` on the two reads; CI runs the regular persona with `--no-ignores`, which reports nothing. The R4 lines are byte-identical in #892, #894, #895, #896 and #909; R5 leaves them unchanged here.
- The report notice at the cap (2026-10-09; the J8 micro read of #892): the publish step wrote with `jq -r`, which adds a newline, so a 60,000-byte report was announced as cut. It writes with `jq -j` now; a test publishes exactly 60,000 and 60,001 bytes, and a copy that writes with `jq -r` fails it. R5 adds the 59,999-byte case.
- From the J8 micro read of 41c9c15 (2026-10-09; 4 of 6 listed findings fixed, PTA-3 and SFH-3 partly because the receipt the record cites was not in this tree): this PR now carries a byte-identical copy of `local-parity-receipt.json` (sha256 `ba498b82…`, the file #892 adds), so the citation no longer depends on which PR lands first, and the docs bullet states the action pin and the federation variables instead of comparing with the harness audit on `main`.
- R5 (2026-10-09, head d1bcb6c, limited by the command center to correctness and test quality) adds a model-free step, `Remove symbolic links from the pull request head`, right after the head checkout: `set -euo pipefail; find pr-head -path pr-head/.git -prune -o -type l -exec rm -f {} +`. It raises the approved caps from measurement: `--max-turns` 12→30 and `--max-budget-usd` 3→5. The prompt's "You have at most 30 turns", the numbers step's bounds (1 to 30 assistant turns, at most $5, with matching messages), the header comment and `docs/github-automation.md` follow. The numbers step now counts each non-string tool-list entry as a forbidden tool named `non-string tool entry` instead of dropping it. The R4 guard lines, the deny list and the `github_token` input are kept exactly as at a1bc3bc. The test module goes from 33 to 41 tests. Eight are new: the symlink step's position and exact text; its behaviour on planted links to `/proc/self/environ`, `../.git/config` and a subdirectory, plus a re-run on a link-free head; a 59,999-byte report published whole; object, null and number tool entries refused and named; the prompt stating the `--max-turns` value; and the caps at 30 turns and $5 and just above. Changed tests pin the full `claude_args` list and `--settings` JSON exactly, assert the exact `if:` strings of the publish and numbers steps, run the guard with production's debug-off values (false/false/empty), and check that the 59,999-byte prefix before a cut character survives. All 28 weakened copies of the workflow fail at least one test with PyYAML present, as the hosted validate job has it; without PyYAML 15 of 28 do, because the pins sit in the PyYAML-only shape tests. The decision record gains a dated R5 section and names the R4 guard change in its what-changed paragraph. It corrects "Read denies for every `.git` directory" to "for the files under every `.git` directory". It restates the cap derivation and the expected cost per run, and withdraws the dry $0.70–$1.30 estimate, which was below the measured first-read cost. It adds the shared follow-up line, the shared `origin` URL sentence and two action sources. `manifests/evidence.json` is rehashed for the four changed files.
  - Sources of the R5 items: the command center's security read of a1bc3bc (the symbolic links, the closed flag list, the exact `if:` strings, the guard's debug-off values, the 59,999-byte prefix, and the stale guard sentence in the record); the command center's caps decision of 2026-10-09; the GPT designated read of #895, P2 (non-string tool entries, applied to all five W4 workflows); the cc-native-practice lane's third audit, P3-1 (the shared `origin` URL sentence).
  - Caps derivation: `LR`, a first read of #897 with this workflow's prompt and `claude_args` on the installed client, used all 12 of its 12 assistant turns and cost $2.23 ($1.87 by the client's estimate). So the need is above 12, and 2.5 times that floor gives 30 turns; twice $2.23 is about $4.5, rounded up to $5. Four delta re-reads (LR2 to LR5) used 5, 6, 7 and 6 turns and cost $1.75, $1.35, $1.54 and $0.73. A run is expected to cost about $1.9 to $2.3 for a first read and $0.7 to $1.8 for a delta re-read.
  - Filed as follow-ups before any enabling variable is set, not in R5: the security hardening from the 2026-10-09 read (debug-value widening, the tool-list shape apart from the non-string entry above, extra deny rules, token-source and guard-step assertions); the rest of F-02 (a non-empty tool list, checking the `tool_use` names); F-16 (the pins that run only with PyYAML).
  - Not run in R5: a fence smoke with a planted symbolic link, which needs a model run; any hosted run. On the first run, watch whether the action appends an `--mcp-config` that `--strict-mcp-config` still honours (the read's G3); the numbers step fails the run on any MCP server.
- Base commit: `aba02ec3` (merge base at a1bc3bc and at d1bcb6c, which builds on a1bc3bc; as read at a1bc3bc, main had moved to `ebe6c223` with #890, #900, #897 and #907; only `manifests/evidence.json` overlaps, and the one rebase happens at landing)
- Lane: `lane:foundation`
- Owned paths touched: `.github/workflows/claude-pr-review.yml` (new), `tests/test_claude_pr_review_workflow.py` (new), `tests/test_workflow_policy.py` (one inventory entry, one exemption), `tests/test_workflow_security_coverage.py` (coverage set and one zizmor test), `docs/decisions/2026-10-08-claude-actions-pr-review.md` (new), `docs/decisions/2026-10-04-ci-least-privilege.md` (dated addendum only), `docs/github-automation.md` (current-practice bullet), `evidence/artifacts/claude-actions-fence-smoke-20261008/receipt.json` (new), `evidence/artifacts/claude-actions-fence-smoke-20261008/local-parity-receipt.json` (new; byte-identical to the copy #892 adds), `manifests/evidence.json` (registration, last commit).

### R6 (2026-10-09), at bd10ba3

- **Cost bound:** the budget × 1.10 = $5.50, from four measured J8 runs on a $5 budget ($5.46, $5.33, $5.16, $5.0007),
  re-derived after the first three hosted runs.
  - A budget stop at or under it publishes the text written before the stop, with a line naming the stop.
  - Publishing is gated on `!cancelled() && steps.numbers.outcome == 'success'`, because the action fails its own step on
    any budget stop.
- **`--max-turns 30` removed:** the pinned action (`base-action/src/run-claude-sdk.ts:241-250` at 2dca132f, since
  6ef6450f) fails a success whose `num_turns` exceeds it.
  - On Claude Code 2.1.295, `num_turns` counts transcript messages: 57 for 12 requests under `--max-turns 12`. That is a
    code reading plus a local measurement, not a hosted run.
  - The numbers step's 30-assistant-turn bound and the prompt's "at most 30 turns" stay.
- **Job timeout:** 20 → 30 minutes. 30 turns at the measured pace (12 turns in 512,670 ms) take about 21.4 minutes before
  setup. #892 uses 30.
- **Settings pins:** both compare canonical JSON, so `1` no longer passes for `true` (J8 R5 micro, N1, confidence 70).
- **Record:**
  - the shared cost paragraph;
  - an upstream-issue draft (not filed);
  - the seven changed test expectations the J8 R5 micro found undeclared (U1–U7).
- **Local checks at bd10ba3:**
  - `validate.py` and the manifest check pass (10,906 files);
  - six modules: 262 tests, OK;
  - this module: 44 tests (16 skip without PyYAML);
  - actionlint and zizmor are clean;
  - 39 of 39 weakened copies fail a test with PyYAML.
- **Hosted checks at bd10ba3:** not read yet.

### SOTA sources

- anthropics/claude-code-action, v1.0.247 at `2dca132ff0e0c4094ce6048b422c6915a071210b` (<https://github.com/anthropics/claude-code-action/tree/2dca132ff0e0c4094ce6048b422c6915a071210b>): `docs/security.md` ("Do not check out an untrusted ref into the workspace root before this action"; base ref at the root and head ref in a subdirectory), `docs/setup.md` (federation inputs; "a static credential takes precedence and federation will not be used"), `action.yml` (inputs and the `execution_file` output), `base-action/src/parse-sdk-options.ts` (argument parsing with `shell-quote`, default setting sources, debug forcing full output), `src/github/operations/git-config.ts` (git authentication in the root checkout; lines 129-134 write the job token into the checkout's `origin` URL when `allowed_non_write_users` is unset) and, new in R5, `src/modes/agent/index.ts` (lines 52-61, where agent mode calls that configuration). R5 read both files at v1.0.246 (`38c80c1`); the command center's compare of v1.0.246 and v1.0.247 changes only the bundled Claude Code version.
- Claude Code 2.1.295, the version the pin installs, `claude --help`: `--restricted` ("removes the built-in tools that run commands or code ... and ignores user, project and local settings files ... Also confines the file tools to the working directories"), `--permission-prompts none` ("anything that would prompt is denied automatically"), `--tools`, `--settings`, `--strict-mcp-config`, `--max-budget-usd` and, for the R5 caps, `--max-turns`. The help does not say whether a symbolic link's target is resolved, which is why R5 removes the links before the model runs.
- GNU findutils `find` 4.10.0 as installed (`find --version`), new in R5: `find pr-head -path pr-head/.git -prune -o -type l -exec rm -f {} +`, measured by the step test on planted links and on a link-free head (exit 0, no `rm` started). The runner image's `find` was not measured.
- GitHub, Contexts reference, `runner.debug` (set only while debug logging is on, so the guard's `RUNNER_DEBUG_SIGNAL` is empty on a normal run), new in R5 as the basis of the guard tests' debug-off values; taken from the command center's read of a1bc3bc, not re-read in R5 (no network).
- GitHub, OpenID Connect reference, "Filtering for pull_request events" and "Filtering for a specific branch" (<https://docs.github.com/en/actions/reference/security/oidc>); GitHub Security Lab, Preventing pwn requests (<https://securitylab.github.com/research/github-actions-preventing-pwn-requests/>).
- Anthropic documentation read 2026-10-08: Workload identity federation (<https://platform.claude.com/docs/en/manage-claude/workload-identity-federation>); How Claude Code uses prompt caching (<https://code.claude.com/docs/en/prompt-caching>); Pricing (<https://platform.claude.com/docs/en/about-claude/pricing>).

### Evidence-class table

| Claim | Evidence class | Command / receipt |
| --- | --- | --- |
| With the fence flags (`--restricted`, `--tools`/`--allowedTools` Read,Glob,Grep, `--strict-mcp-config`, `--permission-prompts none`, `--settings` with hooks off, `claudeMdExcludes` and three deny rules; Haiku 5.5, 10 turns, $0.50, as the receipt lists; this workflow adds `--setting-sources user`, `--add-dir`, `--effort max`, four deny rules and `blockReadsOutsideWorkingDirectories`, a key in the 2.1.295 build that the receipt does not exercise) the session's tools are Glob, Grep and Read; reads outside the tree and of `.git/config` are denied; planted hooks, instruction files and a skill under `pr-head` have no effect | `native_proven` on the installed 2.1.295 client (not through the action, not on a GitHub runner) | `evidence/artifacts/claude-actions-fence-smoke-20261008/receipt.json`, three runs |
| The action's own parser turns this workflow's `claude_args` into those flags and that JSON | `local_integration` (at a1bc3bc only) | the parser's three helpers replayed with `shell-quote` 1.10.0 on the workflow text at a1bc3bc; not re-run at d1bcb6c (no network to install `shell-quote`). R5 changes two plain values (`12` to `30`, `3` to `5`), and `test_claude_has_three_read_tools_and_fixed_bounds` asserts the whole list, split per line with `shlex`, and the `--settings` JSON |
| The guard, binding, symbolic-link, diff, numbers and review steps behave as described | `synthetic` | `tests/test_claude_pr_review_workflow.py`, 41 tests at d1bcb6c (the 26 step tests run without PyYAML too; the 15 shape tests need it); each step's shell is taken from the workflow file and executed against a local stand-in for `gh`, a local git repository or a planted `pr-head` tree |
| Symbolic links under `pr-head` are removed before the model runs, `pr-head/.git` is left as the checkout wrote it, and a link-free head passes | `synthetic` | `test_symbolic_links_in_the_head_are_removed_and_nothing_else` (links to `/proc/self/environ`, `../.git/config` and `../..`; GNU find 4.10.0); no fence smoke with a planted link has run on the client |
| The tests detect a weakened workflow | `synthetic` | 20 mutated copies on 2026-10-08 (that day's bounds), 20 failed at least one test; on 2026-10-09 three turn-bound mutants each failed, and the new bounds have their own tests; at 41c9c15 three artifact-name mutants each pass the 30 earlier tests and fail the new name test; at d1bcb6c 28 mutated copies (the symbolic-link step, the flag and settings pins, the `if:` strings, the guard's debug-off values, the report cap, the non-string tool entries, the caps and the prompt) each fail at least one test with PyYAML 6.0.3 present, as the hosted validate job has it; without PyYAML 15 of 28 do (the other 13 are pins in the shape tests; follow-up F-16) |
| This workflow's own prompt and `claude_args` review a real pull request within bounds at `--effort max` on Opus 5.5 | `local_integration` (installed client, billed to a second key; not through the action) | `evidence/artifacts/claude-actions-fence-smoke-20261008/local-parity-receipt.json`, carried byte-identical in this PR and in #892 (sha256 `ba498b82…`), its `LR` entry: workflow commit a6d2379, whose prompt and `claude_args` are unchanged at d1bcb6c apart from the two caps R5 raised (`--max-turns` 12 to 30, `--max-budget-usd` 3 to 5) and the prompt's turn sentence; #897 at `target_head` 24821c5; 12 of 12 assistant turns; $1.868667 of the then $3 budget; tools Glob/Grep/Read; no MCP; `success`; `report_blocking_findings: 2`, the report's own verdict line, unchecked (`report_sha256` c49bee04, a 2026-10-09 amendment) |
| The caps, 30 assistant turns and $5, bound a runaway at about twice the measured need; a run is expected to cost about $1.9 to $2.3 for a first read and $0.7 to $1.8 for a delta re-read | `local_integration` (installed 2.1.295 client, billed to the second key; not through the action) | the `LR` entry above (12 of 12 turns, client estimate $1.868667); the $2.23 figure and the delta re-reads LR2 to LR5 (5, 6, 7 and 6 turns; $1.75, $1.35, $1.54 and $0.73) come from the command center's caps decision of 2026-10-09 and are not in this repository |
| In agent mode the action writes the job token into the root checkout's `origin` URL, so `persist-credentials: false` does not keep it out of `.git/config`; the deny rules keep the model from reading it | `source_review` for the code path; the denial of `.git/config` reads is the fence receipt's (first row) | `src/github/operations/git-config.ts:129-134` and `src/modes/agent/index.ts:52-61`, read at v1.0.246 (`38c80c1`; the command center's compare shows only the bundled client version changing at v1.0.247); the record's shared sentence also cites two 2026-10-09 probes on the installed 2.1.295, not reproduced here |
| The workflow is well formed and has no audit finding | `local_integration` | at d1bcb6c: `actionlint` 1.17.0 exit 0; `zizmor` 1.30.1 offline, regular (`--no-ignores`) and pedantic, no findings (2 suppressed in each: `secrets-outside-env` on the R4 secret reads, as at a1bc3bc) |
| A hosted run authenticates, stays in bounds and reads the cache | not established | no hosted run is part of this PR |

### Local commands run

```
$ python3 scripts/validate.py            # on the R5 tree, committed as d1bcb6c
{"components": 70, "hashed_files": 10906, "profiles": 4, "receipts": 239, "status": "passed"}   exit 0
$ python3 scripts/evidence_manifest.py --check
{"files": 10906, "status": "passed"}   exit 0
$ uv run --quiet --no-project --python 3.12 --with pyyaml==6.0.3 python -m unittest tests.test_claude_pr_review_workflow tests.test_workflow_policy tests.test_workflow_security_coverage tests.test_workflow_hardening tests.test_github_automation_practice tests.test_adoption_docs_consistency
Ran 259 tests ... OK (skipped=2)   exit 0
$ uv run --quiet --no-project --python 3.12 python -m unittest tests.test_claude_pr_review_workflow    # without PyYAML
Ran 41 tests ... OK (skipped=15)   exit 0
$ actionlint .github/workflows/claude-pr-review.yml
exit 0
$ zizmor --offline --no-config --persona=pedantic .github/workflows/claude-pr-review.yml
No findings to report. Good job! (2 suppressed)   exit 0
$ zizmor --offline --no-config --no-ignores --persona regular .github/workflows/claude-pr-review.yml
No findings to report. Good job! (2 suppressed)   exit 0
$ uv run --quiet --no-project --python 3.12 --with pyyaml==6.0.3 python -B <scratchpad>/r5_pr894_mutants.py . a1bc3bc    # outside the repository
28 mutants, 28 caught, survivors: []   exit 0
$ uv run --quiet --no-project --python 3.12 python -B <scratchpad>/r5_pr894_mutants.py . a1bc3bc    # without PyYAML
28 mutants, 15 caught   exit 1 (expected: the 13 survivors are pins in the PyYAML-only shape tests; follow-up F-16)
```

Hosted checks at d1bcb6c: not read yet.

### Decision record

`docs/decisions/2026-10-08-claude-actions-pr-review.md` (why a manual dispatch and no pull request trigger, how the head is handled, the model's fence and what was checked at runtime, cost of one run with the measured runs and the cap derivation, changed test contracts, alternatives, what would overturn it), with a dated R5 section: the symbolic-link step, the caps from measurement, the non-string tool entries, the closed flag list and the boundary tests, the mutants, and the follow-ups. `docs/decisions/2026-10-04-ci-least-privilege.md` gets a dated addendum naming the second job on the federation rule.

### Host evidence

Not applicable: no file under `evidence/hosts/` changes.

### Checklist

- [x] New/changed GitHub Actions are pinned to a full commit SHA with a
      version comment (no floating tags).
- [x] New/changed workflows declare top-level `permissions: {}` and grant each job
      only what it needs (`contents: read`, or an explicitly justified addition).
- [x] No secrets are printed, logged or committed; no new required secret was
      added without a documented owner.
- [x] No new paid hosting, subscription or billing surface was introduced.
- [x] Peer-owned untracked files and worktrees were preserved (not deleted,
      moved or overwritten).

Activation is a separate step after landing: an owner sets `CLAUDE_PR_REVIEW_ENABLED` to `true`, then dispatches the workflow with a pull request number and head commit. Runs authenticate by workload identity federation and spend from the Console organization the four `ANTHROPIC_*` repository variables name; no Anthropic key is stored in GitHub. Each run is capped at 30 assistant turns and a $5 client budget, the owner's per-run spend limit on that organization. This PR and #892 both edit the write-grant inventory and the evidence manifest, so the second to land rebases once.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@seathatflowsinourveins
seathatflowsinourveins force-pushed the claude/api-actions-w4f-toolkit-review-20261009 branch from 8d5e7fe to 3a12894 Compare October 9, 2026 12:49
seathatflowsinourveins and others added 6 commits October 9, 2026 11:22
claude-pr-toolkit-review.yml runs pr-test-analyzer and
silent-failure-hunter from anthropics/claude-code's pr-review-toolkit
plugin on one pull request head, through claude-code-action v1.0.247.
Both agents list every behaviour and test change the description does
not declare, with a confidence on every finding. It is the command
center's J8 pre-cue step (W4 draft f), the same agents and settings that
found six undeclared changes in #892 and two verifier defects in #900.

The toolkit is checked out at anthropics/claude-code@602df92b, sparse to
its plugin directory, moved outside the workspace, verified file by file
against recorded SHA-256 values (git blob ids matched GitHub's at the
pin), and loaded with --plugin-dir; the action's plugins and
plugin_marketplaces inputs install unpinned code at run time and are
not used. Sonnet 5.5 at max effort; Read, Glob, Grep and the Agent tool;
CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS=0 so background agents are awaited;
a $5 budget; a review time limit of 20 or 35 minutes by diff size.

The bounds step counts the coordinator's own turns only, allows the
budget plus a $2 allowance for the agents' last requests, requires both
reports, uses an allow-list and names every unmet bound. Dispatch only,
owner only, first attempt only, gated by CLAUDE_PR_TOOLKIT_ENABLED; no
write scope beyond the OIDC token; nothing is posted to the pull request.

34 tests run the steps as written (without PyYAML too); eight weakened
copies each fail them; actionlint and zizmor (pedantic) report nothing.
openai/codex-action is recorded as WATCH with its pin and overturn
condition.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ay is missing

In the local J8 run on #894 the coordinator hit --max-budget-usd 5
(error_max_budget_usd) after both agents had handed back their full
reports through SubagentHandback, so it never relayed them and the
workflow would have withheld two complete reviews. The numbers and
report steps now take the coordinator's relay when it carries both
sections, and otherwise assemble the reports from each agent's own
SubagentHandback message in the execution file (model-free); with
neither, the relay text stays. The usage record says which source was
used. All three J8 streams so far carry SubagentHandback calls.

Three new tests cover a coordinator that stopped before relaying, a
missing handback and a complete relay preferred to the handbacks (37
tests); three mutants of the fallback each fail them; actionlint and
zizmor report nothing.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The command center approved a machine-readable trailer for the "defects
caught before a cue" ledger: each agent ends its report with one line,
J8-SUMMARY {"undeclared": <n>, "confidences": [...]}, so counts come
from the agents' own numbers and never from prose. The review itself is
unchanged. A test asserts the trailer in the prompt.

The sweep of the W4 drafts for the classes #894's pre-cue read found,
applied here: the empty paths array is expanded safely on bash before
4.4; a read step that succeeds without an execution file fails the
job; display_report and track_progress are asserted; the record states
that a public run's summary is public. 39 tests; actionlint and zizmor
report nothing.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
- R4, from the cc-native-practice lane's L3 read against claude-code-action
  v1.0.247 practice: the guard binds the ACTIONS_STEP_DEBUG and
  ACTIONS_RUNNER_DEBUG secrets and variables as booleans, the secret first,
  and refuses either. A test pins both bindings, and six weakened copies
  each fail it.
- The publish step writes the reports with `jq -j`, so a 60,000-byte report
  is no longer announced as cut, as the J8 micro read of #892 found for the
  same step. A boundary test covers it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…record and remove links from the head

R5, quality and correctness only. The coordinator moves to Opus 5.5 at max
with a $22 client budget; the cost bound is the budget times the measured
overrun factor 1.10 ($24.20), re-derived after the first three runs. The
numbers step reads every result record: each must be a success, or a budget
stop after both agents handed back, and a budget stop at or under the bound
publishes both reports. Publishing is gated on the numbers step's outcome.
The coordinator must call both toolkit agents and no other (retries
allowed), replayed on three J8 streams. A tool-list entry that is not a
string is a forbidden tool. A model-free step removes symbolic links from
the pull request head right after its checkout (GPT designated read of
b80a6eb, P2). New tests pin claude_args and --settings exactly; the module
goes from 41 to 56 tests. The record names every change and states that the
hosted file ends at the first result record, so a run whose agents have not
handed back by then fails closed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…g, and share the cost-factor wording

R6. The numbers step fails unless both called agents handed back (a
handback counts only under the coordinator's Agent call for that agent,
once per agent); relay headings no longer count anywhere (GPT designated
read of 7aa2c12, P2). The report step publishes the two handbacks,
pr-test-analyzer's then silent-failure-hunter's, and never the
coordinator's relay; the coordinator prompt no longer asks for a relay and
ends with one line, which saves its output tokens. Publishing is gated on
!cancelled() and the numbers step's success, as in the other W4 workflows.
The exact --settings pin compares canonical JSON, so 1 or 0 no longer
passes for true or false (J8 R5 micro of #894, N1). The record adopts the
shared cost-factor paragraph ($22 budget, $24.20 bound) and declares the
eight R5 changes the J8 R5 micro read found undeclared. 57 tests; 49 of 49
weakened copies fail a test.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@seathatflowsinourveins
seathatflowsinourveins force-pushed the claude/api-actions-w4f-toolkit-review-20261009 branch from 3a12894 to c3748c4 Compare October 9, 2026 15:24
@seathatflowsinourveins
seathatflowsinourveins marked this pull request as ready for review October 9, 2026 15:36
@seathatflowsinourveins
seathatflowsinourveins merged commit ca58cdf into main Oct 9, 2026
36 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/api-actions-w4f-toolkit-review-20261009 branch October 9, 2026 15:36
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-09T15:45:33.241654Z c3748c4 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c3748c49e7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +197 to +200
'd369fd3946a814bb7a9d4f32e971722fe259e301878986bc6312d6f6c56014a8 agents/pr-test-analyzer.md' \
'fa9b0daec5a267e7e66435cc48b3328301fc9f70c3af259fe248881327a1babc agents/silent-failure-hunter.md' \
'9435cc134fc72d56175f222894d401b0cf20f700d5bc0098c4257455314695ca .claude-plugin/plugin.json' |
sha256sum -c -

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Pin a toolkit revision that passes plugin validation

When this workflow is enabled, the hash check accepts the pinned silent-failure-hunter.md even though its frontmatter is malformed: Anthropic reproduced that claude plugin validate rejects this agent and that the runtime drops all of its metadata, including its declared name and model (upstream report). The synthetic transcript tests never load the plugin, so every paid run can start with one of the two required reviewers in this degraded, unsupported state; pin a corrected upstream revision and run the upstream validator before accepting the checkout.

AGENTS.md reference: AGENTS.md:L5-L7

Useful? React with 👍 / 👎.

echo '<pre>'
# iconv -c drops a character cut in half at the 60,000-byte mark; it still exits 1 for that, so the
# pipeline keeps its valid prefix instead of failing the step.
head -c 60000 "$report" | { iconv -f UTF-8 -t UTF-8 -c 2>/dev/null || true; } | sed -e 's/&/\&amp;/g' -e 's/</\&lt;/g' -e 's/>/\&gt;/g'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve the machine-readable summaries past the display cap

When the two handbacks exceed 60,000 bytes, this prefix-only truncation drops the tail even though each required J8-SUMMARY is deliberately the final line of its report. The full handbacks are not uploaded elsewhere—the artifact contains only usage.json, which omits undeclared and confidences—so the run can pass all bounds while the command center permanently loses the counts and potentially an entire second report. Extract and retain both summary records separately, or preserve the complete reports outside the capped job-summary rendering.

Useful? React with 👍 / 👎.

Comment on lines +243 to +245
- The full diff from its merge base with main is in ${{ runner.temp }}/pr-toolkit/pr.diff, and the changed
files are listed in ${{ runner.temp }}/pr-toolkit/pr.stat.
- The pull request description is in ${{ runner.temp }}/pr-toolkit/pr-body.md.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Tell agents when the diff is path-scoped

When a maintainer supplies the optional paths input, the preceding diff step restricts both pr.diff and pr.stat to those paths, but this prompt still calls the result the full diff and asks for every undeclared change in the pull request. The published summary also omits the selected scope, so agents and downstream readers can mistake a partial review for complete PR coverage. Describe the path restriction in the prompt and carry it into the report metadata.

Useful? React with 👍 / 👎.

seathatflowsinourveins added a commit that referenced this pull request Oct 11, 2026
…ule for retired slots

Answers the Codex review in GitHub of #1001 at ba6ab3a (four P2s) and the co-op's GPT first read at the same head (three P2s, the first three below); each was reproduced before the change.
- Sentence units split at a full stop or a semicolon whatever follows, and at a line break: "This route was retired; the current default is claude-code-action." was one unit that said retired.
- The generic phrases "in Actions", "through Actions" and "under WIF" count as the retired route only with Claude context, judged per leaf string; the named workflows, "through #909" and the managed app count
  everywhere. The routes the decision retains (every bullet of its "What remains" section, a catalog job in GitHub Actions, a non-model gate under WIF) are accepted.
- The skill's entry point SKILL.md is scanned, and a test requires every file of the skill folder to be read by a check.
- The one-default rule names the retired exception in slots.json, in the slots.md copy and in step 3 of SKILL.md (a retired row has none and says what replaced the route).
- The ruled default text is pinned exactly (the read's measured boundary: the date changed on all three surfaces passed), and a check compares every row's Overturn when across slots.json, the index and the
  layer pages (one step wider than the review of #1005, where an edit left the index cell behind).
Each finding fails its own test when its fix is undone alone, and 43 single-edit mutations are caught; the old guard accepts the four bad scenarios of the read and rejects its two legitimate ones.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Oct 11, 2026
…ew (#1001)

* Retire the skill's Claude-in-GitHub rows and point them to local review

The claude-native-practice skill still selected the route that
docs/decisions/2026-10-10-retire-in-github-claude-review.md retired:
the github-action row (anthropics/claude-code-action at a pin) and the
Actions part of pr-review-pre-cue.

- slots.json, slots.md and the layer pages: `retired` joins the status
  vocabulary; github-action becomes retired with a dated block (record,
  correction #141, scope, the three workflows, the former default, route,
  alternatives, evidence and notes as a snapshot, the revisit triggers);
  pr-review-pre-cue becomes default (scoped) and lists the routes
  available now (local Claude reads, GPT reads, Codex's GitHub review,
  the required verdict-review-gate) apart from the open #953 worker.
- The scope is the owner's repositories (the command center's ruling of
  2026-10-10), not "this repository": a hermetic A/B showed a session in
  another repository concluding that the action was still fine there.
  The same ruling retires the managed Claude Code Review GitHub App.
- Stale texts the independent pre-read and the stricter check found: the
  release-tracking note (Dependabot as the action's tracker), the
  scheduled-tasks route (GitHub Actions schedules only for jobs that run
  no model) and its source mapping, the usage-cost-monitoring note
  (Actions under WIF), and the pre-cue source mapping (in Actions through
  #909). The pre-cue route no longer lists a read the record does not name.
- tests/test_native_practice_retired_routes.py: no current row selects a
  retired route; a retired route is named elsewhere only in a sentence or
  list item that says it is retired; every line of slots.json, slots.md,
  the layer pages and cross-client.md is read (duplicates kept, the source
  mappings of live slots included); only the dated snapshots are exempt.
  Positive controls are the rows as they stood; the mutation table is in
  the pull request description.
- The slot manifest of #923 is relabeled against the new slots.json
  bytes, so the exact-difference check of the provenance test runs again
  instead of skipping.
- docs/decisions/2026-10-09-claude-code-native-practice.md: a dated
  pointer to the retirement, as #981 did for the older records.

Source read for this change: anthropics/claude-code-action at
2dca132ff0e0c4094ce6048b422c6915a071210b, base-action/src/
parse-sdk-options.ts:283-284 (CLAUDE_CODE_ENTRYPOINT is set to
claude-code-github-action).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

* Close the holes in the retirement guard and qualify the one-default rule for retired slots

Answers the Codex review in GitHub of #1001 at ba6ab3a (four P2s) and the co-op's GPT first read at the same head (three P2s, the first three below); each was reproduced before the change.
- Sentence units split at a full stop or a semicolon whatever follows, and at a line break: "This route was retired; the current default is claude-code-action." was one unit that said retired.
- The generic phrases "in Actions", "through Actions" and "under WIF" count as the retired route only with Claude context, judged per leaf string; the named workflows, "through #909" and the managed app count
  everywhere. The routes the decision retains (every bullet of its "What remains" section, a catalog job in GitHub Actions, a non-model gate under WIF) are accepted.
- The skill's entry point SKILL.md is scanned, and a test requires every file of the skill folder to be read by a check.
- The one-default rule names the retired exception in slots.json, in the slots.md copy and in step 3 of SKILL.md (a retired row has none and says what replaced the route).
- The ruled default text is pinned exactly (the read's measured boundary: the date changed on all three surfaces passed), and a check compares every row's Overturn when across slots.json, the index and the
  layer pages (one step wider than the review of #1005, where an edit left the index cell behind).
Each finding fails its own test when its fix is undone alone, and 43 single-edit mutations are caught; the old guard accepts the four bad scenarios of the read and rejects its two legitimate ones.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

* Bind the manifest label to the shipped slots file and close the remaining holes of the retirement guard

Answers the CC's Claude first read of #1001 at 6e97620 (two P2s, six P3s) and Codex's second review in GitHub of the same head (two P2 threads); each was reproduced before the change.
- P2-1: the slot manifest was labeled against the slots.json of ba6ab3a, so the exact-difference check of the provenance test skipped at the head (OK, skipped=1). The label is the sha256 of the shipped file again,
  the check no longer skips, and a new test fails when a change to slots.json does not carry its label (the CC's ruling: a silent skip is this defect's class, so it fails loud).
- P2-2: a live source mapping (skills-library, primary_sources[5]) still ran `claude -p` "in a GitHub workflow ... plus a nightly cron"; the guard had no such form. It is reworded to say the run is retired,
  and "in a GitHub workflow" (also from, via, through, using, as) counts as the retired route when the sentence is about Claude.
- P3-1: the Claude Code GitHub Action, /install-github-app and an @claude mention are the retired route under other names (claude-code-security-review stays out: a documented rejection).
- P3-4: the Former default exemption of a layer page applies to a retired slot only.
- P3-5: a retired slot's own source mapping is read, and now says retired.
- P3-2: "e.g." and "i.e." no longer end a sentence, and wrapped lines of prose are one paragraph (the slots.md prose and SKILL.md were judged line by line).
- P3-6: the manifest prose lists the fields beyond the exported ones that differ from the snapshot.
- P3-7: the entry point directs ("a retired row has no default, so follow the replacement it names"), and the step-3 pin no longer fails with an IndexError on a renumbering.
Each correction fails its own test when it is undone alone, and the mutation table is extended.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

* Judge retired-route prose one physical line at a time and close the guard's other gaps

Answers the command center's Claude micro of #1001 at ef3f3e7 (P2-1, P3-1 to P3-5) and Codex's four threads of the same head (:43, :63, :122, :211). Each case was reproduced before the change:
compare-guards-ef3f3e7b-vs-third-commit.txt runs 19 scenarios through the guard of ef3f3e7 and this one; the old guard is wrong on 13 (it accepts a retired route that is reintroduced), this one on none.
- P2-1 (correction #160): joining wrapped lines into one paragraph let "Retired 2026-10-10" followed by a line that says the current default is claude-code-action pass on four prose surfaces (SKILL.md, slots.md,
  cross-client.md, layer-page preambles), the scenario Codex raised at ba6ab3a line 41. Prose is judged one physical line at a time again, a line break ends a unit, and the message of every line-judged surface
  states the rule (RULE; the legend of slots.md states it too and a test pins the two to each other).
- P3-1 and Codex :43: three more names of the same route, all in the vendor's own text: the page's title "Claude Code GitHub Actions" and its "installs the Claude GitHub App"
  (code.claude.com/docs/en/github-actions.md, capture sha256 87b99aef, lines 5, 9, 33, 63-64), the mirror claude-code-base-action (anthropics/claude-code-action@2dca132ff0e0:
  .github/scripts/check_workflow_hardening.py:18, .github/workflows/sync-base-action.yml:1 and 43, CLAUDE.md:20) and the carrier named first ("Use the GitHub Action for Claude reviews", generic
  like "in Actions": it counts when the field or unit is about Claude). The families "in Actions" and "through Actions" take on, with, inside and via.
- Codex :122 and the command center's P3-3 and P3-5: a unit that negates the retirement outright ("not retired", "never retired", "no longer retired", "isn't retired", "unretired") no longer counts as saying
  retired. The other half of that scenario, "was retired, but is now current", is the boundary the command center ruled (the guard is a tripwire for stale text; review reads what it accepts); the module names it.
- Codex :211 and P3-3: a layer-page section with a label twice (a second Route or Overturn when bullet) is a violation, because the page-to-slots.json comparison keeps the first bullet only.
- Codex :63 and P3-4: the target of every slots.md row is checked against <page>#<id>, from the layer of the slot.
- The description line of SKILL.md names the topic that invokes the skill; it is pinned by sha256 to the text the routing was measured with and read as a topic, not as a route.
- Scheduled-tasks (Codex on #1005, scheduling-supervision.md:9, and the command center's ruling): the route no longer bans every model-backed job. The retirement record
  (docs/decisions/2026-10-10-retire-in-github-claude-review.md:5-6, :43) retires Claude review inside GitHub and keeps Codex's GitHub review, so the route says GitHub Actions schedules repository jobs and a
  scheduled Claude task runs locally; a test pins the wording on slots.json, the layer page and the alternatives.
The label of claude-native-slot-manifest.json equals the sha256 of the shipped slots.json again (relabel last); test_the_label_names_the_shipped_slots_file and test_no_other_exported_field_differs
(no skip) are unchanged from ef3f3e7. 72 of 72 single-edit mutants of the guard and its data are caught and 5 controls pass (mutations-v4-forward-commit-3.md).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

* Align the manifest's scheduled-tasks note, state the rule in every explaining unit, split units after a closing delimiter and pin the scope of the name patterns

The final fix round of #1001 under the command center's convergence ruling: its Claude micro at efaa077 (blockers B1 to B3, P3-2) and its adjudication of 2026-10-11T13:50Z of Codex's third review. Each item was reproduced on that
head before the change.
- claude-native-slot-manifest.json:47 (Codex P2, B1, a text of this PR that is false at the head): the note on the differences from the snapshot still said the route of scheduled-tasks was narrowed "to jobs that run no model",
  while the shipped route says that GitHub Actions schedules repository jobs, that Claude runs inside GitHub are retired (2026-10-10, docs/decisions/2026-10-10-retire-in-github-claude-review.md:5-6, :43) and that a scheduled
  Claude task runs locally. The note says so now, and test_the_scheduled_tasks_route_retires_the_claude_routes_and_not_every_model_backed_job reads the manifest as well as the route (fails with the note undone: M79).
- B2: the module claims that the message of every line-judged surface states the rule, and it did not for the labelled bullets of a layer page or for the Overturn when cell of an index row. Every explaining unit states it now
  (a selecting field names no unit and states nothing); the three expected messages carry the rule and test_every_explaining_unit_of_a_layer_page_and_of_the_index_states_the_rule_and_a_selecting_field_does_not pins all three
  kinds (mutants M83 to M85).
- tests/test_native_practice_retired_routes.py:83 (Codex P2, P3-2): a closing quote, parenthesis or bracket between the full stop and the white space kept `The route was "retired." The current default is claude-code-action.` in one
  unit that said retired. The sentence pattern takes closing delimiters (the curly quotes are built with chr(), so the file stays ASCII); test_a_closing_quote_parenthesis_or_bracket_after_the_full_stop_does_not_join_two_claims fails
  without it (M78, M82) and keeps `claude-code-action was retired (see the record.) No row selects it.` accepted (control C6).
- :48 (Codex P2, workflow-first phrasing) is a follow-up and the class closes (rule 4 of the convergence ruling): the module states that the guard pins the named retired artifacts and the retired slot rows, that generic phrasing of a
  Claude run in a GitHub workflow cannot be enumerated and is out of its scope (the retirement record governs it), and test_generic_workflow_phrasing_is_out_of_scope_and_the_module_says_so pins the boundary. No pattern is added.
- B3 is the description: the two earlier command blocks name the 37 modules they ran (the 46 of the first block without the nine tests.test_local_pages_* modules).
80 single-edit mutants of the guard and its data are caught and 6 controls pass (mutations_retire_v6.py).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

* Close the sentence-splitter class by documentation, pin the module's ASCII and the scope list

The command center's confirmation micro of #1001 at bdf60fc (research-cc-role-20261008/pr1001-claude-micro-bdf60fc6.md: blockers 1 and 2, P3s) and its correction #163 of 2026-10-11T16:18Z on Codex's three threads of 15:39:21Z.
No code of the guard changes in this commit.
- Blocker 2: the module said that it stays ASCII and held one U+2026 in a comment (added by efaa077). It is three full stops now, and test_the_source_of_this_module_is_printable_ascii pins the claim; it fails on the module
  of bdf60fc (['U+2026'] != []).
- Blocker 1 (the pull request description, and this message corrects the message of bdf60fc): the earlier statement that the 37 modules ran "without its nine tests.test_local_pages_* modules" was false. They are the first 37
  modules of the 46-module list, and the nine left out are its last nine (tests.test_local_pages_fleet_execution to tests.test_local_pages_workstation, 144 tests, run and counted); 29 of the 46 are local-pages modules.
- Codex :92 (an exclamation mark or a question mark does not end a unit) is not patched: the sentence splitter has drawn findings in five consecutive rounds (the lowercase clause, "e.g.", the joined lines, the closers,
  the exclamation mark and the question mark) and was patched four times, so rule 4 of the convergence ruling closes the class the way line 48 closed. The module docstring states exactly what the guard accepts as a unit end
  and that review reads everything else; test_the_unit_ends_and_the_scope_of_the_name_patterns_are_documented pins the sentence and the behavior, and fails on the module of bdf60fc, where the sentence is missing.
  A pinned upstream sentence segmenter is a research-queue item (rule 8, layer instructions-skills).
- Codex :51 (a carrier-first name of the managed Code Review app) is confirmed within the closed class: the same scope paragraph says that reordered phrasings of a pinned artifact are out of the guard's scope.
- P3-1: the scope list named 11 of the 17 patterns; it names all 17 now, and test_the_scope_of_the_name_patterns_names_every_pattern compares the list itself with PATTERNS (it fails on the module of bdf60fc).
  P3-2: the right single quotation mark closer is tested (its mutant survived). P3-3: the comment at the sentence pattern says what the code does.
- Codex :170 (retired.on is checked for its shape only) is a follow-up under rule 5: the shipped value is the date of the record, so 2099-12-31 is a mutant and not a reproduced failure of the shipped state.
88 single-edit mutants of the guard and its data are caught and 6 controls pass (mutations_retire_v9.py).

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>

---------

Co-authored-by: Claude Sonnet 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant