Repository navigation
Add a pre-cue read by Anthropic's pr-review-toolkit agents in Actions - #909
Conversation
|
Dependency limit exceeded — report not shown. This pull request scan exceeded the 10,000-dependency limit applied to this scan, so the results are incomplete and may be inaccurate. To avoid reporting false positives, Socket has not posted a report. Upgrade your plan to raise the dependency limit and get complete reports, or view the partial scan in the dashboard. Socket is always free for open source. If this is a non-commercial open source project, contact us to request a free Team account. |
…tries and raise the caps to 30 turns and $5 R5, quality and correctness only. A model-free step removes every symbolic link from the pull request head right after its checkout, so a link to a process environment file or to ../.git/config cannot carry credentials into the job summary (GPT designated read of #909, P2; the command center's security read). The numbers step counts every tool-list entry that is not a string as a forbidden tool, named "non-string tool entry", instead of dropping it (GPT designated read of this PR at f5f1fd2, P2). New tests pin the whole claude_args list and the whole --settings JSON. The caps follow the command center's measurement-based derivation: 30 turns and a $5 client budget. The record gains a dated R5 section, corrects its evidence class, withdraws the earlier dry cost estimate and states which tests run without PyYAML. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…894) ### Scope - What this PR changes, in one or two sentences: adds `claude-pr-review.yml`, an on-demand, read-only review of one same-repository pull request head by `anthropics/claude-code-action` v1.0.247. A maintainer dispatches it from `main` with the pull request number and the exact head commit; the review goes to the job summary and nothing is posted to the pull request. It runs only while the repository variable `CLAUDE_PR_REVIEW_ENABLED` is `true`, so merging this starts nothing. - Effort (added 2026-10-08 at head a6d2379): `--effort max` in `claude_args`, because judgment work (designated reads, adjudication, pull request and security reviews, audits) runs at `max` under the command center's effort mapping, and every job records its level and reason. The level has to be in `claude_args`: `--restricted` ignores the settings files that would otherwise carry it, and Opus 5.5 runs at `medium` on the API when a request leaves effort unset. The bounds test now requires the flag; removing it fails the test. A loopback dry run of the installed 2.1.295 client with this workflow's `claude_args` sent `output_config.effort: "max"` on every request. - Fixes from the pre-cue read of the sibling #892 (2026-10-09, head d6f8909): the record names every behaviour of this new workflow: the trigger; the slug, owner, first-attempt and variable guards; the timeout and grants; the guard and binding exits; the head-as-data checkout and diff limits; the explicit action inputs `display_report` and `track_progress` (defaults in the pinned `action.yml`); and the 14-day usage artifact. The bounds step now uses an allow-list, fails on a start record without lists or on an empty result, and names each unmet bound. The review cut is character-safe and says when it cut. The step tests no longer skip without PyYAML. The 2026-10-03 cooldown waiver is cited. - Fixes from this PR's own pre-cue toolkit read (J8 on d6f8909, head 8e64120): - the empty paths array is now expanded safely on bash before 4.4; - a success without an execution file now fails the job; - `display_report` and `track_progress` are asserted; - the artifact name and the action-pin sentence are corrected, and the parity receipt's dependency on #892 is stated; - `claude-pr-review.yml:review` is now in both grant tables of ci-least-privilege; - the record states that a public run's summary is public. - Pre-cue toolkit read of this PR (J8 on d6f8909, 2026-10-09; `pr-test-analyzer` and `silent-failure-hunter`): every finding at confidence 80 or above, and its status at head 41c9c15. | Finding | Status at 41c9c15 | Fixing commit and evidence | | --- | --- | --- | | `pr-test-analyzer` 1 (90): the record names the wrong artifact | fixed | 8e64120: the record names `claude-pr-review-usage-<run id>-<attempt>`, the upload's name; 41c9c15 adds `test_the_usage_artifact_is_named_for_the_run_and_its_attempt` | | `silent-failure-hunter` 2 (88): the record and the workflow disagree on the artifact | fixed | the same two commits | | `pr-test-analyzer` 2 (85): `docs/github-automation.md` claims the harness audit's action pin | fixed | 8e64120: the bullet says the two pins differ until #892 lands | | `silent-failure-hunter` 1 (92): the same false claim (pin and federation variables) | fixed | 8e64120, as above; the four federation variables are the same | | `pr-test-analyzer` 3 (85): the headline evidence file is not in this PR | fixed | 8e64120 states that #892 adds it and lands first; 41c9c15 cites its `LR` entry field by field | | `silent-failure-hunter` 3 (80): "#897" and "two blocking findings" have no source in that receipt | fixed | #892 adds `target_pull_request`, `target_head`, `report_blocking_findings` and `report_sha256` to the `LR` entry (a marked 2026-10-09 amendment from the run's harness receipt); 41c9c15 cites them, calls the verdict the model's own, and states that the prompt and `claude_args` are unchanged from the run's workflow commit a6d2379 to that head (R5 later raises the two caps and the prompt's turn sentence; see R5 below) | - R4, debug logging set as a repository secret or variable (2026-10-09; the cc-native-practice lane's L3 read against `anthropics/claude-code-action` v1.0.247 practice): the guard binds `(secrets.ACTIONS_STEP_DEBUG || vars.ACTIONS_STEP_DEBUG) == 'true'` and `(secrets.ACTIONS_RUNNER_DEBUG || vars.ACTIONS_RUNNER_DEBUG) == 'true'` as booleans and refuses either before any token is requested; a test pins both bindings, and six weakened copies each fail at least one test. This is hardening: the action step already pins `ACTIONS_STEP_DEBUG: 'false'`, which is what the action's full-output switch reads. zizmor's auditor persona reports `secrets-outside-env` on the two reads; CI runs the regular persona with `--no-ignores`, which reports nothing. The R4 lines are byte-identical in #892, #894, #895, #896 and #909; R5 leaves them unchanged here. - The report notice at the cap (2026-10-09; the J8 micro read of #892): the publish step wrote with `jq -r`, which adds a newline, so a 60,000-byte report was announced as cut. It writes with `jq -j` now; a test publishes exactly 60,000 and 60,001 bytes, and a copy that writes with `jq -r` fails it. R5 adds the 59,999-byte case. - From the J8 micro read of 41c9c15 (2026-10-09; 4 of 6 listed findings fixed, PTA-3 and SFH-3 partly because the receipt the record cites was not in this tree): this PR now carries a byte-identical copy of `local-parity-receipt.json` (sha256 `ba498b82…`, the file #892 adds), so the citation no longer depends on which PR lands first, and the docs bullet states the action pin and the federation variables instead of comparing with the harness audit on `main`. - R5 (2026-10-09, head d1bcb6c, limited by the command center to correctness and test quality) adds a model-free step, `Remove symbolic links from the pull request head`, right after the head checkout: `set -euo pipefail; find pr-head -path pr-head/.git -prune -o -type l -exec rm -f {} +`. It raises the approved caps from measurement: `--max-turns` 12→30 and `--max-budget-usd` 3→5. The prompt's "You have at most 30 turns", the numbers step's bounds (1 to 30 assistant turns, at most $5, with matching messages), the header comment and `docs/github-automation.md` follow. The numbers step now counts each non-string tool-list entry as a forbidden tool named `non-string tool entry` instead of dropping it. The R4 guard lines, the deny list and the `github_token` input are kept exactly as at a1bc3bc. The test module goes from 33 to 41 tests. Eight are new: the symlink step's position and exact text; its behaviour on planted links to `/proc/self/environ`, `../.git/config` and a subdirectory, plus a re-run on a link-free head; a 59,999-byte report published whole; object, null and number tool entries refused and named; the prompt stating the `--max-turns` value; and the caps at 30 turns and $5 and just above. Changed tests pin the full `claude_args` list and `--settings` JSON exactly, assert the exact `if:` strings of the publish and numbers steps, run the guard with production's debug-off values (false/false/empty), and check that the 59,999-byte prefix before a cut character survives. All 28 weakened copies of the workflow fail at least one test with PyYAML present, as the hosted validate job has it; without PyYAML 15 of 28 do, because the pins sit in the PyYAML-only shape tests. The decision record gains a dated R5 section and names the R4 guard change in its what-changed paragraph. It corrects "Read denies for every `.git` directory" to "for the files under every `.git` directory". It restates the cap derivation and the expected cost per run, and withdraws the dry $0.70–$1.30 estimate, which was below the measured first-read cost. It adds the shared follow-up line, the shared `origin` URL sentence and two action sources. `manifests/evidence.json` is rehashed for the four changed files. - Sources of the R5 items: the command center's security read of a1bc3bc (the symbolic links, the closed flag list, the exact `if:` strings, the guard's debug-off values, the 59,999-byte prefix, and the stale guard sentence in the record); the command center's caps decision of 2026-10-09; the GPT designated read of #895, P2 (non-string tool entries, applied to all five W4 workflows); the cc-native-practice lane's third audit, P3-1 (the shared `origin` URL sentence). - Caps derivation: `LR`, a first read of #897 with this workflow's prompt and `claude_args` on the installed client, used all 12 of its 12 assistant turns and cost $2.23 ($1.87 by the client's estimate). So the need is above 12, and 2.5 times that floor gives 30 turns; twice $2.23 is about $4.5, rounded up to $5. Four delta re-reads (LR2 to LR5) used 5, 6, 7 and 6 turns and cost $1.75, $1.35, $1.54 and $0.73. A run is expected to cost about $1.9 to $2.3 for a first read and $0.7 to $1.8 for a delta re-read. - Filed as follow-ups before any enabling variable is set, not in R5: the security hardening from the 2026-10-09 read (debug-value widening, the tool-list shape apart from the non-string entry above, extra deny rules, token-source and guard-step assertions); the rest of F-02 (a non-empty tool list, checking the `tool_use` names); F-16 (the pins that run only with PyYAML). - Not run in R5: a fence smoke with a planted symbolic link, which needs a model run; any hosted run. On the first run, watch whether the action appends an `--mcp-config` that `--strict-mcp-config` still honours (the read's G3); the numbers step fails the run on any MCP server. - Base commit: `aba02ec3` (merge base at a1bc3bc and at d1bcb6c, which builds on a1bc3bc; as read at a1bc3bc, main had moved to `ebe6c223` with #890, #900, #897 and #907; only `manifests/evidence.json` overlaps, and the one rebase happens at landing) - Lane: `lane:foundation` - Owned paths touched: `.github/workflows/claude-pr-review.yml` (new), `tests/test_claude_pr_review_workflow.py` (new), `tests/test_workflow_policy.py` (one inventory entry, one exemption), `tests/test_workflow_security_coverage.py` (coverage set and one zizmor test), `docs/decisions/2026-10-08-claude-actions-pr-review.md` (new), `docs/decisions/2026-10-04-ci-least-privilege.md` (dated addendum only), `docs/github-automation.md` (current-practice bullet), `evidence/artifacts/claude-actions-fence-smoke-20261008/receipt.json` (new), `evidence/artifacts/claude-actions-fence-smoke-20261008/local-parity-receipt.json` (new; byte-identical to the copy #892 adds), `manifests/evidence.json` (registration, last commit). ### R6 (2026-10-09), at bd10ba3 - **Cost bound:** the budget × 1.10 = $5.50, from four measured J8 runs on a $5 budget ($5.46, $5.33, $5.16, $5.0007), re-derived after the first three hosted runs. - A budget stop at or under it publishes the text written before the stop, with a line naming the stop. - Publishing is gated on `!cancelled() && steps.numbers.outcome == 'success'`, because the action fails its own step on any budget stop. - **`--max-turns 30` removed:** the pinned action (`base-action/src/run-claude-sdk.ts:241-250` at 2dca132f, since 6ef6450f) fails a success whose `num_turns` exceeds it. - On Claude Code 2.1.295, `num_turns` counts transcript messages: 57 for 12 requests under `--max-turns 12`. That is a code reading plus a local measurement, not a hosted run. - The numbers step's 30-assistant-turn bound and the prompt's "at most 30 turns" stay. - **Job timeout:** 20 → 30 minutes. 30 turns at the measured pace (12 turns in 512,670 ms) take about 21.4 minutes before setup. #892 uses 30. - **Settings pins:** both compare canonical JSON, so `1` no longer passes for `true` (J8 R5 micro, N1, confidence 70). - **Record:** - the shared cost paragraph; - an upstream-issue draft (not filed); - the seven changed test expectations the J8 R5 micro found undeclared (U1–U7). - **Local checks at bd10ba3:** - `validate.py` and the manifest check pass (10,906 files); - six modules: 262 tests, OK; - this module: 44 tests (16 skip without PyYAML); - actionlint and zizmor are clean; - 39 of 39 weakened copies fail a test with PyYAML. - **Hosted checks at bd10ba3:** not read yet. ### SOTA sources - anthropics/claude-code-action, v1.0.247 at `2dca132ff0e0c4094ce6048b422c6915a071210b` (<https://github.com/anthropics/claude-code-action/tree/2dca132ff0e0c4094ce6048b422c6915a071210b>): `docs/security.md` ("Do not check out an untrusted ref into the workspace root before this action"; base ref at the root and head ref in a subdirectory), `docs/setup.md` (federation inputs; "a static credential takes precedence and federation will not be used"), `action.yml` (inputs and the `execution_file` output), `base-action/src/parse-sdk-options.ts` (argument parsing with `shell-quote`, default setting sources, debug forcing full output), `src/github/operations/git-config.ts` (git authentication in the root checkout; lines 129-134 write the job token into the checkout's `origin` URL when `allowed_non_write_users` is unset) and, new in R5, `src/modes/agent/index.ts` (lines 52-61, where agent mode calls that configuration). R5 read both files at v1.0.246 (`38c80c1`); the command center's compare of v1.0.246 and v1.0.247 changes only the bundled Claude Code version. - Claude Code 2.1.295, the version the pin installs, `claude --help`: `--restricted` ("removes the built-in tools that run commands or code ... and ignores user, project and local settings files ... Also confines the file tools to the working directories"), `--permission-prompts none` ("anything that would prompt is denied automatically"), `--tools`, `--settings`, `--strict-mcp-config`, `--max-budget-usd` and, for the R5 caps, `--max-turns`. The help does not say whether a symbolic link's target is resolved, which is why R5 removes the links before the model runs. - GNU findutils `find` 4.10.0 as installed (`find --version`), new in R5: `find pr-head -path pr-head/.git -prune -o -type l -exec rm -f {} +`, measured by the step test on planted links and on a link-free head (exit 0, no `rm` started). The runner image's `find` was not measured. - GitHub, Contexts reference, `runner.debug` (set only while debug logging is on, so the guard's `RUNNER_DEBUG_SIGNAL` is empty on a normal run), new in R5 as the basis of the guard tests' debug-off values; taken from the command center's read of a1bc3bc, not re-read in R5 (no network). - GitHub, OpenID Connect reference, "Filtering for pull_request events" and "Filtering for a specific branch" (<https://docs.github.com/en/actions/reference/security/oidc>); GitHub Security Lab, Preventing pwn requests (<https://securitylab.github.com/research/github-actions-preventing-pwn-requests/>). - Anthropic documentation read 2026-10-08: Workload identity federation (<https://platform.claude.com/docs/en/manage-claude/workload-identity-federation>); How Claude Code uses prompt caching (<https://code.claude.com/docs/en/prompt-caching>); Pricing (<https://platform.claude.com/docs/en/about-claude/pricing>). ### Evidence-class table | Claim | Evidence class | Command / receipt | | --- | --- | --- | | With the fence flags (`--restricted`, `--tools`/`--allowedTools` Read,Glob,Grep, `--strict-mcp-config`, `--permission-prompts none`, `--settings` with hooks off, `claudeMdExcludes` and three deny rules; Haiku 5.5, 10 turns, $0.50, as the receipt lists; this workflow adds `--setting-sources user`, `--add-dir`, `--effort max`, four deny rules and `blockReadsOutsideWorkingDirectories`, a key in the 2.1.295 build that the receipt does not exercise) the session's tools are Glob, Grep and Read; reads outside the tree and of `.git/config` are denied; planted hooks, instruction files and a skill under `pr-head` have no effect | `native_proven` on the installed 2.1.295 client (not through the action, not on a GitHub runner) | `evidence/artifacts/claude-actions-fence-smoke-20261008/receipt.json`, three runs | | The action's own parser turns this workflow's `claude_args` into those flags and that JSON | `local_integration` (at a1bc3bc only) | the parser's three helpers replayed with `shell-quote` 1.10.0 on the workflow text at a1bc3bc; not re-run at d1bcb6c (no network to install `shell-quote`). R5 changes two plain values (`12` to `30`, `3` to `5`), and `test_claude_has_three_read_tools_and_fixed_bounds` asserts the whole list, split per line with `shlex`, and the `--settings` JSON | | The guard, binding, symbolic-link, diff, numbers and review steps behave as described | `synthetic` | `tests/test_claude_pr_review_workflow.py`, 41 tests at d1bcb6c (the 26 step tests run without PyYAML too; the 15 shape tests need it); each step's shell is taken from the workflow file and executed against a local stand-in for `gh`, a local git repository or a planted `pr-head` tree | | Symbolic links under `pr-head` are removed before the model runs, `pr-head/.git` is left as the checkout wrote it, and a link-free head passes | `synthetic` | `test_symbolic_links_in_the_head_are_removed_and_nothing_else` (links to `/proc/self/environ`, `../.git/config` and `../..`; GNU find 4.10.0); no fence smoke with a planted link has run on the client | | The tests detect a weakened workflow | `synthetic` | 20 mutated copies on 2026-10-08 (that day's bounds), 20 failed at least one test; on 2026-10-09 three turn-bound mutants each failed, and the new bounds have their own tests; at 41c9c15 three artifact-name mutants each pass the 30 earlier tests and fail the new name test; at d1bcb6c 28 mutated copies (the symbolic-link step, the flag and settings pins, the `if:` strings, the guard's debug-off values, the report cap, the non-string tool entries, the caps and the prompt) each fail at least one test with PyYAML 6.0.3 present, as the hosted validate job has it; without PyYAML 15 of 28 do (the other 13 are pins in the shape tests; follow-up F-16) | | This workflow's own prompt and `claude_args` review a real pull request within bounds at `--effort max` on Opus 5.5 | `local_integration` (installed client, billed to a second key; not through the action) | `evidence/artifacts/claude-actions-fence-smoke-20261008/local-parity-receipt.json`, carried byte-identical in this PR and in #892 (sha256 `ba498b82…`), its `LR` entry: workflow commit a6d2379, whose prompt and `claude_args` are unchanged at d1bcb6c apart from the two caps R5 raised (`--max-turns` 12 to 30, `--max-budget-usd` 3 to 5) and the prompt's turn sentence; #897 at `target_head` 24821c5; 12 of 12 assistant turns; $1.868667 of the then $3 budget; tools Glob/Grep/Read; no MCP; `success`; `report_blocking_findings: 2`, the report's own verdict line, unchecked (`report_sha256` c49bee04, a 2026-10-09 amendment) | | The caps, 30 assistant turns and $5, bound a runaway at about twice the measured need; a run is expected to cost about $1.9 to $2.3 for a first read and $0.7 to $1.8 for a delta re-read | `local_integration` (installed 2.1.295 client, billed to the second key; not through the action) | the `LR` entry above (12 of 12 turns, client estimate $1.868667); the $2.23 figure and the delta re-reads LR2 to LR5 (5, 6, 7 and 6 turns; $1.75, $1.35, $1.54 and $0.73) come from the command center's caps decision of 2026-10-09 and are not in this repository | | In agent mode the action writes the job token into the root checkout's `origin` URL, so `persist-credentials: false` does not keep it out of `.git/config`; the deny rules keep the model from reading it | `source_review` for the code path; the denial of `.git/config` reads is the fence receipt's (first row) | `src/github/operations/git-config.ts:129-134` and `src/modes/agent/index.ts:52-61`, read at v1.0.246 (`38c80c1`; the command center's compare shows only the bundled client version changing at v1.0.247); the record's shared sentence also cites two 2026-10-09 probes on the installed 2.1.295, not reproduced here | | The workflow is well formed and has no audit finding | `local_integration` | at d1bcb6c: `actionlint` 1.17.0 exit 0; `zizmor` 1.30.1 offline, regular (`--no-ignores`) and pedantic, no findings (2 suppressed in each: `secrets-outside-env` on the R4 secret reads, as at a1bc3bc) | | A hosted run authenticates, stays in bounds and reads the cache | not established | no hosted run is part of this PR | ### Local commands run ``` $ python3 scripts/validate.py # on the R5 tree, committed as d1bcb6c {"components": 70, "hashed_files": 10906, "profiles": 4, "receipts": 239, "status": "passed"} exit 0 $ python3 scripts/evidence_manifest.py --check {"files": 10906, "status": "passed"} exit 0 $ uv run --quiet --no-project --python 3.12 --with pyyaml==6.0.3 python -m unittest tests.test_claude_pr_review_workflow tests.test_workflow_policy tests.test_workflow_security_coverage tests.test_workflow_hardening tests.test_github_automation_practice tests.test_adoption_docs_consistency Ran 259 tests ... OK (skipped=2) exit 0 $ uv run --quiet --no-project --python 3.12 python -m unittest tests.test_claude_pr_review_workflow # without PyYAML Ran 41 tests ... OK (skipped=15) exit 0 $ actionlint .github/workflows/claude-pr-review.yml exit 0 $ zizmor --offline --no-config --persona=pedantic .github/workflows/claude-pr-review.yml No findings to report. Good job! (2 suppressed) exit 0 $ zizmor --offline --no-config --no-ignores --persona regular .github/workflows/claude-pr-review.yml No findings to report. Good job! (2 suppressed) exit 0 $ uv run --quiet --no-project --python 3.12 --with pyyaml==6.0.3 python -B <scratchpad>/r5_pr894_mutants.py . a1bc3bc # outside the repository 28 mutants, 28 caught, survivors: [] exit 0 $ uv run --quiet --no-project --python 3.12 python -B <scratchpad>/r5_pr894_mutants.py . a1bc3bc # without PyYAML 28 mutants, 15 caught exit 1 (expected: the 13 survivors are pins in the PyYAML-only shape tests; follow-up F-16) ``` Hosted checks at d1bcb6c: not read yet. ### Decision record `docs/decisions/2026-10-08-claude-actions-pr-review.md` (why a manual dispatch and no pull request trigger, how the head is handled, the model's fence and what was checked at runtime, cost of one run with the measured runs and the cap derivation, changed test contracts, alternatives, what would overturn it), with a dated R5 section: the symbolic-link step, the caps from measurement, the non-string tool entries, the closed flag list and the boundary tests, the mutants, and the follow-ups. `docs/decisions/2026-10-04-ci-least-privilege.md` gets a dated addendum naming the second job on the federation rule. ### Host evidence Not applicable: no file under `evidence/hosts/` changes. ### Checklist - [x] New/changed GitHub Actions are pinned to a full commit SHA with a version comment (no floating tags). - [x] New/changed workflows declare top-level `permissions: {}` and grant each job only what it needs (`contents: read`, or an explicitly justified addition). - [x] No secrets are printed, logged or committed; no new required secret was added without a documented owner. - [x] No new paid hosting, subscription or billing surface was introduced. - [x] Peer-owned untracked files and worktrees were preserved (not deleted, moved or overwritten). Activation is a separate step after landing: an owner sets `CLAUDE_PR_REVIEW_ENABLED` to `true`, then dispatches the workflow with a pull request number and head commit. Runs authenticate by workload identity federation and spend from the Console organization the four `ANTHROPIC_*` repository variables name; no Anthropic key is stored in GitHub. Each run is capped at 30 assistant turns and a $5 client budget, the owner's per-run spend limit on that organization. This PR and #892 both edit the write-grant inventory and the evidence manifest, so the second to land rebases once. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
8d5e7fe to
3a12894
Compare
claude-pr-toolkit-review.yml runs pr-test-analyzer and silent-failure-hunter from anthropics/claude-code's pr-review-toolkit plugin on one pull request head, through claude-code-action v1.0.247. Both agents list every behaviour and test change the description does not declare, with a confidence on every finding. It is the command center's J8 pre-cue step (W4 draft f), the same agents and settings that found six undeclared changes in #892 and two verifier defects in #900. The toolkit is checked out at anthropics/claude-code@602df92b, sparse to its plugin directory, moved outside the workspace, verified file by file against recorded SHA-256 values (git blob ids matched GitHub's at the pin), and loaded with --plugin-dir; the action's plugins and plugin_marketplaces inputs install unpinned code at run time and are not used. Sonnet 5.5 at max effort; Read, Glob, Grep and the Agent tool; CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS=0 so background agents are awaited; a $5 budget; a review time limit of 20 or 35 minutes by diff size. The bounds step counts the coordinator's own turns only, allows the budget plus a $2 allowance for the agents' last requests, requires both reports, uses an allow-list and names every unmet bound. Dispatch only, owner only, first attempt only, gated by CLAUDE_PR_TOOLKIT_ENABLED; no write scope beyond the OIDC token; nothing is posted to the pull request. 34 tests run the steps as written (without PyYAML too); eight weakened copies each fail them; actionlint and zizmor (pedantic) report nothing. openai/codex-action is recorded as WATCH with its pin and overturn condition. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ay is missing In the local J8 run on #894 the coordinator hit --max-budget-usd 5 (error_max_budget_usd) after both agents had handed back their full reports through SubagentHandback, so it never relayed them and the workflow would have withheld two complete reviews. The numbers and report steps now take the coordinator's relay when it carries both sections, and otherwise assemble the reports from each agent's own SubagentHandback message in the execution file (model-free); with neither, the relay text stays. The usage record says which source was used. All three J8 streams so far carry SubagentHandback calls. Three new tests cover a coordinator that stopped before relaying, a missing handback and a complete relay preferred to the handbacks (37 tests); three mutants of the fallback each fail them; actionlint and zizmor report nothing. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The command center approved a machine-readable trailer for the "defects
caught before a cue" ledger: each agent ends its report with one line,
J8-SUMMARY {"undeclared": <n>, "confidences": [...]}, so counts come
from the agents' own numbers and never from prose. The review itself is
unchanged. A test asserts the trailer in the prompt.
The sweep of the W4 drafts for the classes #894's pre-cue read found,
applied here: the empty paths array is expanded safely on bash before
4.4; a read step that succeeds without an execution file fails the
job; display_report and track_progress are asserted; the record states
that a public run's summary is public. 39 tests; actionlint and zizmor
report nothing.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
- R4, from the cc-native-practice lane's L3 read against claude-code-action v1.0.247 practice: the guard binds the ACTIONS_STEP_DEBUG and ACTIONS_RUNNER_DEBUG secrets and variables as booleans, the secret first, and refuses either. A test pins both bindings, and six weakened copies each fail it. - The publish step writes the reports with `jq -j`, so a 60,000-byte report is no longer announced as cut, as the J8 micro read of #892 found for the same step. A boundary test covers it. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…record and remove links from the head R5, quality and correctness only. The coordinator moves to Opus 5.5 at max with a $22 client budget; the cost bound is the budget times the measured overrun factor 1.10 ($24.20), re-derived after the first three runs. The numbers step reads every result record: each must be a success, or a budget stop after both agents handed back, and a budget stop at or under the bound publishes both reports. Publishing is gated on the numbers step's outcome. The coordinator must call both toolkit agents and no other (retries allowed), replayed on three J8 streams. A tool-list entry that is not a string is a forbidden tool. A model-free step removes symbolic links from the pull request head right after its checkout (GPT designated read of b80a6eb, P2). New tests pin claude_args and --settings exactly; the module goes from 41 to 56 tests. The record names every change and states that the hosted file ends at the first result record, so a run whose agents have not handed back by then fails closed. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…g, and share the cost-factor wording R6. The numbers step fails unless both called agents handed back (a handback counts only under the coordinator's Agent call for that agent, once per agent); relay headings no longer count anywhere (GPT designated read of 7aa2c12, P2). The report step publishes the two handbacks, pr-test-analyzer's then silent-failure-hunter's, and never the coordinator's relay; the coordinator prompt no longer asks for a relay and ends with one line, which saves its output tokens. Publishing is gated on !cancelled() and the numbers step's success, as in the other W4 workflows. The exact --settings pin compares canonical JSON, so 1 or 0 no longer passes for true or false (J8 R5 micro of #894, N1). The record adopts the shared cost-factor paragraph ($22 budget, $24.20 bound) and declares the eight R5 changes the J8 R5 micro read found undeclared. 57 tests; 49 of 49 weakened copies fail a test. Co-Authored-By: Claude Opus 5.5 <[email protected]>
3a12894 to
c3748c4
Compare
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c3748c49e7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| 'd369fd3946a814bb7a9d4f32e971722fe259e301878986bc6312d6f6c56014a8 agents/pr-test-analyzer.md' \ | ||
| 'fa9b0daec5a267e7e66435cc48b3328301fc9f70c3af259fe248881327a1babc agents/silent-failure-hunter.md' \ | ||
| '9435cc134fc72d56175f222894d401b0cf20f700d5bc0098c4257455314695ca .claude-plugin/plugin.json' | | ||
| sha256sum -c - |
There was a problem hiding this comment.
Pin a toolkit revision that passes plugin validation
When this workflow is enabled, the hash check accepts the pinned silent-failure-hunter.md even though its frontmatter is malformed: Anthropic reproduced that claude plugin validate rejects this agent and that the runtime drops all of its metadata, including its declared name and model (upstream report). The synthetic transcript tests never load the plugin, so every paid run can start with one of the two required reviewers in this degraded, unsupported state; pin a corrected upstream revision and run the upstream validator before accepting the checkout.
AGENTS.md reference: AGENTS.md:L5-L7
Useful? React with 👍 / 👎.
| echo '<pre>' | ||
| # iconv -c drops a character cut in half at the 60,000-byte mark; it still exits 1 for that, so the | ||
| # pipeline keeps its valid prefix instead of failing the step. | ||
| head -c 60000 "$report" | { iconv -f UTF-8 -t UTF-8 -c 2>/dev/null || true; } | sed -e 's/&/\&/g' -e 's/</\</g' -e 's/>/\>/g' |
There was a problem hiding this comment.
Preserve the machine-readable summaries past the display cap
When the two handbacks exceed 60,000 bytes, this prefix-only truncation drops the tail even though each required J8-SUMMARY is deliberately the final line of its report. The full handbacks are not uploaded elsewhere—the artifact contains only usage.json, which omits undeclared and confidences—so the run can pass all bounds while the command center permanently loses the counts and potentially an entire second report. Extract and retain both summary records separately, or preserve the complete reports outside the capped job-summary rendering.
Useful? React with 👍 / 👎.
| - The full diff from its merge base with main is in ${{ runner.temp }}/pr-toolkit/pr.diff, and the changed | ||
| files are listed in ${{ runner.temp }}/pr-toolkit/pr.stat. | ||
| - The pull request description is in ${{ runner.temp }}/pr-toolkit/pr-body.md. |
There was a problem hiding this comment.
Tell agents when the diff is path-scoped
When a maintainer supplies the optional paths input, the preceding diff step restricts both pr.diff and pr.stat to those paths, but this prompt still calls the result the full diff and asks for every undeclared change in the pull request. The published summary also omits the selected scope, so agents and downstream readers can mistake a partial review for complete PR coverage. Describe the path restriction in the prompt and carry it into the report metadata.
Useful? React with 👍 / 👎.
…ule for retired slots Answers the Codex review in GitHub of #1001 at ba6ab3a (four P2s) and the co-op's GPT first read at the same head (three P2s, the first three below); each was reproduced before the change. - Sentence units split at a full stop or a semicolon whatever follows, and at a line break: "This route was retired; the current default is claude-code-action." was one unit that said retired. - The generic phrases "in Actions", "through Actions" and "under WIF" count as the retired route only with Claude context, judged per leaf string; the named workflows, "through #909" and the managed app count everywhere. The routes the decision retains (every bullet of its "What remains" section, a catalog job in GitHub Actions, a non-model gate under WIF) are accepted. - The skill's entry point SKILL.md is scanned, and a test requires every file of the skill folder to be read by a check. - The one-default rule names the retired exception in slots.json, in the slots.md copy and in step 3 of SKILL.md (a retired row has none and says what replaced the route). - The ruled default text is pinned exactly (the read's measured boundary: the date changed on all three surfaces passed), and a check compares every row's Overturn when across slots.json, the index and the layer pages (one step wider than the review of #1005, where an edit left the index cell behind). Each finding fails its own test when its fix is undone alone, and 43 single-edit mutations are caught; the old guard accepts the four bad scenarios of the read and rejects its two legitimate ones. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…ew (#1001) * Retire the skill's Claude-in-GitHub rows and point them to local review The claude-native-practice skill still selected the route that docs/decisions/2026-10-10-retire-in-github-claude-review.md retired: the github-action row (anthropics/claude-code-action at a pin) and the Actions part of pr-review-pre-cue. - slots.json, slots.md and the layer pages: `retired` joins the status vocabulary; github-action becomes retired with a dated block (record, correction #141, scope, the three workflows, the former default, route, alternatives, evidence and notes as a snapshot, the revisit triggers); pr-review-pre-cue becomes default (scoped) and lists the routes available now (local Claude reads, GPT reads, Codex's GitHub review, the required verdict-review-gate) apart from the open #953 worker. - The scope is the owner's repositories (the command center's ruling of 2026-10-10), not "this repository": a hermetic A/B showed a session in another repository concluding that the action was still fine there. The same ruling retires the managed Claude Code Review GitHub App. - Stale texts the independent pre-read and the stricter check found: the release-tracking note (Dependabot as the action's tracker), the scheduled-tasks route (GitHub Actions schedules only for jobs that run no model) and its source mapping, the usage-cost-monitoring note (Actions under WIF), and the pre-cue source mapping (in Actions through #909). The pre-cue route no longer lists a read the record does not name. - tests/test_native_practice_retired_routes.py: no current row selects a retired route; a retired route is named elsewhere only in a sentence or list item that says it is retired; every line of slots.json, slots.md, the layer pages and cross-client.md is read (duplicates kept, the source mappings of live slots included); only the dated snapshots are exempt. Positive controls are the rows as they stood; the mutation table is in the pull request description. - The slot manifest of #923 is relabeled against the new slots.json bytes, so the exact-difference check of the provenance test runs again instead of skipping. - docs/decisions/2026-10-09-claude-code-native-practice.md: a dated pointer to the retirement, as #981 did for the older records. Source read for this change: anthropics/claude-code-action at 2dca132ff0e0c4094ce6048b422c6915a071210b, base-action/src/ parse-sdk-options.ts:283-284 (CLAUDE_CODE_ENTRYPOINT is set to claude-code-github-action). Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * Close the holes in the retirement guard and qualify the one-default rule for retired slots Answers the Codex review in GitHub of #1001 at ba6ab3a (four P2s) and the co-op's GPT first read at the same head (three P2s, the first three below); each was reproduced before the change. - Sentence units split at a full stop or a semicolon whatever follows, and at a line break: "This route was retired; the current default is claude-code-action." was one unit that said retired. - The generic phrases "in Actions", "through Actions" and "under WIF" count as the retired route only with Claude context, judged per leaf string; the named workflows, "through #909" and the managed app count everywhere. The routes the decision retains (every bullet of its "What remains" section, a catalog job in GitHub Actions, a non-model gate under WIF) are accepted. - The skill's entry point SKILL.md is scanned, and a test requires every file of the skill folder to be read by a check. - The one-default rule names the retired exception in slots.json, in the slots.md copy and in step 3 of SKILL.md (a retired row has none and says what replaced the route). - The ruled default text is pinned exactly (the read's measured boundary: the date changed on all three surfaces passed), and a check compares every row's Overturn when across slots.json, the index and the layer pages (one step wider than the review of #1005, where an edit left the index cell behind). Each finding fails its own test when its fix is undone alone, and 43 single-edit mutations are caught; the old guard accepts the four bad scenarios of the read and rejects its two legitimate ones. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * Bind the manifest label to the shipped slots file and close the remaining holes of the retirement guard Answers the CC's Claude first read of #1001 at 6e97620 (two P2s, six P3s) and Codex's second review in GitHub of the same head (two P2 threads); each was reproduced before the change. - P2-1: the slot manifest was labeled against the slots.json of ba6ab3a, so the exact-difference check of the provenance test skipped at the head (OK, skipped=1). The label is the sha256 of the shipped file again, the check no longer skips, and a new test fails when a change to slots.json does not carry its label (the CC's ruling: a silent skip is this defect's class, so it fails loud). - P2-2: a live source mapping (skills-library, primary_sources[5]) still ran `claude -p` "in a GitHub workflow ... plus a nightly cron"; the guard had no such form. It is reworded to say the run is retired, and "in a GitHub workflow" (also from, via, through, using, as) counts as the retired route when the sentence is about Claude. - P3-1: the Claude Code GitHub Action, /install-github-app and an @claude mention are the retired route under other names (claude-code-security-review stays out: a documented rejection). - P3-4: the Former default exemption of a layer page applies to a retired slot only. - P3-5: a retired slot's own source mapping is read, and now says retired. - P3-2: "e.g." and "i.e." no longer end a sentence, and wrapped lines of prose are one paragraph (the slots.md prose and SKILL.md were judged line by line). - P3-6: the manifest prose lists the fields beyond the exported ones that differ from the snapshot. - P3-7: the entry point directs ("a retired row has no default, so follow the replacement it names"), and the step-3 pin no longer fails with an IndexError on a renumbering. Each correction fails its own test when it is undone alone, and the mutation table is extended. Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * Judge retired-route prose one physical line at a time and close the guard's other gaps Answers the command center's Claude micro of #1001 at ef3f3e7 (P2-1, P3-1 to P3-5) and Codex's four threads of the same head (:43, :63, :122, :211). Each case was reproduced before the change: compare-guards-ef3f3e7b-vs-third-commit.txt runs 19 scenarios through the guard of ef3f3e7 and this one; the old guard is wrong on 13 (it accepts a retired route that is reintroduced), this one on none. - P2-1 (correction #160): joining wrapped lines into one paragraph let "Retired 2026-10-10" followed by a line that says the current default is claude-code-action pass on four prose surfaces (SKILL.md, slots.md, cross-client.md, layer-page preambles), the scenario Codex raised at ba6ab3a line 41. Prose is judged one physical line at a time again, a line break ends a unit, and the message of every line-judged surface states the rule (RULE; the legend of slots.md states it too and a test pins the two to each other). - P3-1 and Codex :43: three more names of the same route, all in the vendor's own text: the page's title "Claude Code GitHub Actions" and its "installs the Claude GitHub App" (code.claude.com/docs/en/github-actions.md, capture sha256 87b99aef, lines 5, 9, 33, 63-64), the mirror claude-code-base-action (anthropics/claude-code-action@2dca132ff0e0: .github/scripts/check_workflow_hardening.py:18, .github/workflows/sync-base-action.yml:1 and 43, CLAUDE.md:20) and the carrier named first ("Use the GitHub Action for Claude reviews", generic like "in Actions": it counts when the field or unit is about Claude). The families "in Actions" and "through Actions" take on, with, inside and via. - Codex :122 and the command center's P3-3 and P3-5: a unit that negates the retirement outright ("not retired", "never retired", "no longer retired", "isn't retired", "unretired") no longer counts as saying retired. The other half of that scenario, "was retired, but is now current", is the boundary the command center ruled (the guard is a tripwire for stale text; review reads what it accepts); the module names it. - Codex :211 and P3-3: a layer-page section with a label twice (a second Route or Overturn when bullet) is a violation, because the page-to-slots.json comparison keeps the first bullet only. - Codex :63 and P3-4: the target of every slots.md row is checked against <page>#<id>, from the layer of the slot. - The description line of SKILL.md names the topic that invokes the skill; it is pinned by sha256 to the text the routing was measured with and read as a topic, not as a route. - Scheduled-tasks (Codex on #1005, scheduling-supervision.md:9, and the command center's ruling): the route no longer bans every model-backed job. The retirement record (docs/decisions/2026-10-10-retire-in-github-claude-review.md:5-6, :43) retires Claude review inside GitHub and keeps Codex's GitHub review, so the route says GitHub Actions schedules repository jobs and a scheduled Claude task runs locally; a test pins the wording on slots.json, the layer page and the alternatives. The label of claude-native-slot-manifest.json equals the sha256 of the shipped slots.json again (relabel last); test_the_label_names_the_shipped_slots_file and test_no_other_exported_field_differs (no skip) are unchanged from ef3f3e7. 72 of 72 single-edit mutants of the guard and its data are caught and 5 controls pass (mutations-v4-forward-commit-3.md). Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * Align the manifest's scheduled-tasks note, state the rule in every explaining unit, split units after a closing delimiter and pin the scope of the name patterns The final fix round of #1001 under the command center's convergence ruling: its Claude micro at efaa077 (blockers B1 to B3, P3-2) and its adjudication of 2026-10-11T13:50Z of Codex's third review. Each item was reproduced on that head before the change. - claude-native-slot-manifest.json:47 (Codex P2, B1, a text of this PR that is false at the head): the note on the differences from the snapshot still said the route of scheduled-tasks was narrowed "to jobs that run no model", while the shipped route says that GitHub Actions schedules repository jobs, that Claude runs inside GitHub are retired (2026-10-10, docs/decisions/2026-10-10-retire-in-github-claude-review.md:5-6, :43) and that a scheduled Claude task runs locally. The note says so now, and test_the_scheduled_tasks_route_retires_the_claude_routes_and_not_every_model_backed_job reads the manifest as well as the route (fails with the note undone: M79). - B2: the module claims that the message of every line-judged surface states the rule, and it did not for the labelled bullets of a layer page or for the Overturn when cell of an index row. Every explaining unit states it now (a selecting field names no unit and states nothing); the three expected messages carry the rule and test_every_explaining_unit_of_a_layer_page_and_of_the_index_states_the_rule_and_a_selecting_field_does_not pins all three kinds (mutants M83 to M85). - tests/test_native_practice_retired_routes.py:83 (Codex P2, P3-2): a closing quote, parenthesis or bracket between the full stop and the white space kept `The route was "retired." The current default is claude-code-action.` in one unit that said retired. The sentence pattern takes closing delimiters (the curly quotes are built with chr(), so the file stays ASCII); test_a_closing_quote_parenthesis_or_bracket_after_the_full_stop_does_not_join_two_claims fails without it (M78, M82) and keeps `claude-code-action was retired (see the record.) No row selects it.` accepted (control C6). - :48 (Codex P2, workflow-first phrasing) is a follow-up and the class closes (rule 4 of the convergence ruling): the module states that the guard pins the named retired artifacts and the retired slot rows, that generic phrasing of a Claude run in a GitHub workflow cannot be enumerated and is out of its scope (the retirement record governs it), and test_generic_workflow_phrasing_is_out_of_scope_and_the_module_says_so pins the boundary. No pattern is added. - B3 is the description: the two earlier command blocks name the 37 modules they ran (the 46 of the first block without the nine tests.test_local_pages_* modules). 80 single-edit mutants of the guard and its data are caught and 6 controls pass (mutations_retire_v6.py). Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * Close the sentence-splitter class by documentation, pin the module's ASCII and the scope list The command center's confirmation micro of #1001 at bdf60fc (research-cc-role-20261008/pr1001-claude-micro-bdf60fc6.md: blockers 1 and 2, P3s) and its correction #163 of 2026-10-11T16:18Z on Codex's three threads of 15:39:21Z. No code of the guard changes in this commit. - Blocker 2: the module said that it stays ASCII and held one U+2026 in a comment (added by efaa077). It is three full stops now, and test_the_source_of_this_module_is_printable_ascii pins the claim; it fails on the module of bdf60fc (['U+2026'] != []). - Blocker 1 (the pull request description, and this message corrects the message of bdf60fc): the earlier statement that the 37 modules ran "without its nine tests.test_local_pages_* modules" was false. They are the first 37 modules of the 46-module list, and the nine left out are its last nine (tests.test_local_pages_fleet_execution to tests.test_local_pages_workstation, 144 tests, run and counted); 29 of the 46 are local-pages modules. - Codex :92 (an exclamation mark or a question mark does not end a unit) is not patched: the sentence splitter has drawn findings in five consecutive rounds (the lowercase clause, "e.g.", the joined lines, the closers, the exclamation mark and the question mark) and was patched four times, so rule 4 of the convergence ruling closes the class the way line 48 closed. The module docstring states exactly what the guard accepts as a unit end and that review reads everything else; test_the_unit_ends_and_the_scope_of_the_name_patterns_are_documented pins the sentence and the behavior, and fails on the module of bdf60fc, where the sentence is missing. A pinned upstream sentence segmenter is a research-queue item (rule 8, layer instructions-skills). - Codex :51 (a carrier-first name of the managed Code Review app) is confirmed within the closed class: the same scope paragraph says that reordered phrasings of a pinned artifact are out of the guard's scope. - P3-1: the scope list named 11 of the 17 patterns; it names all 17 now, and test_the_scope_of_the_name_patterns_names_every_pattern compares the list itself with PATTERNS (it fails on the module of bdf60fc). P3-2: the right single quotation mark closer is tested (its mutant survived). P3-3: the comment at the sentence pattern says what the code does. - Codex :170 (retired.on is checked for its shape only) is a follow-up under rule 5: the shipped value is the date of the record, so 2099-12-31 is a mutant and not a reproduced failure of the shipped state. 88 single-edit mutants of the guard and its data are caught and 6 controls pass (mutations_retire_v9.py). Co-Authored-By: Claude Sonnet 5.5 <[email protected]> --------- Co-authored-by: Claude Sonnet 5.5 <[email protected]>
Scope
claude-pr-toolkit-review.yml, runs two of Anthropic's pr-review-toolkit agents,pr-test-analyzerandsilent-failure-hunter, on one pull request head throughanthropics/claude-code-actionv1.0.247. Both list every behaviour and test change the description does not declare. A maintainer dispatches it by hand; it is the command center's pre-cue step J8 (W4 draft f). The toolkit comes fromanthropics/claude-codeat a full commit, verified file by file and loaded with--plugin-dir; the action's unpinnedpluginsandplugin_marketplacesinputs are not used. It runs only whileCLAUDE_PR_TOOLKIT_ENABLEDistrue, and only on the first attempt of a run, so merging this starts nothing.--model claude-opus-5-5at--effort max, by the owner's direction, relayed by the command center on 2026-10-09, that every Claude Action runs on Opus 5.5 at max. The command center's caps of the same day:--max-turns 12for the coordinator (unchanged),--max-budget-usd 22, and a cost bound of $24.20 in the numbers step (the budget × 1.10). The expected cost of a run is about $6.6 to $10.9. The J8 runs this PR cites ran on Sonnet 5.5 at a $5 budget; no run on Opus 5.5 or at $22 exists yet.anthropics/claude-code-actionv1.0.247 practice): the guard binds(secrets.ACTIONS_STEP_DEBUG || vars.ACTIONS_STEP_DEBUG) == 'true'and(secrets.ACTIONS_RUNNER_DEBUG || vars.ACTIONS_RUNNER_DEBUG) == 'true'as booleans and refuses either before any token is requested; a test pins both bindings, and six weakened copies each fail at least one test. This is hardening: the action step already pinsACTIONS_STEP_DEBUG: 'false', which is what the action's full-output switch reads. zizmor's auditor persona reportssecrets-outside-envon the two reads; CI runs the regular persona with--no-ignores, which reports nothing. At R4 these lines were byte-identical in Bound the harness audit and move it to claude-code-action v1.0.247 #892, Add an on-demand, read-only pull request review by the Claude action #894, Add an on-demand Claude security review and a model-free flag on risky paths #895, Add a weekly lane triage: the model proposes, a model-free job applies #896 and Add a pre-cue read by Anthropic's pr-review-toolkit agents in Actions #909; R5 leaves Add a pre-cue read by Anthropic's pr-review-toolkit agents in Actions #909's unchanged.jq -r, which adds a newline, so a 60,000-byte report was announced as cut. It writes withjq -jnow; a test publishes exactly 60,000 and 60,001 bytes, and a copy that writes withjq -rfails it.b80a6eb8, two later audits, a GPT designated read of Add an on-demand Claude security review and a model-free flag on risky paths #895 and the caps decision):id: numbers) reads every result record in the execution file, not only the last, so an idle success at the end cannot hide an earlier error. Each record must be a success withoutis_error, or a budget stop (error_max_budget_usd) after both agents handed back; any other error subtype fails. The cost is the highest any record reports, andmodelscomes from that record.steps.numbers.outcome == 'success'instead ofsuccess(), because the action fails its own step on a first result that is not a plain success.0 of 2 agent reportsand publishes nothing. The first hosted runs measure it.non-string tool entry, instead of being dropped.usage.jsonfields:complete,agents_called,handbacks,result_subtypesandlower_bound_models.Remove symbolic links from the pull request head, runsfind pr-head -path pr-head/.git -prune -o -type l -exec rm -f {} +right after the head checkout.claude_argsline and the whole--settingsJSON.model: inherit, so Opus 5.5; nested agents run on the model their own call names, and pinning that is follow-up F-08.origin-rewrite sentence and the follow-ups line.docs/github-automation.mdnow says Opus 5.5 and $22.72e1f9437aa2c128lane:foundation.github/workflows/claude-pr-toolkit-review.yml(new),tests/test_claude_pr_toolkit_review_workflow.py(new),tests/test_workflow_policy.py(one inventory entry, one exemption),tests/test_workflow_security_coverage.py(coverage set, offline zizmor test),docs/decisions/2026-10-09-claude-actions-pr-toolkit-review.md(new),docs/decisions/2026-10-04-ci-least-privilege.md(dated addendum),docs/github-automation.md(one entry),manifests/evidence.jsonR6 (2026-10-09), at 8d5e7fe
reproduced). The numbers step fails with
<n> of 2 agent reports handed backunless both called agents handed back:SubagentHandbackunder the coordinator's Agent call for that agent, once per agent;pending text, two calls with no handback, one handback with a headed relay, one handback carrying the other
agent's heading, and one agent's two handbacks after a retry.
silent-failure-hunter's, never the coordinator's relay. A relay is model text and can be a template or a paraphrase.
which saves the relay's Opus output tokens on every run. The $24.20 cost bound stays as an upper bound.
!cancelled() && steps.numbers.outcome == 'success', matching the other W4 workflows, so acancelled run publishes nothing.
--settingspin compares canonical JSON, so1or0no longer passes wheretrueorfalseis pinned (J8 R5 micro read of Add an on-demand, read-only pull request review by the Claude action #894, N1).on a $5 budget ($5.46, $5.33, $5.16, $5.0007), re-derived after this workflow's first three hosted runs: $22 → $24.20.
(b80a6eb..7aa2c12, Opus 5.5 at max) found undeclared.
validate.pyandevidence_manifest.py --checkpass (10,904 files);--no-ignores) are clean;SOTA sources
2dca132ff0e0c4094ce6048b422c6915a071210b:action.yml: the federation inputs,claude_args,pluginsandplugin_marketplaces(lines 160-167),display_reportandtrack_progress;base-action/src/parse-sdk-options.ts:--plugin-diris passed through to the CLI.base-action/src/run-claude-sdk.ts(R5), read at v1.0.246 (38c80c1), a file the v1.0.247 compare leaves unchanged:src/entrypoints/run.ts(R5):setExecutionFileOutputIfPresentin its catch.src/github/operations/git-config.ts:129-134, called fromsrc/modes/agent/index.ts:52-61(R5): in agent mode the action rewrites the checkout'soriginURL with the job token.602df92bf481ed904533e95c09f740f40aab5aed,plugins/pr-review-toolkit. The two agent files andplugin.jsonmatch GitHub's git blob ids at that commit.claude --help:--plugin-dir,--tools,--restricted,--permission-prompts,--effortand--max-budget-usd. The headless documentation: background subagents andCLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS.repository,ref,pathandsparse-checkout), and step-leveltimeout-minuteswith thestepscontext. The publish gate'ssteps.<id>.outcome == 'success'follows the patternvalidate.ymlandnative-offhost-app-state.ymlalready use (R5).bdf19a4a223ec2549a3e2274a0cf61556bc07675,action.ymlinputs. This backs the record's WATCH entry; the action is not adopted.Evidence-class table
native_proven(installed client, billed to a second key; not through the action)synthetictests/test_claude_pr_toolkit_review_workflow.py, 56 tests at 7aa2c12. Each step's shell is taken from the workflow file and run; the tests run without PyYAML toosynthetic(from the measured J8 #894 stop)SubagentHandbackmessage when the relay lacks a section; 3 tests, plus R5's budget-stop replaysmodelscomes from the costliest recordsynthetictest_in_the_measured_order_a_budget_stop_after_both_handbacks_passes_wherever_the_file_ends,test_a_budget_stop_after_both_handbacks_publishes_up_to_the_cost_bound($22.40 and $24.20 pass; $24.21 fails),test_a_budget_stop_without_both_handbacks_and_any_other_error_record_fail,test_the_cost_bound_is_the_budget_times_the_measured_overrun_factor,test_the_model_usage_is_the_costliest_result_recordssource_reviewbase-action/src/run-claude-sdk.tslines 190-209 at v1.0.246, unchanged in the v1.0.247 comparelocal_integration(the step's shell replayed on J8 streams recorded on 2026-10-09; the streams stay outside the repository)local_integrationnative_proven(J8 receipts, as the command center reports them)syntheticmain, runtimepluginsinput added, a $70 cost bound). At R5: 36 weakened copies, all 36 fail at least one test (mutant script kept outside the repository)local_integrationactionlint1.17.0 exits 0.zizmor1.30.1, offline, pedantic and regular with--no-ignores: no findings (2 suppressed: the R4 guard'ssecrets-outside-env, auditor persona only, unchanged since R4)local_integration0 of 2 agent reportsand publishes nothingLocal commands run
Decision record
docs/decisions/2026-10-09-claude-actions-pr-toolkit-review.md: why the pinned checkout is used instead of thepluginsinput, the behaviour by name, the bounds, the cost of one run with the caps' derivation and the 1.10 factor, the alternatives (including openai/codex-action as WATCH with its overturn condition) and what would overturn it. Its dated sections name every change of R4, of the cap notice and of R5 ("Round 5: result records, caps, link removal and exact pins (2026-10-09)").Host evidence
Not applicable: no file under
evidence/hosts/changes.Checklist
permissions: {}and grant each job only what it needs (contents: read,pull-requests: read,id-token: writefor federation, reviewed in the inventory).🤖 Generated with Claude Code