Skip to content

Commit 2f75964

Browse files
kjellyclaude
andcommitted
fix(skills): pr-review-merge requires the mandated actual-run evidence
review_gate.py ignored target_validation, so a PR changing playbooks, specs or contracts could merge with no actual-run record. The gate now reads .github/pr-evidence.json at the PR's base SHA (GitHub contents API, never the head) and checks target_validation with issue-to-pr's repo_evidence.py, loaded by path; the evidence-only delta comes from GitHub compare. Missing, malformed, blocked, uncovered or unverifiable records are WAITING; DRAFT still comes only from the reviewer's P1. The manifest gains required_paths (playbooks, verification specs, contracts), for which not-needed is not accepted. GitHub answers in the tests are read-only gh api captures from kjelly/pilot. Integration tests lock a per-test PRM_STATE_DIR instead of ~/.cache. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 40a9b98 commit 2f75964

18 files changed

Lines changed: 887 additions & 25 deletions

File tree

‎.agents/skills/pr-review-merge/PROJECT-CHANGES.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,14 +13,15 @@ below, then run `make skill-test` and `go test ./cmd/pilot/cmd -run TestAgentSki
1313
| `SKILL.md` | "Project runtime configuration" and "Pilot: repository rules the review and the merge must respect" sections (merge commits, no branch rules, history reference, mandated actual-run evidence as a P1, cost, serial merges); merge-policy sentence in §1; `--method merge` in the §4 example |
1414
| `platforms/{claude,codex,agy}/agents/prm-{reviewer,dependency}.*` | one prompt body per role on every platform (upstream shipped three different texts); pilot lines (history reference, `target_validation`, missing actual-run record is a blocking P1, test scope); Claude `tools`/`model`/`effort` and Codex `model`/`model_reasoning_effort` pins |
1515
| `scripts/apply_decision.py` | `--method` defaults to `merge` (upstream: `squash`); locked by `tests/test_project_defaults.py` |
16-
| `references/evidence-contract.md` | `target_validation` object in the example and its rules |
17-
| `references/quality-gate.md` | DRAFT row for missing mandated actual-run evidence; merge command/method wording |
16+
| `references/evidence-contract.md` | `target_validation` object in the example and its rules (enforced by the gate, see the last row) |
17+
| `references/quality-gate.md` | WAITING row for a missing mandated actual-run record (gate) and DRAFT row for the reviewer's P1 on it; merge command/method wording |
1818
| `references/workflow.md` | step 3 (history reference + `.github/pr-evidence.json` check), renumbered steps, `Target validation` line in the reviewer return |
1919
| `references/platforms.md` | rewritten for the repository-local layout (no installer, no global paths) |
2020
| `tests/test_project_defaults.py` | new |
2121
| `scripts/install.sh`, `tests/test_installer.py` | removed: the skill is repository-local, nothing installs it elsewhere |
2222
| `README.md` | install section replaced by the repository layout; examples target `kjelly/pilot`; default merge method wording |
2323
| `scripts/review_gate.py`, `tests/test_gate.py`, `tests/test_pr_ready_handoff.py` (new), `SKILL.md` ("Hand-off with `pr-ready`", "Disputed findings", §2, §3 table), `references/{evidence-contract,quality-gate,workflow}.md`, `platforms/{claude,codex,agy}/agents/prm-reviewer.*` | no Draft/Ready loop with `pr-ready`: `CHANGES_REQUESTED` and reviewer-confirmed `unrelated_failing_checks` decide `WAITING` instead of `DRAFT` (upstream: `DRAFT`); a finding that repeats a disputed claim carries `disputed_reply_url` and is `DRAFT` only with `new_evidence`, otherwise `WAITING` for a person; severity `P4` accepted (shared rubric); the hand-off test runs both gates over the same GitHub states |
24+
| `scripts/review_gate.py`, `scripts/gh_runtime.py`, `tests/test_target_validation.py` (new), `tests/fixtures/github_api.json` (new), `tests/{test_gate,test_integration,test_pr_ready_handoff}.py`, `SKILL.md` ("Mandated actual-run evidence", hand-off, §3 table), `README.md`, `references/{evidence-contract,quality-gate,workflow}.md`, `platforms/{claude,codex,agy}/agents/prm-reviewer.*` | the gate enforces the actual-run record of the base branch's `.github/pr-evidence.json`: manifest read at the base SHA via the GitHub contents API, `target_validation` checked with issue-to-pr's `scripts/repo_evidence.py` (loaded by path; `RemoteContext` takes the evidence-only delta from GitHub `compare`), every gap `WAITING`; the manifest's `required_paths` (new key, only this gate reads it) refuse `not-needed`; integration tests use a per-test `PRM_STATE_DIR` |
2425

2526
Files not listed here are believed unchanged from upstream 1.0.0; verify against the upstream
2627
archive when upgrading.

‎.agents/skills/pr-review-merge/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@ $pr-review-merge 審查 kjelly/pilot 所有非 Draft PR;合格則 merge,不
3939

4040
**參數及預設策略:**最多 2 個獨立 Reviewer Subagents 並行、Merge 串行、預設 merge commit(本 repo 慣例;`apply_decision.py --method` 預設 `merge`)、禁止管理員繞過/Auto-merge/刪除分支(GitHub 的 deleteBranchOnMerge 會自己刪)。CI Pending、等待 Approval 或 Merge Queue 為 `WAITING`,不自動轉 Draft;`CHANGES_REQUESTED`(等原 reviewer 重新審查)、Reviewer 確認與本 PR 無關而失敗的 check、沒有新證據就重提已被反駁的 finding,也是 `WAITING`,否則 pr-ready 會再把它轉回 Ready。重大安全/資料毀損風險若需要額外人類簽核,亦保持 WAITING。
4141

42+
**Actual-run 紀錄:**base branch 的 `.github/pr-evidence.json` 要求 `actual-run` 紀錄時,`review_gate.py` 以 GitHub API 在 PR 的 **base SHA** 讀這份 manifest(不讀 head,PR 不能放寬自己的要求),並檢查 reviewer evidence 的 `target_validation`:缺少或格式錯、`blocked`、PR 改到 manifest `required_paths`(`playbooks/**`、`docs/verification/**`、`contracts/**`)卻寫 `not-needed`、或 `required` 卻沒有 HEAD 上已提交且寫明 tested revision 的 `docs/evidence/**` 紀錄(tested revision 等於 HEAD,或是 HEAD 的祖先且之後只改 evidence-only 檔案),一律 `WAITING`,不會 merge。manifest 格式、glob 與 evidence-only delta 規則沿用 issue-to-pr 的 `scripts/repo_evidence.py`(依路徑載入)。轉 Draft 仍只來自 reviewer 對缺紀錄開的 blocking `P1` finding。
43+
4244
## 內部安全檢查(供 Agent 呼叫)
4345

4446
```bash
@@ -65,7 +67,7 @@ python3 scripts/apply_decision.py --repo OWNER/REPO --pr 123 --evidence /safe/st
6567
python3 -m unittest discover -s tests -v
6668
```
6769

68-
測試使用模擬 `gh`,**不碰真實 GitHub**。必須在自己的 repository 上做一次 end-to-end 驗證,才能確認平台版本、GitHub 權限、branch rules 和 Agent subagent runtime 均正常。
70+
測試使用模擬 `gh`,**不碰真實 GitHub**;contents/compare API 的回應取自 `tests/fixtures/github_api.json`(對 kjelly/pilot 唯讀 `gh api` 的實際擷取)。`apply_decision.py` 的寫入鎖在測試中指向每個測試自己的暫存 `PRM_STATE_DIR`,不碰 `~/.cache/pr-review-merge`,兩個同時跑的 `make skill-test` 不會互搶。必須在自己的 repository 上做一次 end-to-end 驗證,才能確認平台版本、GitHub 權限、branch rules 和 Agent subagent runtime 均正常。
6971

7072
## 與其他 Skill 的關係
7173

‎.agents/skills/pr-review-merge/SKILL.md‎

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -51,20 +51,27 @@ definitions shipped beside it; there is no personal installation. Codex invocati
5151
- **Mandated actual-run evidence.** `.github/pr-evidence.json` on the base branch
5252
declares `docs/evidence/**` as evidence-only and requires an `actual-run` record
5353
for any change to playbooks, specs, contracts, inventory handling or deploy
54-
orchestration (AGENTS.md §1.4/§1.5). A PR that needs one and has no
55-
`docs/evidence/<feature>/<date>-<tested-revision>.md` covering the reviewed HEAD
56-
(or an evidence-only delta from the tested revision) carries a blocking `P1`
57-
finding at the changed playbook/spec path: that is a confirmed violation of a hard
58-
repository rule, not a `WAITING` gap. Docs-, skill- or pure-Go changes proven by
59-
native-tool tests need no VM run; the reviewer says so in `target_validation`.
54+
orchestration (AGENTS.md §1.4/§1.5). `review_gate.py` enforces it: it reads the
55+
manifest at the PR's base SHA through the GitHub API (never the head, so a PR cannot
56+
weaken its own requirement) and checks `target_validation` with issue-to-pr's
57+
`repo_evidence.py` rules. Missing or malformed, `blocked`, `not-needed` on a PR that
58+
changes the manifest's `required_paths` (`playbooks/**`, `docs/verification/**`,
59+
`contracts/**`), or `required` without a committed `docs/evidence/**` record that names
60+
a tested revision covering HEAD (same SHA, or an ancestor whose delta to HEAD is
61+
evidence-only) is `WAITING`, never `MERGE` (`references/evidence-contract.md`). The
62+
reviewer still files a blocking `P1` at the changed path when a change needs a record
63+
and has none: that is a confirmed violation of a hard repository rule and the only way
64+
this becomes `DRAFT`. Docs-, skill- or pure-Go changes proven by native-tool tests need
65+
no VM run; the reviewer says so in `target_validation` with the reason.
6066
- **Cost.** The full Go suite takes about 45 minutes in CI. Reviewers run the
6167
packages the diff touches (and `make skill-test` for `.agents/skills` changes)
6268
locally and rely on the `Go test` check for the whole suite.
6369
- **Serial merges.** After each merge every remaining PR's base moves; re-fetch
6470
base/HEAD and arrange a new independent review before acting on the next one.
6571
- **Hand-off with `pr-ready`.** `pr-ready` marks a PR Ready only when every review
6672
thread is resolved; it may leave a reviewer-confirmed unrelated failing check, a
67-
`CHANGES_REQUESTED` review awaiting re-review, and findings it disputed on the PR.
73+
`CHANGES_REQUESTED` review awaiting re-review, findings it disputed on the PR, and
74+
(its `pr_gate.py` does not read `.github/pr-evidence.json`) no actual-run record.
6875
None of these is a reason to return the PR to Draft (that only starts another
6976
round): the gate decides `WAITING` for them and never `MERGE`.
7077
`tests/test_pr_ready_handoff.py` runs both gates over the same GitHub states and
@@ -95,9 +102,9 @@ definitions shipped beside it; there is no personal installation. Codex invocati
95102

96103
| Decision | Conditions | GitHub action |
97104
|---|---|---|
98-
| `MERGE` | Valid independent reviewed HEAD/Base; zero P0–P2 blockers; relevant HEAD-SHA-bound tests succeed; CI all passed (or narrowly justified no-CI policy); required approvals and rules satisfied; no unresolved threads/dependencies; current mergeable and safe | `gh pr merge --match-head-commit <SHA>`; verify `MERGED` |
105+
| `MERGE` | Valid independent reviewed HEAD/Base; zero P0–P2 blockers; relevant HEAD-SHA-bound tests succeed; the base manifest's required record covers HEAD (or is reasoned `not-needed`); CI all passed (or narrowly justified no-CI policy); required approvals and rules satisfied; no unresolved threads/dependencies; current mergeable and safe | `gh pr merge --match-head-commit <SHA>`; verify `MERGED` |
99106
| `DRAFT` | Confirmed blocking defect, failed CI not confirmed unrelated, failing tests, substantive review finding (a repeated disputed one only with new evidence), merge conflict or unresolved review thread | Post actionable findings and `gh pr ready --undo` |
100-
| `WAITING` | Pending CI, pending required approval, `CHANGES_REQUESTED` awaiting re-review, a failing check the reviewer confirmed unrelated to the PR, a disputed finding repeated without new evidence, unknown GitHub status, evidence incomplete, HEAD/Base changed, merge queue, missing access or unmet dependency | Leave Ready **unchanged**; report blocker and recommended next action |
107+
| `WAITING` | Pending CI, pending required approval, `CHANGES_REQUESTED` awaiting re-review, a failing check the reviewer confirmed unrelated to the PR, a disputed finding repeated without new evidence, the mandated actual-run record missing or not covering HEAD (no reviewer finding), unknown GitHub status, evidence incomplete, HEAD/Base changed, merge queue, missing access or unmet dependency | Leave Ready **unchanged**; report blocker and recommended next action |
101108
| `SKIP` | Already Draft, closed or merged | No action |
102109

103110
`WAITING` is intentional: lack of evidence or a transient pending check is not proof the code is defective. **Do not automatically downgrade in these situations**. If the PR has *any confirmed* blocking defect, classify `DRAFT` even if some checks are pending.

‎.agents/skills/pr-review-merge/platforms/agy/agents/prm-reviewer.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ Read the Root-supplied pilot_history_checks reference, the finding ledger and th
1616
Read the full diff against the current base, the affected source and shared includes, the linked Issue and acceptance criteria, review comments, CI and tests.
1717
Root gives you each earlier finding that was disputed on the PR with the reply disputing it; judge each claim yourself from the code. Repeat a disputed claim only with evidence the reply does not answer, and then set disputed_reply_url to the reply and new_evidence to that evidence. A failing CI check goes in unrelated_failing_checks only with evidence that it fails the same way without this diff (for example on the base); that never makes the PR mergeable.
1818
Assess requirements, correctness, architecture, security, concurrency, edge cases, semantic regression, migration/rollback risk and test adequacy; distinguish executed tests, skipped tests and structural checks.
19-
Record target_validation = required | not-needed | blocked with the invariant it rests on. A change that needs a candidate-bound actual-run record (playbooks, specs, contracts, inventory handling, deploy orchestration) and has none covering the reviewed HEAD or an evidence-only delta is a blocking P1 finding at the changed path, not a verbal reservation.
19+
Record target_validation: decision required | not-needed | blocked, a reason of 30+ characters with the invariant it rests on, and for required the committed docs/evidence artifact and the full 40-hex tested_revision that artifact names; the gate checks these against the base branch's manifest and never accepts not-needed for its required_paths. A change that needs a candidate-bound actual-run record (playbooks, specs, contracts, inventory handling, deploy orchestration) and has none covering the reviewed HEAD or an evidence-only delta is a blocking P1 finding at the changed path, not a verbal reservation.
2020
Use safe read-only commands and trusted test commands in an isolated worktree, scoped to the packages the diff touches; never modify source, branches or PR state, never comment, approve or merge.
2121
Return the compact structure of references/evidence-contract.md: exact headRefOid and baseRefOid, verdict PASS | BLOCK | NEEDS_MORE_EVIDENCE, P0–P4 findings with path:line, evidence, reproduction and remediation (origin judged against the PR base per the shared severity rubric), explicit coverage, test results with links, dependency status, target_validation and scope.
2222
Never invent test success or call a partial diff a complete review; raw diffs and logs stay in separate artifacts. Root owns decisions and GitHub state.

‎.agents/skills/pr-review-merge/platforms/claude/agents/prm-reviewer.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Read the Root-supplied pilot_history_checks reference, the finding ledger and th
1111
Read the full diff against the current base, the affected source and shared includes, the linked Issue and acceptance criteria, review comments, CI and tests.
1212
Root gives you each earlier finding that was disputed on the PR with the reply disputing it; judge each claim yourself from the code. Repeat a disputed claim only with evidence the reply does not answer, and then set disputed_reply_url to the reply and new_evidence to that evidence. A failing CI check goes in unrelated_failing_checks only with evidence that it fails the same way without this diff (for example on the base); that never makes the PR mergeable.
1313
Assess requirements, correctness, architecture, security, concurrency, edge cases, semantic regression, migration/rollback risk and test adequacy; distinguish executed tests, skipped tests and structural checks.
14-
Record target_validation = required | not-needed | blocked with the invariant it rests on. A change that needs a candidate-bound actual-run record (playbooks, specs, contracts, inventory handling, deploy orchestration) and has none covering the reviewed HEAD or an evidence-only delta is a blocking P1 finding at the changed path, not a verbal reservation.
14+
Record target_validation: decision required | not-needed | blocked, a reason of 30+ characters with the invariant it rests on, and for required the committed docs/evidence artifact and the full 40-hex tested_revision that artifact names; the gate checks these against the base branch's manifest and never accepts not-needed for its required_paths. A change that needs a candidate-bound actual-run record (playbooks, specs, contracts, inventory handling, deploy orchestration) and has none covering the reviewed HEAD or an evidence-only delta is a blocking P1 finding at the changed path, not a verbal reservation.
1515
Use safe read-only commands and trusted test commands in an isolated worktree, scoped to the packages the diff touches; never modify source, branches or PR state, never comment, approve or merge.
1616
Return the compact structure of references/evidence-contract.md: exact headRefOid and baseRefOid, verdict PASS | BLOCK | NEEDS_MORE_EVIDENCE, P0–P4 findings with path:line, evidence, reproduction and remediation (origin judged against the PR base per the shared severity rubric), explicit coverage, test results with links, dependency status, target_validation and scope.
1717
Never invent test success or call a partial diff a complete review; raw diffs and logs stay in separate artifacts. Root owns decisions and GitHub state.

‎.agents/skills/pr-review-merge/platforms/codex/agents/prm-reviewer.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ Read the Root-supplied pilot_history_checks reference, the finding ledger and th
88
Read the full diff against the current base, the affected source and shared includes, the linked Issue and acceptance criteria, review comments, CI and tests.
99
Root gives you each earlier finding that was disputed on the PR with the reply disputing it; judge each claim yourself from the code. Repeat a disputed claim only with evidence the reply does not answer, and then set disputed_reply_url to the reply and new_evidence to that evidence. A failing CI check goes in unrelated_failing_checks only with evidence that it fails the same way without this diff (for example on the base); that never makes the PR mergeable.
1010
Assess requirements, correctness, architecture, security, concurrency, edge cases, semantic regression, migration/rollback risk and test adequacy; distinguish executed tests, skipped tests and structural checks.
11-
Record target_validation = required | not-needed | blocked with the invariant it rests on. A change that needs a candidate-bound actual-run record (playbooks, specs, contracts, inventory handling, deploy orchestration) and has none covering the reviewed HEAD or an evidence-only delta is a blocking P1 finding at the changed path, not a verbal reservation.
11+
Record target_validation: decision required | not-needed | blocked, a reason of 30+ characters with the invariant it rests on, and for required the committed docs/evidence artifact and the full 40-hex tested_revision that artifact names; the gate checks these against the base branch's manifest and never accepts not-needed for its required_paths. A change that needs a candidate-bound actual-run record (playbooks, specs, contracts, inventory handling, deploy orchestration) and has none covering the reviewed HEAD or an evidence-only delta is a blocking P1 finding at the changed path, not a verbal reservation.
1212
Use safe read-only commands and trusted test commands in an isolated worktree, scoped to the packages the diff touches; never modify source, branches or PR state, never comment, approve or merge.
1313
Return the compact structure of references/evidence-contract.md: exact headRefOid and baseRefOid, verdict PASS | BLOCK | NEEDS_MORE_EVIDENCE, P0–P4 findings with path:line, evidence, reproduction and remediation (origin judged against the PR base per the shared severity rubric), explicit coverage, test results with links, dependency status, target_validation and scope.
1414
Never invent test success or call a partial diff a complete review; raw diffs and logs stay in separate artifacts. Root owns decisions and GitHub state.

0 commit comments

Comments
 (0)