Skip to content

Commit c66a5bf

Browse files
kjellyclaude
andcommitted
fix(skills): issue-to-pr mark_ready refuses open review feedback
mark_ready.py moved a Draft to Ready without looking at review threads or the review decision. A PR with an unresolved thread is then returned to Draft by pr-review-merge, and a CHANGES_REQUESTED review was simply ignored, although Ready is meant to say nothing is left to do. - mark_ready reads every page of the PR's review threads (GraphQL, query and answers captured from gh 2.103.3: #201 with none, cli/cli#13013 with one unresolved of three) and refuses Ready while any is unresolved; a lookup that fails or is malformed blocks. - reviewDecision CHANGES_REQUESTED blocks and points at pr-ready, which owns review feedback on a Draft (fix, or dispute with a reply); mark_ready does not triage it. "" (no review required), REVIEW_REQUIRED and APPROVED do not block. Two P4s from the #201 review, both already fail-closed: - a mistyped issue_resolution such as "Partial" is now reported as such by create_pr.py, instead of as a missing "Fixes #N"; - a follow_up_issue_url number longer than 10 digits is a format error; over 4300 digits int() raised and the gate CLI died with a traceback. 10 mutants of the new checks all fail the tests; an unmutated copy passes. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent af9189a commit c66a5bf

11 files changed

Lines changed: 204 additions & 20 deletions

File tree

‎.agents/skills/issue-to-pr/PROJECT-CHANGES.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ re-apply the rows below, then run `make skill-test` and
2121
| `scripts/issue_gate.py` (`gh_issue_thread`, `check_disputed_claims`, `issue_resolution`, shared `closes_issue`/`references_issue`), `scripts/create_pr.py` (`issue_reference_problem`), `tests/test_issue_gate.py` (`DisputedIssueClaimTests`), `tests/test_create_and_ci.py` (partial-resolution cases) | the gate verifies each disputed claim's reply against the Issue's comments (fetched only when there are disputes); a `partial` PR must reference the Issue without closing it |
2222
| `scripts/issue_gate.py` (`leaves_issue_open`, `follow_up_number`, `check_follow_ups`, `gh_follow_up_issue`, `issue_resolution` in the gate's info), `scripts/mark_ready.py` (PR `body` re-checked with `issue_reference_problem` before `gh pr ready`), `tests/test_issue_gate.py` (`DisputedIssueClaimTests`), `tests/test_mark_ready.py`, `references/{issue-claims,evidence-contract,quality-gate}.md` | the converse of "partial needs a dispute": `fixes` (also the default) is refused while a `needs_evidence` claim, or an `out_of_scope` claim without a verified follow-up, stays open. The optional `out_of_scope` field `follow_up_issue_url` must be an Issue of the same repository other than this one, linked in the entry's reply, and found by a read-only `gh issue view` (a PR number or any lookup failure blocks; open or closed). Ready re-checks the PR body against the evidence's resolution, because either can change after create |
2323
| `scripts/issue_gate.py` (`command(empty_if=)`), `scripts/pr_ci.py` (`NO_REQUIRED_CHECKS`), `tests/test_create_and_ci.py` (`RequiredChecksQueryTest`), `README.md` | a branch without required checks reads as zero required checks, but only for gh's exact message (exit 1, empty stdout, stderr fullmatching the `no required checks reported on the '<branch>' branch` line captured from gh 2.103.3, case-sensitive); valid JSON on stdout always wins, so a failing required check whose name contains that text stays a failure. Same rule and a parity test in `pr-ready` (`../pr-ready/tests/test_required_checks_parity.py`) |
24+
| `scripts/mark_ready.py` (`unresolved_threads`, `reviewDecision`), `tests/test_mark_ready.py` (`UnresolvedThreadsTest`, review cases), `tests/test_issue_gate.py` (`NO_REVIEW_THREADS`), `SKILL.md`, `README.md`, `references/{quality-gate,workflow}.md` | Ready is refused while a review thread is unresolved (every page, GraphQL answers captured from gh 2.103.3) or a review requests changes; that feedback is `pr-ready`'s to triage, and `pr-review-merge` would return the PR to Draft for an unresolved thread |
25+
| `scripts/issue_gate.py` (`resolution_problem`, follow-up number at most 10 digits), `scripts/create_pr.py` (`issue_reference_problem` names an unknown `issue_resolution` first), `tests/test_issue_gate.py`, `tests/test_create_and_ci.py` | a mistyped `issue_resolution` such as `Partial` is reported as such, not as a missing `Fixes #N`; an oversized follow-up Issue number is a format error, not an `int()` traceback |
2426
| `platforms/{claude,codex,agy}/agents/issue-{sa,coder,reviewer}.*` | Issue claims are hypotheses (SA, Coder); the reviewer returns `disputed_claim_verdicts` and checks every stated acceptance criterion is met or disputed |
2527

2628
Repository-side pieces the skill relies on: `.github/pr-evidence.json` (evidence manifest the

‎.agents/skills/issue-to-pr/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ For multiple repos, supply full URLs or qualified `OWNER/REPO#N` IDs. **An empty
4141
5. Fresh independent `issue-reviewer` reviews the committed HEAD. P0–P2 findings require changes and repeat verification.
4242
6. `issue_gate.py` enforces the local pre-PR gate and duplicate-PR lookup; with disputed claims it also reads the Issue's comments to confirm each one was answered. `create_pr.py` repeats the gate, pushes to the verified remote, verifies remote SHA, and opens a **Draft PR by default**, closing the Issue (`Fixes #N`) only when `issue_resolution` is `fixes`; a `partial` resolution uses `Refs #N`.
4343
7. `pr_ci.py` inspects GitHub checks of the Draft PR for the exact head SHA. Pending CI does not mean success. A branch without required checks (`gh pr checks --required` exits 1 with nothing on stdout and exactly `no required checks reported on the '<branch>' branch` on stderr) counts as zero required checks; valid JSON on stdout is always read, and any other failure is `CI_UNKNOWN`. If CI fails, repair on the same branch, repeat test/review, push and re-check CI; never open a second PR. Finish every other follow-up (final independent review of the whole head diff, accurate PR body, evidence/doc follow-ups) while it is still a Draft. Keep the Issue claim (`claim.py renew`) the whole time.
44-
`mark_ready.py` converts Draft → Ready **only** when the PR is an open Draft of the validated branch at the validated HEAD, the pre-PR gate still passes, CI is `CI_PASSED` on that HEAD, there are no merge conflicts, and `--confirm-complete` attests that nothing else is left. **Ready means merge-quality; a PR with anything outstanding stays Draft.**
44+
`mark_ready.py` converts Draft → Ready **only** when the PR is an open Draft of the validated branch at the validated HEAD, the pre-PR gate still passes, CI is `CI_PASSED` on that HEAD, there are no merge conflicts, no review thread is unresolved and no review requests changes, and `--confirm-complete` attests that nothing else is left. **Ready means merge-quality; a PR with anything outstanding stays Draft.**
4545
8. An Issue whose claims are all refuted is reported `INVALID`, and one that needs the reporter's evidence or a person's decision `NEEDS_DECISION`; both get a reply on the Issue and no PR, and the Issue is left open for a person. If blocked before PR creation, no PR is created. If CI or review is unresolved, the PR stays Draft and is reported `DRAFT_CI_BLOCKED` / `DRAFT_PENDING` with evidence. No auto-merge, no branch-protection bypass.
4646

4747
## Quality helper CLI

‎.agents/skills/issue-to-pr/SKILL.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -106,7 +106,7 @@ python3 /path/to/issue-to-pr/scripts/issue_gate.py \
106106
- Create a **Draft PR** with `gh pr create --draft --repo OWNER/REPO --base VERIFIED_BASE --head VERIFIED_HEAD --title ... --body-file ...`. Prefer `scripts/create_pr.py` (Draft by default) to enforce both local and remote gates, push, prevent duplicate PRs, and verify the resulting PR is a Draft. Only pass `--ready` (a Ready PR from the start) if explicitly requested by the user. Body must include issue-closing keyword `Fixes #ISSUE` **only if** it actually resolves that Issue when merged; when part of the Issue stays open, set `issue_resolution: "partial"` in the evidence and write `Refs #ISSUE` instead (`create_pr.py` enforces both); include summary, root cause, acceptance criteria, tests and HEAD SHA, dependency/stack notes, risks. The packaged `create_pr.py` requires a same-repository remote; fork workflows require a separately verified manual procedure.
107107
- Confirm `gh pr view --json url,number,isDraft,headRefOid,baseRefName` matches expected branch/commit and `isDraft=true`. Do not create an additional PR on retries.
108108
- **Finish the PR while it is a Draft, still holding the Issue claim** (`claim.py renew` at every checkpoint; release only in the final report below). Run `scripts/pr_ci.py` on the exact PR HEAD until it reports `CI_PASSED`. CI failures must be recorded, not misreported as successful; repair on the same branch with renewed tests and independent review, push, and re-check. Complete every remaining follow-up: a final independent review of the whole head diff (a new commit invalidates the old review), an accurate PR body, and any evidence/doc follow-ups. If something cannot be resolved, leave the PR Draft and report `DRAFT_CI_BLOCKED` / `DRAFT_PENDING` with the exact blocker.
109-
- **Mark Ready last, only via `scripts/mark_ready.py`** (it re-runs the gate, requires `CI_PASSED` on the exact HEAD, no merge conflicts, and `--confirm-complete`). Do not run `gh pr ready` by hand, and never mark Ready to "get CI going" or with work still pending.
109+
- **Mark Ready last, only via `scripts/mark_ready.py`** (it re-runs the gate, requires `CI_PASSED` on the exact HEAD, no merge conflicts, no unresolved review thread, no review requesting changes, and `--confirm-complete`). Do not run `gh pr ready` by hand, and never mark Ready to "get CI going" or with work still pending.
110110
- Never merge, enable auto-merge, or bypass branch protection. A Ready PR is not human-approved; say so.
111111

112112
## 5. State and result contract
@@ -130,12 +130,12 @@ For **every target Issue**, report: Issue number/URL, `READY_FOR_REVIEW` (conver
130130
- `scripts/issue_gate.py` — fail-closed deterministic pre-PR gate
131131
- `scripts/create_pr.py` — guarded PR creation; Draft by default (`--ready` only on explicit user request)
132132
- `scripts/pr_ci.py` — SHA-bound GitHub CI status verifier (works on Draft and Ready PRs)
133-
- `scripts/mark_ready.py` — guarded Draft → Ready transition (gate + CI_PASSED on exact HEAD + mergeable + `--confirm-complete`)
133+
- `scripts/mark_ready.py` — guarded Draft → Ready transition (gate + CI_PASSED on exact HEAD + mergeable + no unresolved review thread or `CHANGES_REQUESTED` + `--confirm-complete`)
134134

135135
## Draft-first / Ready semantics (mandatory)
136136

137137
- **Ready for Review = merge-quality, nothing left to do.** Anything outstanding means the PR is a Draft. This overrides any urge to open Ready early "so CI starts" or "so a reviewer can look"; Draft PRs run CI too.
138138
- Default: `gh pr create --draft`; after creation check `isDraft == true`. `scripts/create_pr.py` is the guarded creation route: it validates clean committed HEAD and fresh receipts, pushes the dedicated branch, verifies the remote SHA, checks duplicates, creates one Draft PR and verifies the result. `--ready` is an opt-in only for an explicit user request.
139139
- While Draft: run `scripts/pr_ci.py` on the current PR HEAD (revalidates the SHA on every query). If CI fails, fix on the **same Issue branch**, rerun affected tests and an independent review, then push; never create a second PR. If no checks exist or CI cannot be confirmed, report the unknown state, not success.
140-
- Ready checklist (all required, for the exact current HEAD): pre-PR gate passes; independent review of the final head has no open P0–P2; GitHub CI `CI_PASSED` on that SHA; no merge conflicts; PR body accurate and complete; every evidence/doc follow-up done. Then run `scripts/mark_ready.py ... --confirm-complete`. If the HEAD changes afterwards, the Ready claim is stale: re-verify, and if it can no longer be justified, say so rather than leaving a misleading Ready PR.
140+
- Ready checklist (all required, for the exact current HEAD): pre-PR gate passes; independent review of the final head has no open P0–P2; GitHub CI `CI_PASSED` on that SHA; no merge conflicts; every review comment on the PR answered (review feedback on the Draft is handled with `pr-ready`: fixed, or disputed with evidence and a reply; `mark_ready.py` refuses an unresolved thread or a `CHANGES_REQUESTED` review); PR body accurate and complete; every evidence/doc follow-up done. Then run `scripts/mark_ready.py ... --confirm-complete`. If the HEAD changes afterwards, the Ready claim is stale: re-verify, and if it can no longer be justified, say so rather than leaving a misleading Ready PR.
141141
- If the task cannot pass the pre-PR quality gate, **do not create a PR at all** without explicit user instruction. A PR whose CI/review cannot be brought to green stays Draft and is reported as blocked; never mark it Ready.

‎.agents/skills/issue-to-pr/references/quality-gate.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818

1919
**After PR (PR is a Draft)** — GitHub CI runs on the Draft PR. Report CI state; a CI failure does **not** retroactively make the pre-PR gate pass. Retain the existing Draft PR, repair failures on the same branch and rerun checks; if blocked, leave it Draft and report `DRAFT_CI_BLOCKED` without falsely claiming CI success. Never claim CI passed if it is pending/not observed.
2020

21-
**Before Ready** — all of: pre-PR gate still passes for the PR HEAD (`issue_gate.py --stage pushed --own-pr N`); the final full-diff independent review of the HEAD has no open P0–P2 (rubric: `severity.md`); `pr_ci.py` reports `CI_PASSED` on that exact SHA; no merge conflicts; PR body accurate; evidence/doc follow-ups done. `mark_ready.py --confirm-complete` enforces the mechanical parts.
21+
**Before Ready** — all of: pre-PR gate still passes for the PR HEAD (`issue_gate.py --stage pushed --own-pr N`); the final full-diff independent review of the HEAD has no open P0–P2 (rubric: `severity.md`); `pr_ci.py` reports `CI_PASSED` on that exact SHA; no merge conflicts; no unresolved review thread and no review requesting changes (feedback that arrives on the Draft is `pr-ready`'s job, since `mark_ready.py` does not triage it; `pr-review-merge` returns a PR with an unresolved thread to Draft); PR body accurate; evidence/doc follow-ups done. `mark_ready.py --confirm-complete` enforces the mechanical parts.
2222

2323
`issue_gate.py` is an automated structural/remote-fact gate. It cannot prove that acceptance evidence is truthful, detect every linked PR, perform semantic code review, or guarantee future CI results. Root owns those judgments and must not represent file-based assertions as machine-proven correctness.
2424

‎.agents/skills/issue-to-pr/references/workflow.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,5 +89,5 @@ PR creation should be safe to rerun: it must detect and report an existing PR ra
8989
- If CI fails: Coder changes current Issue branch, commits; Root invalidates old evidence; checks/review on new SHA; push; verify same PR head and rerun CI.
9090
- If CI never starts or cannot be observed: report `DRAFT_PENDING` / CI unknown, not `CI_PASSED`, and keep the PR Draft.
9191
- Before any retry, query the existing PR using branch and Issue refs to prevent duplicates.
92-
- When CI is `CI_PASSED` on the exact HEAD and every follow-up is done (final independent review of the whole head diff, accurate PR body, evidence/doc follow-ups, no conflicts), run `mark_ready.py --confirm-complete`; checkpoint `phase: READY_FOR_REVIEW`. A new commit afterwards invalidates the Ready claim: re-verify immediately.
92+
- When CI is `CI_PASSED` on the exact HEAD and every follow-up is done (final independent review of the whole head diff, every review comment answered, accurate PR body, evidence/doc follow-ups, no conflicts), run `mark_ready.py --confirm-complete`; checkpoint `phase: READY_FOR_REVIEW`. A new commit afterwards invalidates the Ready claim: re-verify immediately.
9393
- Never auto-merge, request auto-merge, or bypass branch protection.

‎.agents/skills/issue-to-pr/scripts/create_pr.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,8 @@
1616
import sys
1717
from pathlib import Path
1818

19-
from issue_gate import GateError, closes_issue, command, evaluate, references_issue
19+
from issue_gate import (ISSUE_RESOLUTIONS, GateError, closes_issue, command, evaluate, references_issue,
20+
resolution_problem)
2021

2122

2223
def execute(args: list[str], cwd: Path) -> str:
@@ -32,6 +33,9 @@ def execute(args: list[str], cwd: Path) -> str:
3233
def issue_reference_problem(body: str, repo: str, issue: int, resolution: str) -> str | None:
3334
"""A 'fixes' PR must close the Issue; a 'partial' one must reference it without closing it,
3435
so merging does not silently close the claims that stay open."""
36+
if resolution not in ISSUE_RESOLUTIONS:
37+
# Named first: a typo such as 'Partial' is otherwise reported as a missing 'Fixes #N'.
38+
return resolution_problem(resolution)
3539
if resolution == "partial":
3640
if closes_issue(body, repo, issue):
3741
return f"issue_resolution is 'partial': the PR body must not close #{issue}; use 'Refs #{issue}'"

‎.agents/skills/issue-to-pr/scripts/issue_gate.py‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -193,7 +193,7 @@ def check_evidence(evidence: dict[str, Any], *, repo: str, issue: int, sha: str,
193193
# `fixes` is the default when the key is absent: the PR body then closes the Issue.
194194
resolution = evidence.get("issue_resolution", "fixes")
195195
if resolution not in ISSUE_RESOLUTIONS:
196-
errors.append("issue_resolution must be 'fixes' or 'partial'")
196+
errors.append(resolution_problem(resolution))
197197
elif resolution == "partial" and not evidence.get("disputed_issue_claims"):
198198
errors.append("issue_resolution 'partial' needs the disputed_issue_claims that stay open")
199199
elif resolution == "fixes":
@@ -211,6 +211,10 @@ def check_evidence(evidence: dict[str, Any], *, repo: str, issue: int, sha: str,
211211

212212

213213
ISSUE_RESOLUTIONS = ("fixes", "partial")
214+
215+
216+
def resolution_problem(resolution: Any) -> str:
217+
return f"issue_resolution must be 'fixes' or 'partial' (exactly, lower case), not {json.dumps(resolution)}"
214218
DISPUTE_DISPOSITIONS = ("invalid", "needs_evidence", "out_of_scope")
215219

216220

@@ -232,7 +236,8 @@ def follow_up_number(item: dict[str, Any], repo: str, issue: int) -> tuple[int |
232236
url = item.get("follow_up_issue_url")
233237
if url is None:
234238
return None, None
235-
found = re.fullmatch(re.escape(f"https://github.com/{repo}/issues/") + r"([1-9][0-9]*)",
239+
# At most 10 digits (GitHub numbers are 32-bit), so int() never sees an oversized literal.
240+
found = re.fullmatch(re.escape(f"https://github.com/{repo}/issues/") + r"([1-9][0-9]{0,9})",
236241
url, re.IGNORECASE) if isinstance(url, str) else None
237242
if found is None:
238243
return None, f"follow_up_issue_url must be https://github.com/{repo}/issues/<number>"

0 commit comments

Comments
 (0)