Skip to content

Commit 6fe7d21

Browse files
committed
fix(review): close round-2 fail-opens in the structured-payload path
Second adversarial review of the fixes found six more, three of them letting a not-clean artifact score clean. - pre-push-review.py: the rewritten fingerprint anchor still backtracked exponentially. Removing the empty `\s*` alternative was necessary and not sufficient: `={3,}` is self-ambiguous under the outer `*`, since a run of `=` splits into chunks of size >= 3 exponentially many ways. Measured on this tool's OWN `"=" * 60` report banner followed by any non-matching text: 0.50s at 36, 4.01s at 42, 14.18s at 45. Replaced the nested-quantifier regex with a line scan, which is linear by construction and cannot reacquire the class from a future alternative (0.36ms at 4000 `=`). - review_payload.py: `find_fence_spans` defaults to `swallow_unclosed=False`, which records only an unclosed fence's OPENER line and leaves its interior live -- so the mask failed in both directions. A truncated review quoting a CLEAN template scored clean, and a quoted NOT_CLEAN payload minted an undischargeable finding. check-pr-fully-clean.py already passes `swallow_unclosed=True` for this reason; now so does this. - check-pr-fully-clean.py: a payload's self-declared `reviewer` no longer confers identity. It is body text the reviewer writes about itself, so trusting it inverts #2308's invariant -- and it is load-bearing for quorum: measured, two clean comments from the single `github-actions[bot]` login whose payloads named different reviewers satisfied `--quorum 2`, which shared/workflow/fully-clean.md makes the normal invocation. - Drop the `review-json` payload alias. Nothing in the corpus emitted or documented it, so it was a second verdict-bearing input spelling with no producer. - Drop the dead `(oid and oid.lower() == sha_lower)` term from `is_sha_match`. It is read only where `oid` is falsy, so it can never contribute -- while reading as having made formal-review OID matching case-insensitive, which `is_match = (oid == sha)` is not. - check-verdict-scan-parity.py: the corpus carried no `review-data` fragment, so its zero for this change was a coverage statement -- the same failure its own VOCAB comment records for #2668. Added a bounded payload arm crossing placement (bare, fenced, unclosed fence, code span, indented), which payload is authoritative, and verdict/findings agreement. Yielded FIRST, deliberately: `--limit N` truncates the generator and the full product is ~242k bodies, so an arm yielded last sat at index 241,920 and was unreachable at any limit run by hand. Verified: 516 + 50 + 12 tests pass; parity over 600 bodies (all 32 payload bodies among them) reports 0 widened, 0 narrowed, negative control 157 divergences -> DISCRIMINATES; links, memory-file-size, skills, test-suite coverage, reviewer-placeholder and hook-catalog checks green; semantic-line-breaks clean across all four changed markdown files.
1 parent 9230833 commit 6fe7d21

6 files changed

Lines changed: 175 additions & 26 deletions

‎scripts/check-pr-fully-clean.py‎

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -321,7 +321,6 @@ def _is_bot_author(login: Optional[str]) -> bool:
321321
"_posted by opencode (ai agent)",
322322
"verdict:",
323323
"review-data:",
324-
"review-json:",
325324
)
326325

327326

@@ -366,12 +365,15 @@ def _reviewer_identity(body: str, author: str = "") -> str:
366365
if agent:
367366
return agent
368367

369-
structured = extract_structured_review(body or "")
370-
if structured and structured.get("reviewer"):
371-
rev = str(structured["reviewer"]).strip()
372-
if rev and _is_bot_author(login):
373-
return rev
374-
368+
# The payload's own `reviewer` field deliberately does NOT feed identity,
369+
# even from a bot login. It is body text a reviewer writes about itself,
370+
# so trusting it is the inverse of #2308's invariant restated in
371+
# REVIEW_AGENT_MARKERS above -- and it is load-bearing for quorum, not
372+
# merely stylistic: measured, two clean comments from the single login
373+
# `github-actions[bot]` whose payloads named different reviewers satisfied
374+
# `--quorum 2`, which `shared/workflow/fully-clean.md` makes the normal
375+
# invocation. Identity comes from the marker on the first or last line, or
376+
# from the login.
375377
if login:
376378
return login
377379
return "unknown"
@@ -2214,7 +2216,12 @@ def check_review_comments(pr, quorum: int = 1) -> Tuple[bool, List[str]]:
22142216
is_struct_sha_match = bool(
22152217
struct_sha and len(struct_sha) >= 7 and (struct_sha == sha_lower or sha_lower.startswith(struct_sha))
22162218
)
2217-
is_sha_match = bool((oid and oid.lower() == sha_lower) or is_struct_sha_match or sha_short.lower() in body_lower or sha_lower in body_lower)
2219+
# No `oid` term here: `is_sha_match` is read only in the `else:` arm
2220+
# below, where `oid` is falsy by construction, so an `oid` comparison
2221+
# can never contribute -- and writing one reads as having made formal-
2222+
# review OID matching case-insensitive, which `is_match = (oid == sha)`
2223+
# in the `if oid:` arm is not.
2224+
is_sha_match = bool(is_struct_sha_match or sha_short.lower() in body_lower or sha_lower in body_lower)
22182225
if oid:
22192226
# Formal reviews with an explicit commit OID must match the target HEAD SHA exactly
22202227
is_match = (oid == sha)

‎scripts/check-verdict-scan-parity.py‎

Lines changed: 55 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -108,9 +108,63 @@ def load_rev(rev: str, name: str = "base_checker"):
108108
FILLER_EXTRA = ["-", "#", ">", "_", "+", "a.py:10", "**Location:**", "[Defect]"]
109109

110110

111+
# Structured `review-data` payload bodies (ai-config#2736). A SEPARATE,
112+
# bounded arm rather than new entries in the lists above, because those are
113+
# crossed with every other list and one more entry roughly doubles the sweep
114+
# (the runtime cost VOCAB's own comment already records, tracked as #2702).
115+
#
116+
# The corpus carried no payload fragment when #2736 first ran this tool, so its
117+
# "0 widened" was a coverage statement -- the same failure VOCAB's comment
118+
# records for #2668. These bodies cross the three axes a payload can be wrong
119+
# on: WHERE it sits (bare, fenced, unclosed-fence, code span, indented block),
120+
# WHICH of several payloads is authoritative (a quoted template before the real
121+
# one), and whether its verdict and its findings AGREE.
122+
_P_CLEAN = '<!-- review-data: {"verdict": "CLEAN", "findings": []} -->'
123+
_P_BLOCK = ('<!-- review-data: {"verdict": "NOT_CLEAN", "findings": '
124+
'[{"file": "a.py", "message": "boom"}]} -->')
125+
_P_CONTRADICT = ('<!-- review-data: {"verdict": "CLEAN", "findings": '
126+
'[{"file": "a.py", "message": "boom"}]} -->')
127+
128+
PAYLOAD_PLACEMENTS = {
129+
"bare": "{p}",
130+
"fenced": "```json\n{p}\n```",
131+
"unclosed-fence": "```json\n{p}",
132+
"code-span": "the schema is `{p}` appended last",
133+
"indented": " {p}",
134+
}
135+
136+
137+
def payload_bodies():
138+
"""Bodies exercising the structured-payload path, in every placement."""
139+
for payload in (_P_CLEAN, _P_BLOCK, _P_CONTRADICT):
140+
for placement in PAYLOAD_PLACEMENTS.values():
141+
rendered = placement.format(p=payload)
142+
for verdict_line in ("## Verdict: Ready for merge",
143+
"## Verdict: Needs more work"):
144+
yield (f"**Claude finished** review\n\n{verdict_line}\n\n"
145+
f"Reviewed-Commit: abc1234\n\n{rendered}\n")
146+
# A quoted template BEFORE the reviewer's own payload: the authoritative
147+
# one is last, so first-match-wins inverts the verdict.
148+
for real in (_P_BLOCK, _P_CLEAN):
149+
yield ("**Claude finished** review\n\n"
150+
f"The template reads {_P_CLEAN} and you append your own.\n\n"
151+
"## Verdict: Needs more work\n\nReviewed-Commit: abc1234\n\n"
152+
f"{real}\n")
153+
154+
111155
def generated_bodies(exhaustive=False):
112156
seen = set()
113-
157+
158+
# FIRST, deliberately. `--limit N` truncates this generator and the full
159+
# product is ~242k bodies, so an arm yielded last is unreachable at any
160+
# limit anyone runs by hand -- reporting a zero that is a coverage
161+
# statement, which is the exact trap this arm exists to close. Measured:
162+
# yielded last, the first payload body sat at index 241,920.
163+
for body in payload_bodies():
164+
if body not in seen:
165+
seen.add(body)
166+
yield body
167+
114168
if exhaustive:
115169
leads, vocabs, negs = LEAD, VOCAB, NEGATION
116170
else:

‎scripts/lib/review_payload.py‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,11 @@
4242
# Verdict strings that clear, under the same normalization.
4343
CLEAN_VERDICTS = frozenset({"CLEAN", "READY_FOR_MERGE", "APPROVED", "APPROVE"})
4444

45+
# One accepted spelling only. `review-json` was accepted here and in
46+
# REVIEW_BODY_MARKERS while nothing in the corpus emitted or documented it --
47+
# a second verdict-bearing input spelling with no producer, bought for nothing.
4548
_PAYLOAD_RE = re.compile(
46-
r"<!--\s*review-(?:data|json)\s*:\s*(\{[\s\S]*?\})\s*-->",
49+
r"<!--\s*review-data\s*:\s*(\{[\s\S]*?\})\s*-->",
4750
re.IGNORECASE,
4851
)
4952

@@ -66,7 +69,13 @@ def code_region_mask(body: str) -> bytearray:
6669
"""
6770
mask = bytearray(len(body))
6871
try:
69-
fenced, _, orphans = find_fence_spans(body)
72+
# swallow_unclosed=True, matching check-pr-fully-clean.py's own call:
73+
# the default records only an unclosed fence's OPENER line and leaves
74+
# its interior live, which breaks this function in both directions --
75+
# a truncated review's quoted CLEAN template counts as a real verdict,
76+
# and a quoted NOT_CLEAN payload mints a finding no ARD round can
77+
# discharge (the ai-config#2482 class).
78+
fenced, _, orphans = find_fence_spans(body, swallow_unclosed=True)
7079
fenced_lines = set(fenced) | set(orphans)
7180
except Exception:
7281
fenced_lines = set()

‎scripts/pre-push-review.py‎

Lines changed: 29 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -204,6 +204,17 @@ def _load_hook_module():
204204
return _HOOK_MODULE
205205

206206

207+
# Lines tolerated AFTER the final `Reviewed-Commit:` fingerprint. Each must
208+
# start with a known marker: a status banner, a rule line, a disclosure footer,
209+
# a stopping-point declaration, or a restated verdict. Anything else -- a
210+
# chatty sign-off, a smuggled "actually final verdict" line -- means the
211+
# fingerprint is not last, which is what the check exists to establish.
212+
_TRAILING_AFTER_FINGERPRINT = re.compile(
213+
r"(?i)^\s*(?:_?Posted by\b.*|={3,}\s*|Status:.*|\*\*Stopping Point\*\*:.*"
214+
r"|(?:###\s*)?(?:Summary\s+)?Verdict:.*)$"
215+
)
216+
217+
207218
def _structured_contradiction(report: str) -> Optional[str]:
208219
"""Reason string when the report's structured payload blocks, else ``None``.
209220
@@ -393,19 +404,24 @@ def extract_section(text, header_pattern):
393404
return False, False, f"Fingerprint SHA mismatch: found {found_sha_raw!r}, expected {expected_commit_sha!r}."
394405

395406
# Also ensure the final fingerprint is anchored at the end of the report (allowing optional trailing status/disclosure footer)
396-
# Every alternative must consume at least one NON-whitespace character.
397-
# The first cut carried a trailing `|\s*` alternative inside the `*`
398-
# quantifier: it matched empty, duplicating the group's own leading
399-
# `\s*`, so a whitespace run before a non-matching trailing character
400-
# could be partitioned exponentially many ways. Measured on darwin:
401-
# 1.26s at 12 whitespace characters, 21.4s at 14, roughly 3x per added
402-
# character -- and `parse_review_verdict` runs in-process with no
403-
# timeout, so the guard hung rather than failing. It was redundant as
404-
# well: `\s*\Z` below already reaches the same strings with zero
405-
# iterations of the group.
406-
if not re.search(
407-
r"(?im)^\*{0,2}Reviewed[- ]Commit\*{0,2}[ \t]*:[ \t]*[a-f0-9A-F]+(?:\s*(?:_?Posted by[^\n]+|={3,}|Status:[^\n]+|\*\*Stopping Point\*\*:[^\n]+|(?:###\s*)?(?:Summary\s+)?Verdict:[^\n]+))*\s*\Z",
408-
unfenced_report,
407+
# A LINE SCAN, not a nested-quantifier regex. Two successive regex
408+
# cuts each backtracked exponentially and each looked fixed: first a
409+
# `\s*` alternative that matched empty, then -- after removing it --
410+
# the `={3,}` alternative, which is self-ambiguous under the outer `*`
411+
# because a run of `=` splits into chunks of size >= 3 in exponentially
412+
# many ways. Measured on the tool's OWN `"=" * 60` banner
413+
# (`log_error`/report separator) followed by any non-matching text:
414+
# 0.50s at 36 `=`, 4.01s at 42, 14.18s at 45. `parse_review_verdict`
415+
# runs in-process with no timeout, so the guard hung rather than
416+
# failing. Matching each trailing line independently is linear by
417+
# construction, and no further alternative can reintroduce the class.
418+
last_fp = None
419+
for fp_match in re.finditer(r"(?i)Reviewed-Commit:\s*[a-f0-9A-F]+", unfenced_report):
420+
last_fp = fp_match
421+
tail_lines = unfenced_report[last_fp.end():].split("\n")
422+
if not all(
423+
not line.strip() or _TRAILING_AFTER_FINGERPRINT.match(line)
424+
for line in tail_lines
409425
):
410426
return False, False, "Reviewed-Commit fingerprint must be at the very end of the report."
411427

‎scripts/test_check_pr_fully_clean.py‎

Lines changed: 45 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4091,7 +4091,12 @@ class _RealStderr:
40914091
data = checker.extract_structured_review(struct_clean)
40924092
check("extract_structured_review: extracts valid comment payload", data is not None and data.get("verdict") == "CLEAN")
40934093
check("classify_verdict: structured clean review returns clean", checker.classify_verdict(struct_clean) == "clean")
4094-
check("_reviewer_identity: reads reviewer from structured payload for bot login", checker._reviewer_identity(struct_clean, author="github-actions[bot]") == "Claude")
4094+
# The payload's self-declared `reviewer` must NOT confer identity: it is
4095+
# body text the reviewer writes about itself, and two comments from the one
4096+
# `github-actions[bot]` login naming different reviewers otherwise satisfied
4097+
# `--quorum 2`, which shared/workflow/fully-clean.md makes the normal
4098+
# invocation. #2308: approval authority comes from author identity.
4099+
check("_reviewer_identity: a payload's reviewer field does not override the login", checker._reviewer_identity(struct_clean, author="github-actions[bot]") == "github-actions[bot]")
40954100
check("_reviewer_identity: does not escalate unauthenticated member comment to bot identity", checker._reviewer_identity(struct_clean, author="human_member") == "human_member")
40964101
check("_reviewer_identity: agent marker takes precedence over structured reviewer field", checker._reviewer_identity("**claude finished review**\n" + struct_clean.replace('"Claude"', '"adversarial-reviewer"'), author="github-actions[bot]") == "Claude")
40974102
check("_unresolved_finding_pattern: clean structured review has no findings", checker._unresolved_finding_pattern(struct_clean) is None)
@@ -4138,7 +4143,7 @@ class _RealStderr:
41384143
-->
41394144
"""
41404145
check("classify_verdict: structured not-clean review returns not-clean", checker.classify_verdict(struct_not_clean) == "not-clean")
4141-
check("_reviewer_identity: reads reviewer from structured not-clean payload", checker._reviewer_identity(struct_not_clean, author="github-actions[bot]") == "Codex")
4146+
check("_reviewer_identity: a not-clean payload's reviewer field does not override the login", checker._reviewer_identity(struct_not_clean, author="github-actions[bot]") == "github-actions[bot]")
41424147
check("_unresolved_finding_pattern: not-clean structured review returns finding pattern", checker._unresolved_finding_pattern(struct_not_clean) is not None)
41434148

41444149
# Findings override clean verdict in structured JSON
@@ -4268,6 +4273,44 @@ class _RealStderr:
42684273
checker._unresolved_finding_pattern(summary_key_payload)
42694274
== "structured finding in b.py: off-by-one")
42704275

4276+
# --- Regression tests for review round 2 of #2736 ---
4277+
4278+
payload_clean = '<!-- review-data: {"verdict": "CLEAN", "findings": []} -->'
4279+
payload_block = ('<!-- review-data: {"verdict": "NOT_CLEAN", "findings": '
4280+
'[{"file": "a.py", "message": "old"}]} -->')
4281+
4282+
# F2: find_fence_spans defaults to swallow_unclosed=False, which records
4283+
# only an UNCLOSED fence's opener line and leaves its interior live. That
4284+
# broke the mask in both directions.
4285+
unclosed_clean = ("**Claude finished** review\n\nReviewers must append:\n\n```json\n"
4286+
+ payload_clean)
4287+
check("extract_structured_review: an unclosed fence's interior is masked too",
4288+
checker.extract_structured_review(unclosed_clean) is None)
4289+
check("classify_verdict: a truncated review quoting a CLEAN template is not clean",
4290+
checker.classify_verdict(unclosed_clean) != "clean")
4291+
unclosed_block = ("### Verdict: Ready for merge\n\n"
4292+
"Reviewed-Commit: 3a7b9c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b\n\n"
4293+
"Prior round, for reference:\n\n```json\n" + payload_block)
4294+
check("classify_verdict: a payload quoted in an unclosed fence mints no veto",
4295+
checker.classify_verdict(unclosed_block) == "clean")
4296+
check("_unresolved_finding_pattern: a payload quoted in an unclosed fence mints no finding",
4297+
checker._unresolved_finding_pattern(unclosed_block) is None)
4298+
4299+
# F6: `review-json` was a second accepted spelling with no producer.
4300+
check("extract_structured_review: the review-json spelling is not accepted",
4301+
checker.extract_structured_review(
4302+
'<!-- review-json: {"verdict": "CLEAN", "findings": []} -->') is None)
4303+
check("REVIEW_BODY_MARKERS: carries no review-json entry",
4304+
"review-json:" not in checker.REVIEW_BODY_MARKERS)
4305+
4306+
# A real trailing payload must still be read -- the guards above must not
4307+
# have cost the feature they guard.
4308+
real_block = ("### Verdict: Needs more work\n\n"
4309+
"Reviewed-Commit: 3a7b9c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b\n\n"
4310+
+ payload_block)
4311+
check("classify_verdict: an unfenced trailing payload is still authoritative",
4312+
checker.classify_verdict(real_block) == "not-clean")
4313+
42714314
# Code fence citation guard (quoted structured review in markdown fence is ignored)
42724315
quoted_struct = """
42734316
Here is an example of structured review format:

‎scripts/test_pre_push_review.py‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,26 @@ def test_fingerprint_anchor_accepts_each_known_trailing_marker(self):
137137
self.assertFalse(is_valid, f"{label} should be rejected after the fingerprint")
138138
self.assertIn("must be at the very end", reason)
139139

140+
def test_fingerprint_anchor_is_linear_on_the_tools_own_separator(self):
141+
"""Two successive regex cuts each backtracked exponentially and each looked fixed.
142+
143+
First a `\\s*` alternative that matched empty; then, after removing it,
144+
the `={3,}` alternative, self-ambiguous under the outer `*` because a run
145+
of `=` splits into chunks of size >= 3 exponentially many ways. Measured
146+
on this tool's own `"=" * 60` report banner followed by non-matching
147+
text: 0.50s at 36 `=`, 4.01s at 42, 14.18s at 45.
148+
"""
149+
import time
150+
commit = "12345678abcdef00"
151+
report = self._clean_report(
152+
commit, tail="\n\n" + ("=" * 200) + "\nStatus: green\nThanks for reading!")
153+
start = time.perf_counter()
154+
is_valid, _, reason = reviewer.parse_review_verdict(report, expected_commit_sha=commit)
155+
elapsed = time.perf_counter() - start
156+
self.assertLess(elapsed, 2.0, f"anchor check took {elapsed:.1f}s -- backtracking regressed")
157+
self.assertFalse(is_valid)
158+
self.assertIn("must be at the very end", reason)
159+
140160
def test_structured_payload_blocks_a_clean_prose_verdict(self):
141161
"""HTML comments are stripped before every check, so the payload the prompt
142162
asks for was never validated -- the local parser and

0 commit comments

Comments
 (0)