no-mistakes-required.yml gate self-certifies from the frozen PR body — a required check can go green on a replayed payload the author controls #1637

Summary

The PR must be raised via no-mistakes job in .github/workflows/no-mistakes-required.yml has two properties that together let a required check report success against a body that no longer exists:

env:
  PR_BODY: ${{ github.event.pull_request.body }}
run: |
  marker='Updates from [git push no-mistakes](https://github.com/kunchenguid/no-mistakes)'
  if printf '%s' "${PR_BODY:-}" | grep -qF -- "$marker"; then
    echo "Found no-mistakes signature in PR #${PR_NUMBER} body."
    exit 0
  fi

1. It reads the body from github.event.pull_request.body — the frozen event payload — never from the live API.
2. It is a fixed-string substring test over prose the pull request author fully controls.

Observed on #1485.

Measurement 1 — one commit carried both a green and a red instance of the same required check

On a single head, gh api .../commits/<sha>/check-runs returned two runs of the same named check:

Both were re-runs, and a re-run replays its own original event payload. The synchronize payload was captured when the reduction force-pushed the new head but before the body was rewritten, so it still carries the body containing the marker, and it passes. The edited payload carries the rewritten body with no marker, and it fails.

Neither instance describes the current body. The green one is a replay artifact: a required check reporting success against a body snapshot that no longer exists anywhere on the pull request — and GitHub renders it beside the red one with no indication that they describe different snapshots. A bare "N passed, 1 failed" tally is actively misleading here.

Measurement 2 — a forged signature would pass

The check matches one fixed string, so pasting any block containing that marker into the body turns the check green regardless of whether the attestation is true — including a stale block that prints a commit range for a different head, lists test files no longer in the pull request, and names deleted code. The guard cannot see any of that.

Why tightening the match does not help

The check's evidence and the thing being checked have the same author: the "proof" that a pull request went through the pipeline is a sentence the author types into a text box. A longer marker, a regex, or a checksum over the prose are all still author-written prose. The fix has to change what the claim is checked against, not how tightly it is compared.

Suggested fix

Bind the attestation to the artifact rather than to prose. The pipeline already knows the head it validated, so it can publish its own check-run or commit status against that exact SHA, and the required gate consults a record the contributor cannot write. Separately, a body-derived check should read the live body via the API rather than the frozen github.event.pull_request.body, since re-runs — which fork-PR maintainer approval produces — replay stale payloads.

Provenance

Read directly from the GitHub API and the workflow source:

• gh api repos/kunchenguid/firstmate/contents/.github/workflows/no-mistakes-required.yml
• gh api repos/kunchenguid/firstmate/commits/<sha>/check-runs
• gh api repos/kunchenguid/firstmate/actions/runs/<run-id>