Skip to content

Commit 8ea8155

Browse files
myasnikovdaniilmyasnikovdaniil
committed
feat(release): add backport-audit, a pre-release backport triage tool
Before a release or an rc is cut there was no way to answer whether everything labelled for backport had actually reached the release branch. The documented approach, `gh pr list --search "is:merged label:backport"`, lists every PR ever labelled and says nothing about whether the change arrived, which is the only thing that matters at that point. Answering it by hand means reading the branch history per PR, so backports go missing quietly -- and one class of them was being lost with no draft PR and no failed check to notice. A `backport` label does not name a branch: backport.yaml resolves the target from getLatestRelease at merge time, so the same label means release-1.5 on a PR merged in June and release-1.6 on one merged in August. The audit reproduces that rule rather than trusting the label text, treating a line as current from the publication of its first non-prerelease release. Landing is then established from three independent kinds of evidence: the merge commit already being reachable from the branch (it merged before the cut, so nothing was needed); a linked backport PR, found by the bot's head branch or by a "Backport of #N" body reference, which is what a hand-written backport carries; or the branch's own history -- the bot's merge subject, a commit subject identical to one of the PR's, or an -x cherry-pick reference. What is left over is reported as MISSING, pending on an open backport PR (including the drafts the bot opens with the conflict committed), or dropped with whatever reason someone recorded on the closed backport PR, since that reason is what stops the next release re-opening the same investigation. The exit code is computed from the results rather than as a side effect of printing them, so it holds under --json, which is the mode a gate would consume. Color follows cmd/check-readiness and auto-disables on a non-TTY and under --json, so piping into jq stays clean. Verified against an independently written prototype rather than against its own reimplementation of these rules: over release-1.6, release-1.5 and release-1.4 the JSON is byte-identical across all 68 verdicts, as is the text report and the exit code. That parity also covers the one shortcut taken -- the history walk the audit already makes collects the reachable commit oids, so a set lookup replaces one `git merge-base --is-ancestor` per candidate per branch, 813 subprocesses for three lines. Reachable from the branch tip is exactly what --is-ancestor tests. It is not faster end to end: wall-clock is dominated by the gh API calls. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 67b4e23 commit 8ea8155

3 files changed

Lines changed: 927 additions & 0 deletions

File tree

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ _repos
1111
# root-level binary waiting to happen — each ~80 MB, and nothing else here
1212
# matched them, leaving `git add .` one keystroke from committing one.
1313
/api-gate
14+
/backport-audit
1415
/backup-controller
1516
/backupstrategy-controller
1617
/check-readiness

‎cmd/backport-audit/README.md‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# backport-audit
2+
3+
Answers one question, before a release or an rc is cut: did everything labelled for backport actually land on the release branch? It prints the URLs of the ones that did not, and exits non-zero, so it can gate the cut.
4+
5+
The manual alternative is `gh pr list --search "is:merged label:backport"`, which lists every PR ever labelled and says nothing about whether the change reached the branch. Answering that by hand means reading the branch history per PR, which is where backports quietly go missing.
6+
7+
```console
8+
$ go run ./cmd/backport-audit release-1.5
9+
10+
=== release-1.5 === 33 candidate PRs: MISSING=4 pending=9 dropped=3 backported=17
11+
12+
MISSING -- labelled for this line, no trace of it here (4):
13+
https://github.com/cozystack/cozystack/pull/2936
14+
#2936 fix(kubernetes): make the default md0 node group removable
15+
label=backport author=myasnikovdaniil merged=2026-06-30 -- no backport PR, nothing on branch
16+
```
17+
18+
The URL is always the **original PR on main** — the thing you decide about. The evidence line names the backport PR when one exists. Anything already on the branch is counted in the header and never listed, so silence means done.
19+
20+
## Usage
21+
22+
```bash
23+
go run ./cmd/backport-audit # the two newest release lines
24+
go run ./cmd/backport-audit release-1.5 # one line
25+
go run ./cmd/backport-audit release-1.6 release-1.5 --json
26+
```
27+
28+
Requires `git` and an authenticated `gh` on PATH. The audit is read-only: it fetches, queries the GitHub API, and writes nothing.
29+
30+
| Flag | Meaning |
31+
|------|---------|
32+
| `--remote NAME` | Git remote holding the release branches (default `origin`) |
33+
| `--limit N` | Max merged PRs scanned per label (default 400) |
34+
| `--no-fetch` | Trust the local refs as-is, skipping `git fetch` |
35+
| `--json` | Machine-readable output |
36+
| `--no-color` | Disable color (auto-disabled on a non-TTY and under `--json`) |
37+
38+
Arguments and flags may be given in any order.
39+
40+
## Exit code
41+
42+
`0` when nothing is outstanding, `1` when something is. That is the point of the tool, so it holds in `--json` mode too:
43+
44+
```bash
45+
go run ./cmd/backport-audit release-1.5 || echo "do not cut yet"
46+
```
47+
48+
`go run` forwards the exit code but also prints its own `exit status 1` line to stderr. Build the binary first (`go build ./cmd/backport-audit`) where that noise is unwelcome.
49+
50+
## Statuses
51+
52+
| Status | Meaning | Action |
53+
|--------|---------|--------|
54+
| `in-branch` | The PR merged before the branch was cut, so its change is already there | none, no backport was ever needed |
55+
| `backported` | A merged backport PR, or the change is present in the branch's history | none |
56+
| `pending` | A backport PR exists and is still open, including the drafts the bot opens with the conflict committed | merge it, or finish the draft |
57+
| `dropped` | A backport PR was closed unmerged, reported with whatever reason someone left on it | none if the reason still holds |
58+
| `MISSING` | No backport PR ever existed and nothing on the branch matches | cherry-pick it by hand, or drop it deliberately |
59+
60+
`dropped` is usually healthy — a maintainer deciding a fix does not apply to that line, e.g. because the feature it repairs never shipped there. The recorded reason is printed so the next release does not re-open the same investigation. A `dropped` entry reading `no reason recorded` is the one that needs a human.
61+
62+
## How a PR is matched to a release line
63+
64+
A `backport` label does not name a branch. [`backport.yaml`](../../.github/workflows/backport.yaml) resolves the target from `getLatestRelease` **at merge time**, so the same label means `release-1.5` on a PR merged in June and `release-1.6` on one merged in August.
65+
66+
The audit reproduces that rule rather than trusting the label text: a PR is a candidate for `release-X.Y` when it merged carrying `backport` while `X.Y` was the current line, or `backport-previous` while `X.Y` was one minor behind. A line is treated as current from the publication of its first non-prerelease release, which is what `getLatestRelease` returns. Without this, auditing an older line reports every newer-era `backport` PR as missing.
67+
68+
## How landing is established
69+
70+
Three independent kinds of evidence, strongest first:
71+
72+
1. **Reachability.** The PR's merge commit is reachable from the release branch, i.e. it merged before the branch was cut (or `main` was later merged in). Nothing was ever needed.
73+
2. **A linked backport PR.** Found by the bot's head branch `backport-<N>-to-release-X.Y`, or by a `Backport of #N` reference in the body, which is what a hand-written backport carries. `MERGED` settles it; `OPEN` is `pending`; `CLOSED` is `dropped`.
74+
3. **The branch's own history.** The bot's `[Backport release-X.Y] <title>` merge subject, a commit subject identical to one of the PR's, or an `-x` cherry-pick reference to one of its commits. This is what catches a hand-backport nobody linked.
75+
76+
## Machine-readable output
77+
78+
```bash
79+
# URLs of what never landed, ready to paste into a tracking issue
80+
go run ./cmd/backport-audit --json release-1.5 | jq -r '.[][] | select(.status=="MISSING") | .url'
81+
82+
# every open backport PR to go merge, across lines
83+
go run ./cmd/backport-audit --json release-1.6 release-1.5 \
84+
| jq -r '.[][] | select(.status=="pending") | .backport_prs[].url'
85+
86+
# dropped items with the reason someone recorded
87+
go run ./cmd/backport-audit --json release-1.4 \
88+
| jq -r '.[][] | select(.status=="dropped") | "#\(.number)\t\(.reason)"'
89+
```
90+
91+
Output is an object keyed by branch, each holding one record per candidate:
92+
93+
```json
94+
{
95+
"number": 3372,
96+
"title": "fix(keycloak-configure): patch HelmRelease in release namespace on teardown",
97+
"url": "https://github.com/cozystack/cozystack/pull/3372",
98+
"author": "lexfrei",
99+
"mergedAt": "2026-07-28T15:45:47Z",
100+
"label": "backport",
101+
"backport_prs": [{"number": 3478, "state": "OPEN", "url": "..."}],
102+
"status": "pending",
103+
"evidence": "backport PR #3478 open"
104+
}
105+
```
106+
107+
## Limits
108+
109+
A `MISSING` verdict is a prompt to check, not proof of absence. A hand-backport that was squash-merged, reworded, and referenced no original PR is invisible to every evidence layer and reads as `MISSING`; confirm at diff level before redoing the work. A `backport` label added long after merge, once the release line has moved on, resolves to the newer line. And the audit reports *that* something is missing, never *why* — a run of `MISSING` entries clustered in time usually means the bot itself was failing during that window, which is worth checking before cherry-picking them one by one.

0 commit comments

Comments
 (0)