Skip to content

Commit c77616a

Browse files
test(release): pin the audit's label set to the one the backport bot acts on
cmd/backport-audit finds its candidates with `gh pr list --label`, which matches by exact name, so the label set is a contract between it and backport.yaml that nothing else couples. When the labels were namespaced under kind/, backport.yaml was updated in lockstep and the audit was not. Nothing errored: `--label backport` went on matching and returned 7 of the 290 merged PRs carrying a request, so the audit reported two branches clean that between them had 5 missing and 7 pending backports, and exited 0. That is the failure mode this file already exists to prevent for the workflows themselves -- an invariant whose first report otherwise arrives at a real release, on the commit nobody wants to be debugging. A rename is worse than the cases already pinned here, because it breaks nothing loudly at all. Pin the two sets equal rather than either one's contents. The audit must not miss a spelling the bot honours, and must not keep one the bot has retired: the legacy names are transitional, and the two files have to drop them together. The extraction reads the bot's label-detection script and the audit's Go string literals. The audit's own prose quotes these names with backticks rather than double quotes, so a comment cannot satisfy the pin. Verified by mutation: dropping one legacy spelling from the audit alone turns the test red. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 2dd2296 commit c77616a

1 file changed

Lines changed: 43 additions & 2 deletions

File tree

‎hack/release-freeze-contract.bats‎

Lines changed: 43 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
#!/usr/bin/env bats
22

33
# Contract for the rc freeze, the backport target resolution that depends on it,
4-
# and the triggering and concurrency rules that decide whether a backport runs at
5-
# all. Like promote-gate-contract.bats, these tests pin executable/structural
4+
# the triggering and concurrency rules that decide whether a backport runs at
5+
# all, and the agreement between those rules and cmd/backport-audit, which
6+
# gates a cut by reproducing them. Like promote-gate-contract.bats, these tests pin executable/structural
67
# workflow lines rather than prose: a commented-out gate or a guard demoted to a
78
# comment must never satisfy the contract.
89
#
@@ -19,6 +20,7 @@
1920
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
2021
CUT="$REPO_ROOT/.github/workflows/cut-prerelease.yaml"
2122
BACKPORT="$REPO_ROOT/.github/workflows/backport.yaml"
23+
AUDIT="$REPO_ROOT/cmd/backport-audit/main.go"
2224

2325
# Strip YAML comments. POSIX `grep` only: the unit-test runner has no ripgrep.
2426
# grep exits 1 when nothing is selected (legitimate for an all-comment block)
@@ -594,3 +596,42 @@ closed-merged=true"
594596
printf '%s\n' "$b" | grep -qF 'github.event.pull_request.merged == true &&'
595597
done
596598
}
599+
600+
@test "the audit accepts exactly the backport labels the bot acts on" {
601+
# cmd/backport-audit answers "has everything labelled for this line landed on
602+
# it" and exits non-zero while anything has not, so docs/release.md gates a
603+
# patch cut on it. It reaches its candidates with `gh pr list --label`, which
604+
# matches a label by exact name.
605+
#
606+
# That makes the label set a contract between two files that nothing else
607+
# couples. When the labels were namespaced under kind/, backport.yaml was
608+
# updated in lockstep and the audit was not: `--label backport` went on
609+
# matching, so nothing errored, and it returned 7 of the 290 merged PRs that
610+
# carried a request. The audit then reported two branches clean that between
611+
# them had 5 missing and 7 pending backports, and exited 0. A gate that goes
612+
# green because it stopped seeing the work is worse than no gate, and it is
613+
# invisible precisely because a rename does not break anything loudly.
614+
#
615+
# So pin the two sets equal rather than pinning either one's contents: the
616+
# audit must not miss a spelling the bot honours, and must not keep one the
617+
# bot has retired. Both directions matter, because the legacy names are
618+
# transitional and the two files have to drop them together.
619+
[ -f "$AUDIT" ]
620+
621+
# Every name the bot tests for, from the label-detection script.
622+
bot="$(grep -o "labels\.includes('[^']*')" "$BACKPORT" | sed "s/.*('//;s/')//" | sort -u)"
623+
[ -n "$bot" ]
624+
625+
# Every backport label spelled as a Go string literal in the audit. Its own
626+
# prose quotes these names with backticks, not double quotes, so a comment
627+
# cannot satisfy this.
628+
audit="$(grep -o '"\(kind/\)\{0,1\}backport\(-previous\)\{0,1\}"' "$AUDIT" | tr -d '"' | sort -u)"
629+
[ -n "$audit" ]
630+
631+
[ "$bot" = "$audit" ]
632+
633+
# Both requests are represented, so the sets cannot agree by both having lost
634+
# the same half.
635+
printf '%s\n' "$audit" | grep -qx 'kind/backport'
636+
printf '%s\n' "$audit" | grep -qx 'kind/backport-previous'
637+
}

0 commit comments

Comments
 (0)