Skip to content

Commit e532844

Browse files
test(release): pin the freeze and backport-target contracts
Neither cut-prerelease.yaml nor backport.yaml had any test coverage, and neither can be exercised by a PR lane: the first runs only on workflow_dispatch and its whole purpose is a one-way write-once side effect, the second only on a merged main-targeted PR. Every invariant here is one that would otherwise first report a slip at a real release. Pinned, following promote-gate-contract.bats so a guard demoted to a comment cannot satisfy a pin: - the freeze is gated to kind == 'rc' && patch == '0'. Widening either half re-opens a frozen line with no error surfacing. - neither the branch create nor the tag push is forced, in any spelling including a `+refs/` refspec. A force-push on the branch would drag in everything merged since the freeze. - an existing release-X.Y is left untouched rather than reused. - the refuse gate is gated on a main dispatch and runs BEFORE the tag push, so a mistaken dispatch cannot burn a write-once tag name; the freeze runs after it, at the commit that was actually tagged. - backport targets come from enumerated branches, with no getLatestRelease and no min-1 arithmetic, and `previous` is the second-newest existing line. - the comparator sorts numerically descending, which is the only reason release-1.10 outranks release-1.9. A slip to the default lexicographic sort aims every backport at a stale line silently. - freezing a line dispatches its first build-release run. backport.yaml's logic is a github-script block whose comments name getLatestRelease and min-1, so the assertions over it strip `//` comments; the YAML `#` filter alone would let a comment satisfy a pin. The Makefile discovers hack/*.bats, so this runs under `make unit-tests` with no wiring. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 5ed6b22 commit e532844

1 file changed

Lines changed: 224 additions & 0 deletions

File tree

‎hack/release-freeze-contract.bats‎

Lines changed: 224 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,224 @@
1+
#!/usr/bin/env bats
2+
3+
# Contract for the rc freeze and the backport target resolution that depends on
4+
# it. Like promote-gate-contract.bats, these tests pin executable/structural
5+
# workflow lines rather than prose: a commented-out gate or a guard demoted to a
6+
# comment must never satisfy the contract.
7+
#
8+
# Neither workflow can be exercised by a PR lane. cut-prerelease.yaml runs only
9+
# on workflow_dispatch from main or a release line, and its whole purpose is a
10+
# one-way, write-once side effect — the first vX.Y.0-rc.N freezes the line, and
11+
# there is no dry-run mode and no undo short of deleting the branch by hand.
12+
# backport.yaml runs only on a merged main-targeted PR. So every invariant below
13+
# is one that first reports a slip at a real release, on the one commit nobody
14+
# wants to be debugging: getting the freeze condition wrong re-opens a frozen
15+
# line silently, and getting the comparator wrong aims every backport at the
16+
# wrong branch. Cheap to pin here, expensive to discover there.
17+
18+
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
19+
CUT="$REPO_ROOT/.github/workflows/cut-prerelease.yaml"
20+
BACKPORT="$REPO_ROOT/.github/workflows/backport.yaml"
21+
22+
# Strip YAML comments. POSIX `grep` only: the unit-test runner has no ripgrep.
23+
# grep exits 1 when nothing is selected (legitimate for an all-comment block) and
24+
# 2 on a real error, so only the latter propagates.
25+
code_lines() {
26+
local rc=0
27+
grep -v '^[[:space:]]*#' || rc=$?
28+
[ "$rc" -le 1 ]
29+
}
30+
31+
# Strip JavaScript comments. backport.yaml's logic lives in a github-script
32+
# `script: |` block, where comments start with `//` and survive code_lines
33+
# untouched — and that block's comments name the very APIs these tests assert
34+
# are gone, so filtering on `#` alone would let a comment satisfy a pin.
35+
script_lines() {
36+
local rc=0
37+
grep -v '^[[:space:]]*//' || rc=$?
38+
[ "$rc" -le 1 ]
39+
}
40+
41+
# Body of one job, up to the next top-level job key.
42+
job_block() {
43+
awk -v job=" $1:" '
44+
$0 == job { inside = 1; next }
45+
/^ [a-z0-9_-]+:$/ { inside = 0 }
46+
inside' "$2"
47+
}
48+
49+
# Body of one `- name:` step, up to the next step at the same indent.
50+
step_block() {
51+
awk -v want=" - name: $1" '
52+
$0 == want { inside = 1; next }
53+
/^ - name: / { inside = 0 }
54+
inside' "$2"
55+
}
56+
57+
# Line number of a step within the comment-stripped file, for ordering asserts.
58+
# Compared only against other stripped line numbers.
59+
step_line() {
60+
code_lines < "$2" | grep -nF " - name: $1" | awk -F: 'NR == 1 { print $1 }'
61+
}
62+
63+
@test "workflows under contract exist" {
64+
[ -f "$CUT" ]
65+
[ -f "$BACKPORT" ]
66+
}
67+
68+
# ── the freeze condition ─────────────────────────────────────────────────────
69+
# The freeze is create-only and one-way. Widening this condition is the single
70+
# change that re-opens a frozen line without any error surfacing: an alpha/beta
71+
# cut, or a patch-line rc, would branch or re-touch release-X.Y off a tree that
72+
# is not the one the line was frozen at.
73+
74+
@test "freeze is gated to the first rc of a minor (kind rc, patch 0)" {
75+
block="$(step_block 'Freeze the line (create release-X.Y)' "$CUT")"
76+
[ -n "$block" ]
77+
78+
# Both halves, as one expression, on a live (non-comment) line. Dropping
79+
# patch == '0' freezes on every patch-line rc; widening kind freezes on an
80+
# alpha/beta cut from a still-open main.
81+
count="$(code_lines < "$CUT" | grep -cF " if: steps.parse.outputs.kind == 'rc' && steps.parse.outputs.patch == '0'" || true)"
82+
[ "${count:-0}" -eq 1 ]
83+
}
84+
85+
@test "freeze creates the branch without force, and no-ops when it exists" {
86+
block="$(step_block 'Freeze the line (create release-X.Y)' "$CUT")"
87+
[ -n "$block" ]
88+
89+
# The create push, verbatim and non-forced. A force-push here would move an
90+
# existing release-X.Y onto the new tip, dragging in everything merged since
91+
# the freeze — the exact outcome the branch exists to prevent.
92+
count="$(printf '%s\n' "$block" | code_lines | grep -cF 'if ! git push origin "HEAD:refs/heads/$BRANCH"; then' || true)"
93+
[ "${count:-0}" -eq 1 ]
94+
95+
# No force in any shape, including a `+refs/` refspec.
96+
count="$(printf '%s\n' "$block" | code_lines | grep -cE 'git push[^|&;]*(--force|--force-with-lease|[[:space:]]-f[[:space:]])' || true)"
97+
[ "${count:-0}" -eq 0 ]
98+
count="$(printf '%s\n' "$block" | code_lines | grep -cF '+refs/heads/' || true)"
99+
[ "${count:-0}" -eq 0 ]
100+
101+
# An existing branch is left alone rather than reused as a create target.
102+
printf '%s\n' "$block" | code_lines | grep -qF 'leaving it untouched'
103+
}
104+
105+
@test "the pre-release tag push is not forced either" {
106+
block="$(step_block 'Cut and push the pre-release tag' "$CUT")"
107+
[ -n "$block" ]
108+
109+
# Non-forced is what makes the tag write-once at the git layer, independently
110+
# of the ls-remote guard above it.
111+
count="$(printf '%s\n' "$block" | code_lines | grep -cF 'git push origin "HEAD:refs/tags/$TAG"' || true)"
112+
[ "${count:-0}" -eq 1 ]
113+
count="$(printf '%s\n' "$block" | code_lines | grep -cE 'git push[^|&;]*(--force|--force-with-lease|[[:space:]]-f[[:space:]])' || true)"
114+
[ "${count:-0}" -eq 0 ]
115+
count="$(printf '%s\n' "$block" | code_lines | grep -cF '+refs/tags/' || true)"
116+
[ "${count:-0}" -eq 0 ]
117+
}
118+
119+
# ── the refuse gate ──────────────────────────────────────────────────────────
120+
121+
@test "refusing a frozen line is gated on a main dispatch" {
122+
block="$(step_block 'Refuse to cut a frozen line from main' "$CUT")"
123+
[ -n "$block" ]
124+
125+
# Only a main dispatch can smuggle main's tip into a frozen line; a dispatch
126+
# from release-X.Y is already the frozen tree. Losing this guard lets rc.2 be
127+
# cut from main again, which is the regression the freeze was built to fix.
128+
count="$(code_lines < "$CUT" | grep -cF " if: github.ref_name == 'main'" || true)"
129+
[ "${count:-0}" -eq 1 ]
130+
131+
# It must fail closed: an ls-remote that neither found the branch (2) nor
132+
# succeeded (0) is a transport failure, not proof the line is unfrozen.
133+
printf '%s\n' "$block" | code_lines | grep -qF 'refusing to proceed'
134+
}
135+
136+
@test "the refuse gate runs before the tag push, and the freeze after it" {
137+
refuse="$(step_line 'Refuse to cut a frozen line from main' "$CUT")"
138+
push="$(step_line 'Cut and push the pre-release tag' "$CUT")"
139+
freeze="$(step_line 'Freeze the line (create release-X.Y)' "$CUT")"
140+
[ -n "$refuse" ] && [ -n "$push" ] && [ -n "$freeze" ]
141+
142+
# Refuse first: pre-release tag names are write-once, so a mistaken dispatch
143+
# must fail before it burns one.
144+
[ "$refuse" -lt "$push" ]
145+
146+
# Freeze last: the branch has to be created at the commit that was actually
147+
# tagged, so it follows the push that proved the tip had not moved.
148+
[ "$push" -lt "$freeze" ]
149+
}
150+
151+
# ── the frozen line's packages artifact ──────────────────────────────────────
152+
153+
@test "freezing a line dispatches its first build-release run" {
154+
BUILD_RELEASE="$REPO_ROOT/.github/workflows/build-release.yaml"
155+
[ -f "$BUILD_RELEASE" ]
156+
157+
# build-release.yaml publishes cozystack-packages:<line> on push to a line
158+
# branch, but it carries a paths-ignore filter, and GitHub does not run a
159+
# path-filtered workflow for a push that changes no files. The freeze push
160+
# carries no new commits, so it cannot be what builds the new line — the
161+
# dispatch below is. Keep the button that makes the dispatch possible.
162+
count="$(code_lines < "$BUILD_RELEASE" | grep -cF ' workflow_dispatch:' || true)"
163+
[ "${count:-0}" -eq 1 ]
164+
165+
block="$(step_block "Build the frozen line's packages artifact" "$CUT")"
166+
[ -n "$block" ]
167+
printf '%s\n' "$block" | code_lines | grep -qF 'gh workflow run build-release.yaml --ref "$BRANCH"'
168+
169+
# Only when this run actually created the branch. Re-dispatching for a line
170+
# that was already frozen would rebuild it for no reason.
171+
count="$(code_lines < "$CUT" | grep -cF " if: steps.freeze.outputs.created == 'true'" || true)"
172+
[ "${count:-0}" -eq 1 ]
173+
count="$(code_lines < "$CUT" | grep -cF ' id: freeze' || true)"
174+
[ "${count:-0}" -eq 1 ]
175+
}
176+
177+
# ── backport target resolution ───────────────────────────────────────────────
178+
# Both assertions below run over script_lines: this logic sits in a
179+
# github-script block whose comments discuss getLatestRelease and min-1 by name.
180+
181+
@test "backport targets come from enumerated branches, not the latest release" {
182+
block="$(job_block prepare "$BACKPORT")"
183+
[ -n "$block" ]
184+
185+
# Enumerate real branches. getLatestRelease returns the newest published
186+
# STABLE, which throughout a freeze window still names the PREVIOUS line — so
187+
# a `backport` on a fix for the release being stabilised would land on the
188+
# wrong branch and miss the release it was written for.
189+
printf '%s\n' "$block" | script_lines | grep -qF 'github.paginate(github.rest.repos.listBranches'
190+
count="$(printf '%s\n' "$block" | script_lines | grep -cF 'getLatestRelease' || true)"
191+
[ "${count:-0}" -eq 0 ]
192+
193+
# Only branches matching release-<major>.<minor> qualify, so per-release
194+
# staging branches (release-1.6.1) and rc branches never become targets.
195+
printf '%s\n' "$block" | script_lines | grep -qF 'release-(\d+)\.(\d+)$'
196+
197+
# previous is the second-newest EXISTING line, never an arithmetic guess. The
198+
# old min-1 derivation named a non-existent branch whenever a minor was
199+
# skipped.
200+
printf '%s\n' "$block" | script_lines | grep -qF 'lines.length > 1 ? lines[1].name'
201+
count="$(printf '%s\n' "$block" | script_lines | grep -cE 'parseInt\(min\)[[:space:]]*-[[:space:]]*1|min[[:space:]]*-[[:space:]]*1' || true)"
202+
[ "${count:-0}" -eq 0 ]
203+
204+
# Asking for a previous line when only one exists must fail, not silently
205+
# backport nowhere.
206+
printf '%s\n' "$block" | script_lines | grep -qF 'core.setFailed'
207+
}
208+
209+
@test "release lines sort numerically descending so 1.10 outranks 1.9" {
210+
block="$(job_block prepare "$BACKPORT")"
211+
[ -n "$block" ]
212+
213+
# Numeric descending on major then minor is the ONLY reason release-1.10
214+
# ranks above release-1.9. A slip to the default lexicographic sort compares
215+
# the names as strings, puts "release-1.9" first, and aims every backport at
216+
# a stale line — with nothing going red.
217+
printf '%s\n' "$block" | script_lines | grep -qF '.sort((a, b) => (b.maj - a.maj) || (b.min - a.min))'
218+
219+
# The comparands must be numbers, not the captured strings.
220+
printf '%s\n' "$block" | script_lines | grep -qF 'maj: Number(m[1]), min: Number(m[2])'
221+
222+
# And the head of that sort is what `backport` uses.
223+
printf '%s\n' "$block" | script_lines | grep -qF 'lines[0].name'
224+
}

0 commit comments

Comments
 (0)