Skip to content

Commit c6ffb1b

Browse files
ci(release): add dispatchable lane to redo finalize registry side effects
The finalize job's registry steps (retag rc images to stable, publish the stable cozy-installer chart) run on the promote-PR merge commit, so a re-run after a mid-flight failure reuses the same tree — a script fix landed on main can never reach it. v1.6.0 hit exactly this: 32 of 43 repos were left untagged and the chart unpublished, recovered by hand with 64 skopeo copies plus helm package/push from the tag tree. Add redo-registry-finalize.yaml, a workflow_dispatch(tag) lane that repeats the registry part for an existing stable tag with a dual checkout: scripts come from the dispatch-time main ref, the release tree (digest pins, chart sources) from refs/tags/<tag>. Guards: strict stable-tag validation, dispatch-from-main check, published-release check, and the same max-published-stable decision as finalize so replaying an old release can repair its immutable tags but never steal :latest. Images stay write-once via promote-retag.sh; the chart step is verify-or-fail — an already-published chart is pulled and compared by extracted file contents, a mismatch fails red before any :latest move unless force_chart=true consciously republishes. Auth is the built-in GITHUB_TOKEN with packages: write only; the lane pushes no git refs. Assisted-By: GPT-5 <[email protected]> Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent c503ae2 commit c6ffb1b

1 file changed

Lines changed: 295 additions & 0 deletions

File tree

Lines changed: 295 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,295 @@
1+
name: Redo Registry Finalize
2+
3+
# Recovery lane for the registry-only tail of a stable release promotion.
4+
#
5+
# A normal finalize run executes from the promote PR's merge commit. Re-running
6+
# that job therefore reuses the same scripts even when its failure was fixed on
7+
# main afterwards. This workflow deliberately checks out two trees: the exact
8+
# main commit selected for the dispatch supplies the current promotion scripts,
9+
# while the immutable release tag supplies the digest pins and installer chart
10+
# sources that were actually released.
11+
#
12+
# Stable tags only: promote-retag.sh promotes rc-built digests to a stable
13+
# destination, and pull-requests-release.yaml intentionally skips these registry
14+
# steps for prereleases. An alpha, beta, or rc tag is therefore rejected.
15+
# force_chart is a break-glass input that permits overwriting an existing stable
16+
# chart only when its extracted contents differ from the release tag tree.
17+
18+
on:
19+
workflow_dispatch:
20+
inputs:
21+
tag:
22+
description: "Existing stable release tag to finalize again, e.g. v1.6.0 (prereleases rejected)"
23+
required: true
24+
type: string
25+
force_chart:
26+
description: "Overwrite the stable chart if its contents differ from the release tag tree"
27+
required: false
28+
default: false
29+
type: boolean
30+
31+
concurrency:
32+
# Serialize every redo, not merely repeats of the same tag. Different stable
33+
# versions can target the same repositories and the same floating :latest
34+
# tags, so letting old-line and current-line recovery runs overlap would make
35+
# the final floating tags timing-dependent. Never cancel a run mid-promotion:
36+
# let its write-once/idempotent operations finish before the next one starts.
37+
# GitHub keeps only one pending run, so a third dispatch may drop the middle run.
38+
group: redo-registry-finalize
39+
cancel-in-progress: false
40+
41+
# Top-level token defaults to read-only; the job opts into only the GHCR write
42+
# scope used by skopeo and helm. No app token or persisted git credential exists
43+
# in this recovery lane, and no step writes a git ref.
44+
permissions:
45+
contents: read # Read release metadata and repository trees; no repository writes.
46+
47+
jobs:
48+
redo:
49+
name: Redo registry finalize for ${{ inputs.tag }}
50+
# Match the normal finalize job: this ephemeral runner is the release lane
51+
# that supports skopeo + helm, rather than the self-hosted build pool.
52+
runs-on: [oracle-vm-4cpu-16gb-x86-64]
53+
timeout-minutes: 45
54+
permissions:
55+
contents: read # Read release metadata and check out both repository trees.
56+
packages: write # Push and copy the GHCR image and chart tags being repaired.
57+
58+
env:
59+
# Must match pull-requests-release.yaml finalize and
60+
# hack/promote-retag.sh's default. This is the release registry, not the
61+
# OCIR build registry.
62+
REGISTRY: ghcr.io/cozystack/cozystack
63+
64+
steps:
65+
# Pure-input validation runs before either checkout or registry login. The
66+
# main-only gate is load-bearing: the purpose of this lane is to apply the
67+
# corrected scripts from main to an older immutable release tree.
68+
- name: Parse and validate request
69+
id: request
70+
env:
71+
TAG: ${{ inputs.tag }}
72+
REF_NAME: ${{ github.ref_name }}
73+
REF_TYPE: ${{ github.ref_type }}
74+
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
75+
with:
76+
script: |
77+
const tag = process.env.TAG;
78+
const m = tag.match(/^v(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)$/);
79+
if (!m) {
80+
core.setFailed(
81+
`tag '${tag}' must be a stable tag matching 'vX.Y.Z' (without leading zeroes). ` +
82+
`Prerelease tags are not accepted because normal finalize skips registry promotion for them.`
83+
);
84+
return;
85+
}
86+
if (process.env.REF_TYPE !== 'branch' || process.env.REF_NAME !== 'main') {
87+
core.setFailed(
88+
`Dispatch this recovery workflow from the 'main' branch so it uses the corrected promotion scripts; ` +
89+
`got ${process.env.REF_TYPE} '${process.env.REF_NAME}'.`
90+
);
91+
return;
92+
}
93+
core.setOutput('tag', tag);
94+
95+
# Reproduce finalize's make_latest decision for an arbitrary historical
96+
# stable: compare numeric major/minor/patch components against all
97+
# published stable releases. The current release itself may be the max;
98+
# only a strictly newer release makes this redo leave :latest untouched.
99+
# Pagination strengthens the same rule for repositories with more than
100+
# one API page of releases.
101+
- name: Determine whether tag should move latest
102+
id: latest
103+
env:
104+
TAG: ${{ steps.request.outputs.tag }}
105+
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7
106+
with:
107+
script: |
108+
const tag = process.env.TAG;
109+
110+
function parseSemver(value) {
111+
const match = value.match(/^v(\d+)\.(\d+)\.(\d+)$/);
112+
if (!match) return null;
113+
return {
114+
major: Number.parseInt(match[1], 10),
115+
minor: Number.parseInt(match[2], 10),
116+
patch: Number.parseInt(match[3], 10),
117+
};
118+
}
119+
120+
function compareSemver(a, b) {
121+
if (a.major !== b.major) return a.major - b.major;
122+
if (a.minor !== b.minor) return a.minor - b.minor;
123+
return a.patch - b.patch;
124+
}
125+
126+
const releases = await github.paginate(github.rest.repos.listReleases, {
127+
owner: context.repo.owner,
128+
repo: context.repo.repo,
129+
per_page: 100,
130+
});
131+
const publishedStable = releases
132+
.filter(release => !release.draft && !release.prerelease)
133+
.map(release => ({
134+
tag: release.tag_name,
135+
semver: parseSemver(release.tag_name),
136+
}))
137+
.filter(release => release.semver !== null);
138+
139+
if (!publishedStable.some(release => release.tag === tag)) {
140+
core.setFailed(
141+
`Published stable GitHub release '${tag}' was not found. ` +
142+
`This lane only repairs the registry tail of an already-published release.`
143+
);
144+
return;
145+
}
146+
147+
let maxRelease = null;
148+
for (const release of publishedStable) {
149+
if (!maxRelease || compareSemver(release.semver, maxRelease.semver) > 0) {
150+
maxRelease = release;
151+
}
152+
}
153+
154+
const current = parseSemver(tag);
155+
const isOutdated = compareSemver(current, maxRelease.semver) < 0;
156+
core.setOutput('move_latest', isOutdated ? 'false' : 'true');
157+
if (isOutdated) {
158+
core.info(`${tag} < ${maxRelease.tag} (max stable semver): leaving :latest unmoved.`);
159+
} else {
160+
core.info(`${tag} is the highest stable version: :latest will be reconciled.`);
161+
}
162+
163+
# Checkout A: only scripts and helpers come from the exact main commit at
164+
# which workflow_dispatch started. Pinning github.sha also prevents main
165+
# advancing in the concurrency queue from changing code under a queued run.
166+
- name: Checkout workflow source
167+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
168+
with:
169+
ref: ${{ github.sha }}
170+
path: workflow-source
171+
fetch-depth: 1
172+
persist-credentials: false
173+
174+
# Checkout B: digest pins and chart sources must come from the immutable
175+
# release tree, never from main. Explicit refs/tags/ avoids a branch/tag
176+
# ambiguity if a similarly named ref exists.
177+
- name: Checkout release tree
178+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
179+
with:
180+
ref: refs/tags/${{ steps.request.outputs.tag }}
181+
path: release-tree
182+
fetch-depth: 1
183+
persist-credentials: false
184+
185+
# Mirror normal finalize's tool bootstrap. These tools act only on GHCR
186+
# and on the ephemeral release-tree checkout; there is no git write path.
187+
- name: Set up toolchain (skopeo, yq, helm)
188+
run: |
189+
command -v yq >/dev/null \
190+
|| { sudo curl -sSL -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 && sudo chmod +x /usr/local/bin/yq; }
191+
command -v skopeo >/dev/null \
192+
|| { sudo apt-get update -qq && sudo apt-get install -y -qq skopeo; }
193+
command -v helm >/dev/null \
194+
|| curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
195+
196+
# Authenticate exactly like normal finalize: the built-in per-run token
197+
# receives packages:write only for this job and is accepted by both tools.
198+
- name: Login to registry (GHCR)
199+
env:
200+
GHCR_USER: ${{ github.actor }}
201+
GHCR_TOKEN: ${{ secrets.GITHUB_TOKEN }}
202+
run: |
203+
echo "$GHCR_TOKEN" | skopeo login "${REGISTRY%%/*}" -u "$GHCR_USER" --password-stdin
204+
echo "$GHCR_TOKEN" | helm registry login "${REGISTRY%%/*}" -u "$GHCR_USER" --password-stdin
205+
206+
# promote-retag.sh resolves its helper beside the SCRIPT but deliberately
207+
# scans packages/ below the CURRENT DIRECTORY. Running the main script with
208+
# release-tree as cwd gives it the fixed code and the released digest pins
209+
# without changing the script or copying files between checkouts.
210+
#
211+
# Each stable image tag is write-once: the script skips an existing tag at
212+
# the expected digest, creates a missing tag, and fails rather than moving
213+
# a stable tag that already points elsewhere. :latest is mutable but is
214+
# enabled only by the max-semver decision above.
215+
- name: Retag rc images to stable
216+
working-directory: release-tree
217+
env:
218+
STABLE_TAG: ${{ steps.request.outputs.tag }}
219+
MOVE_LATEST: ${{ steps.latest.outputs.move_latest == 'true' && '1' || '0' }}
220+
run: ../workflow-source/hack/promote-retag.sh "${STABLE_TAG}"
221+
222+
# Package the stable cozy-installer chart from the release tree, matching
223+
# normal finalize's version/appVersion/platformVersion stamping. An
224+
# existing stable chart must match that package file-for-file; mismatches
225+
# fail unless the dispatch explicitly sets the break-glass force_chart
226+
# input. `--raw` is required because a Helm OCI artifact is not a container
227+
# image and an image-specific skopeo inspect can reject it.
228+
- name: Publish stable cozy-installer chart
229+
working-directory: release-tree
230+
env:
231+
TAG: ${{ steps.request.outputs.tag }}
232+
MAKE_LATEST: ${{ steps.latest.outputs.move_latest }}
233+
FORCE_CHART: ${{ inputs.force_chart }}
234+
run: |
235+
set -euo pipefail
236+
STABLE_VERSION="${TAG#v}"
237+
WORKDIR="$(mktemp -d)"
238+
mkdir -p "${WORKDIR}/fresh" "${WORKDIR}/published" \
239+
"${WORKDIR}/fresh-tree" "${WORKDIR}/published-tree"
240+
241+
# Stamp only the ephemeral tagged checkout. The git tag remains
242+
# immutable, while the packaged chart gets the same stable default
243+
# and metadata as one produced by normal finalize.
244+
export STABLE_VERSION
245+
yq -i '.cozystackOperator.platformVersion = "v" + strenv(STABLE_VERSION)' \
246+
packages/core/installer/values.yaml
247+
helm package packages/core/installer \
248+
--version "${STABLE_VERSION}" --app-version "${STABLE_VERSION}" \
249+
--destination "${WORKDIR}/fresh"
250+
PKG="${WORKDIR}/fresh/cozy-installer-${STABLE_VERSION}.tgz"
251+
252+
if skopeo inspect --raw "docker://${REGISTRY}/cozy-installer:${STABLE_VERSION}" >/dev/null 2>&1; then
253+
helm pull "oci://${REGISTRY}/cozy-installer" \
254+
--version "${STABLE_VERSION}" --destination "${WORKDIR}/published"
255+
PUBLISHED_PKG="${WORKDIR}/published/cozy-installer-${STABLE_VERSION}.tgz"
256+
tar -xzf "${PKG}" -C "${WORKDIR}/fresh-tree"
257+
tar -xzf "${PUBLISHED_PKG}" -C "${WORKDIR}/published-tree"
258+
259+
DIFF_REPORT="${WORKDIR}/chart.diff"
260+
if diff -qr "${WORKDIR}/fresh-tree/cozy-installer" \
261+
"${WORKDIR}/published-tree/cozy-installer" >"${DIFF_REPORT}" 2>&1; then
262+
echo "cozy-installer:${STABLE_VERSION} matches the release tag tree; skipping stable chart push."
263+
else
264+
DIFF_STATUS=$?
265+
if [ "${DIFF_STATUS}" -ne 1 ]; then
266+
echo "::error::Unable to compare published cozy-installer:${STABLE_VERSION} with the release tag tree."
267+
cat "${DIFF_REPORT}"
268+
exit "${DIFF_STATUS}"
269+
fi
270+
271+
echo "Published cozy-installer:${STABLE_VERSION} differs from the chart packaged from release tag ${TAG}."
272+
echo "Differing files:"
273+
cat "${DIFF_REPORT}"
274+
if [ "${FORCE_CHART}" = "true" ]; then
275+
echo "::warning::force_chart=true; overwriting cozy-installer:${STABLE_VERSION} with the release tag tree."
276+
helm push "${PKG}" "oci://${REGISTRY}"
277+
else
278+
echo "::error::Published cozy-installer:${STABLE_VERSION} differs from the release tag tree."
279+
echo "::error::Refusing to overwrite published chart bytes. Review the files above and re-dispatch with force_chart=true to republish deliberately."
280+
exit 1
281+
fi
282+
fi
283+
else
284+
echo "cozy-installer:${STABLE_VERSION} is not published; pushing the release tag tree."
285+
helm push "${PKG}" "oci://${REGISTRY}"
286+
fi
287+
288+
# cozy-installer:latest follows the identical max-semver decision as
289+
# component-image :latest. Replaying an old stable can repair its
290+
# immutable chart tag but can never steal the floating tag.
291+
if [ "${MAKE_LATEST}" = "true" ]; then
292+
skopeo copy --multi-arch all \
293+
"docker://${REGISTRY}/cozy-installer:${STABLE_VERSION}" \
294+
"docker://${REGISTRY}/cozy-installer:latest"
295+
fi

0 commit comments

Comments
 (0)