Skip to content

Commit 2a1e981

Browse files
ci(release): make the registry promotion a re-runnable job
Finalize's registry side effects run after the stable tag and the GitHub release are already irreversible, and v1.6.0 — the first release to execute them — died partway through, leaving 32 of 43 repositories untagged and the chart unpublished. Recovery took 64 hand-run skopeo copies plus a manual helm package/push, because re-running the job checks out the same merge commit and therefore the same broken script: a fix landed on main could never reach it. Split the registry side effects out of the finalize job so that GitHub's own "Re-run failed jobs" is the recovery path. As its own job it replays without re-entering tag creation or release publication, and the make_latest decision is passed in as a job output rather than recomputed, so the release's `latest` flag and the images' :latest tag cannot drift apart. Two changes make that re-run able to carry a fix rather than only a retry. The chart publish moves out of the workflow into hack/promote-publish-chart.sh, so both registry operations are now scripts; and the job checks the scripts out from main at run time while the release tree stays pinned to the promoted merge commit. Landing a correction on main and pressing re-run then applies it to the unchanged release tree. The new script also makes the chart step idempotent, which the inline version was not: helm package is not byte-reproducible, so re-pushing an already-published version would move the stable chart tag onto new bytes. An existing chart is pulled and compared by extracted content — identical is a no-op, different is a hard failure unless FORCE_CHART says otherwise. Its existence probe uses `skopeo inspect --raw`, because the image-shaped inspect rejects a Helm OCI artifact on media type. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 2823b76 commit 2a1e981

4 files changed

Lines changed: 459 additions & 67 deletions

File tree

‎.github/workflows/pull-requests-release.yaml‎

Lines changed: 106 additions & 65 deletions
Original file line numberDiff line numberDiff line change
@@ -16,34 +16,34 @@ permissions:
1616
contents: read
1717

1818
jobs:
19+
# Git-side finalization: cut the write-once tags, ensure the maintenance
20+
# branch, publish the release. The registry side effects live in the separate
21+
# `registry` job below so that a failure there can be re-run on its own,
22+
# without re-entering any of the ref-mutating steps here.
1923
finalize:
2024
name: Finalize Release
21-
# Runs skopeo + helm against GHCR (the promotion's registry side effects),
22-
# so it needs an ephemeral VM runner with those tools, not the self-hosted
23-
# pool.
2425
runs-on: [oracle-vm-4cpu-16gb-x86-64]
2526
permissions:
2627
contents: write
27-
packages: write
28-
29-
# Finalize now both cuts the write-once tags AND performs the promotion's
30-
# registry side effects (retag rc->stable, move :latest, publish the stable
31-
# chart), so restrict who can trigger it: the PR must be merged, carry the
32-
# `release` label, AND be authored by the release bot (promote-rc.yaml opens
33-
# it as cozystack-ci[bot]). Merge rights already gate this; asserting the
34-
# author additionally closes the "name a branch release-X.Y.Z, get the label
35-
# applied, merge it" hole.
28+
29+
# Consumed by the `registry` job. make_latest is deliberately NOT recomputed
30+
# there: the release's `latest` flag and the images' :latest tag must be the
31+
# same decision, so it is made once, here, and passed on.
32+
outputs:
33+
tag: ${{ steps.get_tag.outputs.tag }}
34+
make_latest: ${{ steps.publish.outputs.make_latest }}
35+
36+
# This job cuts the write-once tags and publishes the release, and gates the
37+
# registry job below, so restrict who can trigger it: the PR must be merged,
38+
# carry the `release` label, AND be authored by the release bot
39+
# (promote-rc.yaml opens it as cozystack-ci[bot]). Merge rights already gate
40+
# this; asserting the author additionally closes the "name a branch
41+
# release-X.Y.Z, get the label applied, merge it" hole.
3642
if: |
3743
github.event.pull_request.merged == true &&
3844
contains(github.event.pull_request.labels.*.name, 'release') &&
3945
github.event.pull_request.user.login == 'cozystack-ci[bot]'
4046
41-
env:
42-
# The registry promotion writes to: component images and the cozy-installer
43-
# chart. GHCR — the release registry tags.yaml builds rc images into — NOT
44-
# the OCIR CI registry. Must match hack/promote-retag.sh's default.
45-
REGISTRY: ghcr.io/cozystack/cozystack
46-
4747
steps:
4848
- name: Generate GitHub App token
4949
id: app-token
@@ -335,16 +335,81 @@ jobs:
335335
console.log(`✅ Restored ${maxRelease.tag} as latest release`);
336336
}
337337
338-
# ── Transactional promotion: the registry side effects run HERE, after the
339-
# stable tag is cut and the release is published — i.e. after the promote
340-
# PR's e2e passed and the PR merged. promote-rc.yaml did none of this at
341-
# dispatch time, so an abandoned promotion left no stable-named bytes.
338+
# ── Transactional promotion: the registry side effects run HERE, after the
339+
# stable tag is cut and the release is published — i.e. after the promote
340+
# PR's e2e passed and the PR merged. promote-rc.yaml did none of this at
341+
# dispatch time, so an abandoned promotion left no stable-named bytes.
342+
#
343+
# Why a separate job rather than more steps on `finalize`: this is the part
344+
# that fails. It touches 40+ external repositories over a network, and
345+
# v1.6.0 — the first release to run it — died partway through, leaving 32
346+
# images untagged and the chart unpublished AFTER the tag and the GitHub
347+
# release had become irreversible. Recovery is therefore a first-class
348+
# requirement, and the recovery mechanism is GitHub's own "Re-run failed
349+
# jobs": as its own job this replays without re-entering tag creation or
350+
# release publication, and every operation it performs is idempotent
351+
# (promote-retag.sh skips a tag already at the expected digest and refuses
352+
# to move one that is not; promote-publish-chart.sh compares an already
353+
# published chart by content).
354+
#
355+
# Two things make that re-run able to carry a FIX, not merely a retry.
356+
# First, all the logic lives in hack/ scripts rather than in this file.
357+
# Second, those scripts are checked out from `main` at run time (see below),
358+
# so a correction landed after the failure is picked up — whereas the run's
359+
# own workflow file and the release tree are, correctly, frozen at the
360+
# commit that failed.
361+
registry:
362+
name: Promote Registry Artifacts
363+
needs: finalize
364+
# Prereleases never reach here — promote-rc.yaml opens a release PR only for
365+
# a stable promotion — but the guard is kept explicit: an rc must not take a
366+
# stable image tag or publish a stable chart.
367+
if: ${{ !contains(needs.finalize.outputs.tag, '-') }}
368+
# Runs skopeo + helm against GHCR, so it needs an ephemeral VM runner with
369+
# those tools, not the self-hosted pool.
370+
runs-on: [oracle-vm-4cpu-16gb-x86-64]
371+
permissions:
372+
contents: read # Check out the release tree and the promotion scripts.
373+
packages: write # Push the image tags and the chart being promoted.
374+
375+
env:
376+
# The registry promotion writes to: component images and the cozy-installer
377+
# chart. GHCR — the release registry tags.yaml builds rc images into — NOT
378+
# the OCIR CI registry. Must match hack/promote-retag.sh's default.
379+
REGISTRY: ghcr.io/cozystack/cozystack
380+
381+
steps:
382+
# The release tree: the merge commit finalize tagged, i.e. the rc's digests
383+
# with the tag string rewritten to stable. This is release content and must
384+
# stay pinned to the commit under promotion — never re-resolved to a branch
385+
# tip. No git ref is written from this job, so the credential is dropped.
386+
- name: Checkout release tree
387+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
388+
with:
389+
ref: ${{ github.sha }}
390+
fetch-depth: 1
391+
persist-credentials: false
392+
393+
# The promotion tooling, from `main` rather than from the release tree.
394+
# These scripts are CI machinery, not shipped release content, and taking
395+
# them from the branch is what makes a re-run of a failed promotion able to
396+
# apply a fix: land the correction on main, press "Re-run failed jobs", and
397+
# the corrected script runs against the same frozen release tree. Taking
398+
# them from the merge commit instead would pin a re-run to the very code
399+
# that failed, which is exactly how v1.6.0 ended up being repaired by hand.
342400
#
343-
# The checked-out merge commit is the promoted stable tree: the rc's
344-
# digests with the tag string rewritten to stable. skopeo retags those
345-
# digests to the stable image tag; helm packages+pushes the stable chart.
401+
# The cost is that a broken promotion script on main breaks the next
402+
# release even when the merge commit carried a good one; hack/*_test.bats
403+
# runs on main under `make unit-tests` to keep that honest.
404+
- name: Checkout promotion scripts from main
405+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
406+
with:
407+
ref: main
408+
path: .promote-scripts
409+
fetch-depth: 1
410+
persist-credentials: false
411+
346412
- name: Set up toolchain (skopeo, yq, helm)
347-
if: ${{ !contains(steps.get_tag.outputs.tag, '-') }}
348413
run: |
349414
command -v yq >/dev/null \
350415
|| { sudo curl -sSL -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 && sudo chmod +x /usr/local/bin/yq; }
@@ -354,7 +419,6 @@ jobs:
354419
|| curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
355420
356421
- name: Login to registry (GHCR)
357-
if: ${{ !contains(steps.get_tag.outputs.tag, '-') }}
358422
env:
359423
GHCR_USER: ${{ github.actor }}
360424
GHCR_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -365,48 +429,25 @@ jobs:
365429
# Retag every rc image/digest to the stable tag. No rebuild — bit-for-bit
366430
# the e2e-tested rc. :latest moves only when this stable is the newest
367431
# published version, gated by the SAME make_latest decision the release got.
432+
#
433+
# The script comes from .promote-scripts (main) while the working directory
434+
# is the release tree, so it reads the released digest pins and applies the
435+
# current promotion logic. promote-retag.sh resolves its own helper
436+
# relative to $0 and scans `packages` relative to the working directory,
437+
# which is exactly this split; keep that property if either moves.
368438
- name: Retag rc images to stable
369-
if: ${{ !contains(steps.get_tag.outputs.tag, '-') }}
370439
env:
371-
STABLE_TAG: ${{ steps.get_tag.outputs.tag }}
372-
MOVE_LATEST: ${{ steps.publish.outputs.make_latest == 'true' && '1' || '0' }}
373-
run: ./hack/promote-retag.sh "${STABLE_TAG}"
440+
STABLE_TAG: ${{ needs.finalize.outputs.tag }}
441+
MOVE_LATEST: ${{ needs.finalize.outputs.make_latest == 'true' && '1' || '0' }}
442+
run: ./.promote-scripts/hack/promote-retag.sh "${STABLE_TAG}"
374443

375444
# Publish the stable cozy-installer OCI chart — the documented install path
376445
# (`helm upgrade --install cozystack oci://…/cozy-installer --version X.Y.Z`).
377-
# The retag above covers component images + the cozystack-packages artifact,
378-
# but the chart is a separate OCI artifact referenced in no values.yaml, so
379-
# the stable install 404s (manifest unknown) unless it is published here.
380-
# No rebuild: the chart is packaged from the merged stable tree, which pins
381-
# the retagged images + artifact by digest; only the chart's version and
382-
# its platformVersion default are stamped.
446+
# Same split as the retag above: main's script, the release tree's chart
447+
# sources. See hack/promote-publish-chart.sh for why an already-published
448+
# chart is compared by content instead of being re-pushed.
383449
- name: Publish stable cozy-installer chart
384-
if: ${{ !contains(steps.get_tag.outputs.tag, '-') }}
385450
env:
386-
TAG: ${{ steps.get_tag.outputs.tag }}
387-
MAKE_LATEST: ${{ steps.publish.outputs.make_latest }}
388-
run: |
389-
set -eu
390-
STABLE_VERSION="${TAG#v}"
391-
# Bake the correct platform version into the chart's DEFAULT values so
392-
# the primary `helm --install --version X.Y.Z` path reports the stable
393-
# version in telemetry — the operator image itself still carries the rc
394-
# version baked at build time, and this env override is what corrects
395-
# it. Done on the working tree only (never committed): committing it
396-
# would leak platformVersion=vX.Y.Z onto main/the release line when the
397-
# promote PR merges, mis-stamping the next rc built from that branch.
398-
export STABLE_VERSION
399-
yq -i '.cozystackOperator.platformVersion = "v" + strenv(STABLE_VERSION)' \
400-
packages/core/installer/values.yaml
401-
PKG="$(helm package packages/core/installer \
402-
--version "${STABLE_VERSION}" --app-version "${STABLE_VERSION}" | awk '{print $NF}')"
403-
helm push "$PKG" "oci://${REGISTRY}"
404-
rm -f "$PKG"
405-
# cozy-installer:latest tracks the newest stable, matching the image
406-
# :latest and the release's `latest` flag — never moved by a patch on
407-
# an older line.
408-
if [ "${MAKE_LATEST}" = "true" ]; then
409-
skopeo copy --multi-arch all \
410-
"docker://${REGISTRY}/cozy-installer:${STABLE_VERSION}" \
411-
"docker://${REGISTRY}/cozy-installer:latest"
412-
fi
451+
STABLE_TAG: ${{ needs.finalize.outputs.tag }}
452+
MOVE_LATEST: ${{ needs.finalize.outputs.make_latest == 'true' && '1' || '0' }}
453+
run: ./.promote-scripts/hack/promote-publish-chart.sh "${STABLE_TAG}"

‎docs/release.md‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -181,7 +181,7 @@ The numbered process above is implemented by five workflows. Knowing which job d
181181
1. [`cut-prerelease.yaml`](../.github/workflows/cut-prerelease.yaml) — manual dispatch (from `main` or `release-X.Y`): the sole entry point for creating a pre-release tag. Validates the `-alpha`/`-beta`/`-rc` tag, then pushes it (write-once, at the branch tip) as the CI app so `tags.yaml` fires. Stable tags are never created here.
182182
2. [`tags.yaml`](../.github/workflows/tags.yaml) — fires on an rc tag push: runs `prepare-release`, then `generate-changelog`, then `update-website-docs`.
183183
3. [`promote-rc.yaml`](../.github/workflows/promote-rc.yaml) — manual dispatch: stages the `release-X.Y.Z` tree (rc digests, tag string rewritten to stable), drafts the stable release, and opens the `release-X.Y.Z` promote PR. No registry mutation — transactional.
184-
4. [`pull-requests-release.yaml`](../.github/workflows/pull-requests-release.yaml) — fires when the `release-X.Y.Z` PR merges; finalizes the release: cuts the write-once stable + Go-module tags, publishes, then retags the rc images to stable by digest (`:latest` gated on newest-stable) and publishes the stable chart.
184+
4. [`pull-requests-release.yaml`](../.github/workflows/pull-requests-release.yaml) — fires when the `release-X.Y.Z` PR merges; finalizes the release: cuts the write-once stable + Go-module tags and publishes, then — as a separate, independently re-runnable job — retags the rc images to stable by digest (`:latest` gated on newest-stable) and publishes the stable chart. See [Recovering a failed registry promotion](#recovering-a-failed-registry-promotion).
185185
5. [`update-releasenotes.yaml`](../.github/workflows/update-releasenotes.yaml) — fires on every push to `main`; syncs `docs/changelogs/v*.md` content into the corresponding GitHub Release body.
186186

187187
### Phase 1 — `prepare-release` (hard gate)
@@ -238,7 +238,17 @@ Fires on merge of a PR that is merged, carries the `release` label, and is autho
238238
2. **Cut the `api/apps/v1alpha1/vX.Y.Z` Go-module tag** (write-once, stable only) at the same commit, so Go consumers of `api/apps/v1alpha1` get the release. Moved here from `tags.yaml`, whose `prepare-release` body is skipped for a promoted stable (the draft already exists).
239239
3. **Ensure the maintenance branch `release-X.Y` exists** at the tag commit. Created if missing; updated fast-forward-only — a non-fast-forward update warns and is left for a maintainer rather than being force-updated.
240240
4. **Publish the draft release.** `make_latest` is computed against published-non-prerelease tags: prereleases stay `false`; tags older than the current max stay `false` (and the current max is force-restored to `latest` if necessary, so an older patch tag cut after a newer minor won't downgrade `latest`).
241-
5. **Retag the rc images to stable** by digest (`hack/promote-retag.sh`, no rebuild) and **publish the stable `cozy-installer` chart**. `:latest` (on both the images and the chart) moves only when this release's `make_latest` was `true` — the same decision as step 4, so the release's `latest` and the images' `:latest` never disagree. The chart is packaged with `platformVersion` stamped into its default values so the documented `helm --install --version X.Y.Z` path reports the stable version.
241+
Steps 1–4 are the `Finalize Release` job. The registry side effects are a second job, `Promote Registry Artifacts`, which `needs` it:
242+
243+
5. **Retag the rc images to stable** by digest ([`hack/promote-retag.sh`](../hack/promote-retag.sh), no rebuild) and **publish the stable `cozy-installer` chart** ([`hack/promote-publish-chart.sh`](../hack/promote-publish-chart.sh)). `:latest` (on both the images and the chart) moves only when this release's `make_latest` was `true` — computed once in step 4 and passed to this job as an output, so the release's `latest` and the images' `:latest` cannot disagree. The chart is packaged with `platformVersion` stamped into its default values so the documented `helm --install --version X.Y.Z` path reports the stable version.
244+
245+
#### Recovering a failed registry promotion
246+
247+
Step 5 is the part that fails: it touches 40+ external repositories over a network, and it runs *after* the tag and the GitHub release have become irreversible. v1.6.0 died partway through it, leaving 32 images untagged and the chart unpublished. Recovery is **"Re-run failed jobs" on the same workflow run** — that is why it is a separate job and why its logic lives in `hack/` scripts rather than inline in the workflow.
248+
249+
Re-running is safe at any point of failure. `promote-retag.sh` skips a tag already at the expected digest, creates a missing one, and refuses to move a stable tag that points elsewhere. `promote-publish-chart.sh` compares an already-published chart against the tree by extracted content and skips an identical one, because `helm package` is not byte-reproducible and an unconditional re-push would move the stable chart tag onto new bytes. A run in which everything already landed comes out green.
250+
251+
Re-running also picks up a *fix*, not just a retry. The job checks out the release tree at the promoted merge commit — frozen, as release content must be — but checks out the promotion scripts from `main` at run time, into `.promote-scripts/`. So the sequence for a script bug is: land the correction on `main`, press "Re-run failed jobs", and the corrected script runs against the unchanged release tree. Note GitHub allows a re-run only within 30 days of the original run; past that, or if the failure is in the workflow YAML rather than in a script (a re-run reuses the original workflow file), run the script by hand from a checkout of the release tag — `cd <tag-tree> && /path/to/main/hack/promote-retag.sh vX.Y.Z`, with `MOVE_LATEST=1` only if this is the newest stable.
242252

243253
### Phase 6 — `update-releasenotes.yaml` (sync GitHub Release body)
244254

0 commit comments

Comments
 (0)