You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2a1e981
Browse filesBrowse the repository at this point in the historyBrowse files
ci(release): make the registry promotion a re-runnable job
Finalize's registry side effects run after the stable tag and the GitHub
release are already irreversible, and v1.6.0 — the first release to
execute them — died partway through, leaving 32 of 43 repositories
untagged and the chart unpublished. Recovery took 64 hand-run skopeo
copies plus a manual helm package/push, because re-running the job
checks out the same merge commit and therefore the same broken script:
a fix landed on main could never reach it.
Split the registry side effects out of the finalize job so that
GitHub's own "Re-run failed jobs" is the recovery path. As its own job
it replays without re-entering tag creation or release publication, and
the make_latest decision is passed in as a job output rather than
recomputed, so the release's `latest` flag and the images' :latest tag
cannot drift apart.
Two changes make that re-run able to carry a fix rather than only a
retry. The chart publish moves out of the workflow into
hack/promote-publish-chart.sh, so both registry operations are now
scripts; and the job checks the scripts out from main at run time while
the release tree stays pinned to the promoted merge commit. Landing a
correction on main and pressing re-run then applies it to the unchanged
release tree.
The new script also makes the chart step idempotent, which the inline
version was not: helm package is not byte-reproducible, so re-pushing
an already-published version would move the stable chart tag onto new
bytes. An existing chart is pulled and compared by extracted content —
identical is a no-op, different is a hard failure unless FORCE_CHART
says otherwise. Its existence probe uses `skopeo inspect --raw`,
because the image-shaped inspect rejects a Helm OCI artifact on media
type.
Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
Copy file name to clipboardExpand all lines: docs/release.md
+12-2Lines changed: 12 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -181,7 +181,7 @@ The numbered process above is implemented by five workflows. Knowing which job d
181
181
1.[`cut-prerelease.yaml`](../.github/workflows/cut-prerelease.yaml) — manual dispatch (from `main` or `release-X.Y`): the sole entry point for creating a pre-release tag. Validates the `-alpha`/`-beta`/`-rc` tag, then pushes it (write-once, at the branch tip) as the CI app so `tags.yaml` fires. Stable tags are never created here.
182
182
2.[`tags.yaml`](../.github/workflows/tags.yaml) — fires on an rc tag push: runs `prepare-release`, then `generate-changelog`, then `update-website-docs`.
183
183
3.[`promote-rc.yaml`](../.github/workflows/promote-rc.yaml) — manual dispatch: stages the `release-X.Y.Z` tree (rc digests, tag string rewritten to stable), drafts the stable release, and opens the `release-X.Y.Z` promote PR. No registry mutation — transactional.
184
-
4.[`pull-requests-release.yaml`](../.github/workflows/pull-requests-release.yaml) — fires when the `release-X.Y.Z` PR merges; finalizes the release: cuts the write-once stable + Go-module tags, publishes, then retags the rc images to stable by digest (`:latest` gated on newest-stable) and publishes the stable chart.
184
+
4.[`pull-requests-release.yaml`](../.github/workflows/pull-requests-release.yaml) — fires when the `release-X.Y.Z` PR merges; finalizes the release: cuts the write-once stable + Go-module tags and publishes, then — as a separate, independently re-runnable job — retags the rc images to stable by digest (`:latest` gated on newest-stable) and publishes the stable chart. See [Recovering a failed registry promotion](#recovering-a-failed-registry-promotion).
185
185
5.[`update-releasenotes.yaml`](../.github/workflows/update-releasenotes.yaml) — fires on every push to `main`; syncs `docs/changelogs/v*.md` content into the corresponding GitHub Release body.
186
186
187
187
### Phase 1 — `prepare-release` (hard gate)
@@ -238,7 +238,17 @@ Fires on merge of a PR that is merged, carries the `release` label, and is autho
238
238
2.**Cut the `api/apps/v1alpha1/vX.Y.Z` Go-module tag** (write-once, stable only) at the same commit, so Go consumers of `api/apps/v1alpha1` get the release. Moved here from `tags.yaml`, whose `prepare-release` body is skipped for a promoted stable (the draft already exists).
239
239
3.**Ensure the maintenance branch `release-X.Y` exists** at the tag commit. Created if missing; updated fast-forward-only — a non-fast-forward update warns and is left for a maintainer rather than being force-updated.
240
240
4.**Publish the draft release.**`make_latest` is computed against published-non-prerelease tags: prereleases stay `false`; tags older than the current max stay `false` (and the current max is force-restored to `latest` if necessary, so an older patch tag cut after a newer minor won't downgrade `latest`).
241
-
5.**Retag the rc images to stable** by digest (`hack/promote-retag.sh`, no rebuild) and **publish the stable `cozy-installer` chart**. `:latest` (on both the images and the chart) moves only when this release's `make_latest` was `true` — the same decision as step 4, so the release's `latest` and the images' `:latest` never disagree. The chart is packaged with `platformVersion` stamped into its default values so the documented `helm --install --version X.Y.Z` path reports the stable version.
241
+
Steps 1–4 are the `Finalize Release` job. The registry side effects are a second job, `Promote Registry Artifacts`, which `needs` it:
242
+
243
+
5.**Retag the rc images to stable** by digest ([`hack/promote-retag.sh`](../hack/promote-retag.sh), no rebuild) and **publish the stable `cozy-installer` chart** ([`hack/promote-publish-chart.sh`](../hack/promote-publish-chart.sh)). `:latest` (on both the images and the chart) moves only when this release's `make_latest` was `true` — computed once in step 4 and passed to this job as an output, so the release's `latest` and the images' `:latest` cannot disagree. The chart is packaged with `platformVersion` stamped into its default values so the documented `helm --install --version X.Y.Z` path reports the stable version.
244
+
245
+
#### Recovering a failed registry promotion
246
+
247
+
Step 5 is the part that fails: it touches 40+ external repositories over a network, and it runs *after* the tag and the GitHub release have become irreversible. v1.6.0 died partway through it, leaving 32 images untagged and the chart unpublished. Recovery is **"Re-run failed jobs" on the same workflow run** — that is why it is a separate job and why its logic lives in `hack/` scripts rather than inline in the workflow.
248
+
249
+
Re-running is safe at any point of failure. `promote-retag.sh` skips a tag already at the expected digest, creates a missing one, and refuses to move a stable tag that points elsewhere. `promote-publish-chart.sh` compares an already-published chart against the tree by extracted content and skips an identical one, because `helm package` is not byte-reproducible and an unconditional re-push would move the stable chart tag onto new bytes. A run in which everything already landed comes out green.
250
+
251
+
Re-running also picks up a *fix*, not just a retry. The job checks out the release tree at the promoted merge commit — frozen, as release content must be — but checks out the promotion scripts from `main` at run time, into `.promote-scripts/`. So the sequence for a script bug is: land the correction on `main`, press "Re-run failed jobs", and the corrected script runs against the unchanged release tree. Note GitHub allows a re-run only within 30 days of the original run; past that, or if the failure is in the workflow YAML rather than in a script (a re-run reuses the original workflow file), run the script by hand from a checkout of the release tag — `cd <tag-tree> && /path/to/main/hack/promote-retag.sh vX.Y.Z`, with `MOVE_LATEST=1` only if this is the newest stable.
0 commit comments