Skip to content

Commit 858ebaa

Browse files
myasnikovdaniilmyasnikovdaniil
committed
fix(release): retain live promotion candidates
Protect package candidates referenced by open release PRs while pruning abandoned run-specific promotion artifacts. Document the stable-candidate lifecycle and maintenance-branch rollout requirement. Assisted-By: Claude <[email protected]> Assisted-By: GPT-5 <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 7e85869 commit 858ebaa

4 files changed

Lines changed: 162 additions & 36 deletions

File tree

‎.github/workflows/retention.yaml‎

Lines changed: 78 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Nightly Retention
1+
name: Artifact Retention
22

33
# Prune old nightly artifacts from GHCR: keep the newest KEEP dated versions of
44
# each cozystack package, delete older ones. Nightlies are GHCR OCI tags
@@ -13,6 +13,14 @@ name: Nightly Retention
1313
# latest, rc, main) is never touched, so stable/rc/main artifacts can never be
1414
# pruned.
1515
#
16+
# Promotion also creates one temporary cozystack-packages tag of the form
17+
# promotion-vX.Y.Z-from-vX.Y.Z-rc.N-run-ID-ATTEMPT. Every dispatch gets a unique
18+
# tag, so a retry never moves one and leaves an untagged manifest behind. A
19+
# successful promotion adds a stable tag to the same manifest, which makes it
20+
# ineligible below. A candidate stays protected while an open bot-authored
21+
# release PR pins its digest; otherwise it becomes eligible after
22+
# PROMOTION_TTL_DAYS only when ALL its tags are still temporary.
23+
#
1624
# Dry-run by default; flip the input or the RETENTION_APPLY repo variable to
1725
# actually delete. Deletion is irreversible.
1826
#
@@ -37,18 +45,20 @@ permissions:
3745
env:
3846
ORG: cozystack
3947
KEEP: '14'
48+
PROMOTION_TTL_DAYS: '30'
4049

4150
concurrency:
42-
group: nightly-retention
51+
group: artifact-retention
4352
cancel-in-progress: false
4453

4554
jobs:
4655
prune:
47-
name: Prune old GHCR nightlies
56+
name: Prune old GHCR temporary artifacts
4857
runs-on: ubuntu-latest
4958
permissions:
5059
contents: read
5160
packages: write
61+
pull-requests: read
5262
steps:
5363
- name: Generate GitHub App token
5464
id: app-token
@@ -64,9 +74,44 @@ jobs:
6474
# workflow_dispatch input wins; otherwise the repo variable; else dry-run.
6575
APPLY: ${{ github.event.inputs.apply || vars.RETENTION_APPLY || 'false' }}
6676
run: |
67-
set -eu
77+
set -euo pipefail
6878
if [ "$APPLY" = "true" ]; then echo "MODE: APPLY (deleting)"; else echo "MODE: dry-run"; fi
6979
80+
# Protect the immutable packages digest pinned by every open promotion
81+
# PR. Age alone does not imply abandonment: an approved release PR can
82+
# legitimately remain open longer than the TTL. Resolve only trusted,
83+
# same-repository release branches authored by the release bot; fail
84+
# closed before any deletion if their installer pin cannot be read.
85+
release_pr_shas="$(gh api --paginate "/repos/${GITHUB_REPOSITORY}/pulls?state=open&per_page=100" \
86+
| jq -s -r --arg repo "$GITHUB_REPOSITORY" 'add[]
87+
| select(.user.login == "cozystack-ci[bot]")
88+
| select(.head.repo.full_name == $repo)
89+
| select(.head.ref | test("^release-[0-9]+\\.[0-9]+\\.[0-9]+$"))
90+
| select(any(.labels[]?; .name == "release"))
91+
| .head.sha')"
92+
protected_promotion_digests=""
93+
for sha in $release_pr_shas; do
94+
values="$(gh api \
95+
-H 'Accept: application/vnd.github.raw+json' \
96+
"/repos/${GITHUB_REPOSITORY}/contents/packages/core/installer/values.yaml?ref=${sha}")"
97+
candidate_ref="$(printf '%s\n' "$values" | awk '
98+
/^[[:space:]]*platformSourceRef:[[:space:]]*/ {
99+
sub(/^[[:space:]]*platformSourceRef:[[:space:]]*/, "")
100+
gsub(/^[\047"]|[\047"][[:space:]]*$/, "")
101+
print
102+
}')"
103+
ref_lines="$(printf '%s\n' "$candidate_ref" | wc -l | tr -d ' ')"
104+
if ! printf '%s\n' "$candidate_ref" | grep -Eq '^digest=sha256:[0-9a-f]{64}$' \
105+
|| [ "$ref_lines" -ne 1 ]; then
106+
echo "::error::Open release PR at ${sha} has no single immutable platformSourceRef; refusing retention."
107+
exit 1
108+
fi
109+
candidate_digest="${candidate_ref#digest=}"
110+
protected_promotion_digests="${protected_promotion_digests:+${protected_promotion_digests} }${candidate_digest}"
111+
done
112+
protected_count="$(printf '%s\n' "$protected_promotion_digests" | awk '{print NF}')"
113+
echo "Protecting ${protected_count} promotion candidate digest(s) referenced by open release PRs."
114+
70115
# All container packages under the org that live in the cozystack/
71116
# registry namespace (ghcr.io/cozystack/cozystack/<name>).
72117
pkgs="$(gh api --paginate "/orgs/${ORG}/packages?package_type=container" \
@@ -107,4 +152,33 @@ jobs:
107152
fi
108153
done
109154
done
155+
156+
# A promoted manifest acquires vX.Y.Z (and possibly latest) in addition
157+
# to its temporary tag, so the all(tags match promotion-*) predicate is
158+
# the safety boundary: published bytes are never eligible. An open PR's
159+
# pinned digest is excluded separately. Only the cozystack-packages
160+
# repository receives these candidate tags.
161+
pkg='cozystack/cozystack-packages'
162+
enc="$(jq -rn --arg p "$pkg" '$p|@uri')"
163+
cutoff="$(date -u -d "${PROMOTION_TTL_DAYS} days ago" '+%Y-%m-%dT%H:%M:%SZ')"
164+
promotion_ids="$(gh api --paginate "/orgs/${ORG}/packages/container/${enc}/versions" \
165+
| jq -s --arg cutoff "$cutoff" --arg protected "$protected_promotion_digests" '[add[]
166+
| .name as $digest
167+
| select((.metadata.container.tags | length) > 0)
168+
| select(.metadata.container.tags | all(test("^promotion-v[0-9]+\\.[0-9]+\\.[0-9]+-from-v[0-9]+\\.[0-9]+\\.[0-9]+-rc\\.[0-9]+-run-[0-9]+-[0-9]+$")))
169+
| select(.updated_at < $cutoff)
170+
| select(($protected | split(" ") | index($digest)) == null)
171+
| .id]')"
172+
promotion_total="$(printf '%s' "$promotion_ids" | jq 'length')"
173+
echo "── ${pkg}: ${promotion_total} abandoned promotion candidate(s) older than ${PROMOTION_TTL_DAYS} days"
174+
printf '%s' "$promotion_ids" | jq -r '.[]' | while IFS= read -r id; do
175+
[ -n "$id" ] || continue
176+
if [ "$APPLY" = "true" ]; then
177+
echo " ✗ deleting ${pkg} promotion candidate version ${id}"
178+
gh api -X DELETE "/orgs/${ORG}/packages/container/${enc}/versions/${id}" >/dev/null \
179+
|| echo "::warning::failed to delete ${pkg} version ${id} (permissions?)"
180+
else
181+
echo " • would delete ${pkg} promotion candidate version ${id}"
182+
fi
183+
done
110184
echo "Done."

‎docs/agents/image-refs.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,8 @@ If a package stamps a reference somewhere the two globs do not cover, add that f
5353
| Tool | Reads | Job |
5454
| --- | --- | --- |
5555
| `hack/promote-rewrite-tags.sh` | all three shapes | rewrite `X.Y.Z-rc.N` → `X.Y.Z` in the tree, then fail if any survives |
56+
| `hack/promote-packages-artifact.sh` | the rewritten packages tree | publish a reproducible temporary stable-candidate artifact and pin its digest in the installer |
57+
| `hack/verify-promoted-packages.sh` | pinned candidate + merged tree + embedded rc artifact ref | before any stable tag is created, prove the candidate contains the merged tree and the rc's container digests |
5658
| `hack/promote-retag.sh` | all three shapes | copy each digest to its stable tag in the registry |
5759
| `hack/nightly-mirror.sh` | all three shapes | copy each digest to the public registry, then rewrite hosts in the same file set |
5860
| `hack/overlay-main-images.sh` | `values.yaml` + `*.tag` | overlay current-main references onto packages a PR did not rebuild |
@@ -77,10 +79,12 @@ Collection understands both shapes — the host is rejoined, so a split-key ref
7779

7880
**`charts/` is excluded, and is almost always right.** Vendored upstream chart values are regenerated by `make update` and their images are neither built nor pushed here. The one exception is `packages/system/cozystack-scheduler/charts/cozystack-scheduler/values.yaml`, which pins a first-party-namespace image. Excluding it is still correct — it is built by a separate repository, component-versioned, and already published to the public registry, so no retag or mirror is owed — but "everything under `charts/` is third-party" is not literally true.
7981

80-
## Why promotion does not rebuild
82+
## Why promotion does not rebuild containers
8183

82-
Promotion copies the release candidate's images to the stable tag **by digest**, so the stable release is bit-for-bit what passed end-to-end testing. Nothing is recompiled, and promotion cannot diverge from what was tested. The only thing that changes in the tree is the cosmetic tag string, which is what `hack/promote-rewrite-tags.sh` does — and why getting its file coverage right is the difference between a release that reads as a release and one that reads as a release candidate.
84+
Promotion copies the release candidate's container images to the stable tag **by digest**, so every executable image in the stable release is bit-for-bit what passed end-to-end testing. Nothing is recompiled. The packages OCI artifact is declarative configuration rather than a container image, and its content must change: `hack/promote-rewrite-tags.sh` replaces the rc tag strings in the tree, the workflow commits that exact content, then `hack/promote-packages-artifact.sh` serializes it under a run-specific temporary `promotion-*` tag and a second commit pins its new digest in the installer. Promotion first requires the target base to carry the candidate-aware pipeline. The promotion PR checks the candidate against its prospective merge; finalize repeats the check before creating any stable name. Both require the candidate and its embedded rc baseline to use the trusted public `cozystack-packages` repository, follow the old self-reference back to that rc artifact, and prove that all container digests stayed fixed.
85+
86+
The packages artifact includes `core/installer/values.yaml`, so it cannot contain its own digest: writing that digest changes the payload and produces another digest forever. The separately published installer chart carries the real candidate pin used at runtime; the older `platformSourceRef` embedded inside the packages artifact is inert because the platform PackageSource does not install `core/installer`. Verification normalizes exactly that field and rejects every other difference. It also models Flux's established archive semantics: symlinks in the source tree are omitted rather than followed, matching the RC packages artifacts produced by the existing build.
8387

8488
## Testing
8589

86-
`hack/promote-rewrite-tags_test.bats` round-trips the rewrite against the real tree: it stamps a synthetic release-candidate version into every first-party reference, runs the rewrite, and asserts nothing retains it. This is the check that catches a new storage shape, and it runs under `make unit-tests` without a cluster, a registry or a release. Any change to how references are stored must keep it passing.
90+
`hack/promote-rewrite-tags_test.bats` round-trips the rewrite against the real tree: it stamps a synthetic release-candidate version into every first-party reference, runs the rewrite, and asserts nothing retains it. `hack/promote-packages-artifact_test.bats` covers deterministic candidate naming, metadata and digest pinning; `hack/verify-promoted-packages_test.bats` proves the self-reference exception is narrow and catches rc refs or post-publish tree drift. These checks run under `make unit-tests` without a cluster, a registry or a release. Any change to how references are stored or promoted must keep them passing.

0 commit comments

Comments
 (0)