Skip to content

Commit bbfe9f9

Browse files
fix(backupstrategy-controller): give chBackupClientImage a producer (#3405)
## This is a workaround `backupstrategy-controller` should not be running the platform **migrations** image. This PR does not fix that — it makes the existing reuse *safe* so the v1.6.0 promote can proceed, and documents in the tree exactly what is wrong and what should replace it. Treat it as a stopgap with a TODO, not as an endorsement of the design. ## What is actually wrong The Altinity strategy Pod drives clickhouse-backup's HTTP API. Its script uses exactly four binaries — `curl`, `jq`, `sleep`, `date`. For that it currently pulls `platform-migrations`: | | | | --- | --- | | Size | **349 MB** (a dedicated `alpine + curl + jq + ca-certificates` measures **23 MB**) | | Also contains | `kubectl`, `helm`, `git`, `cozyhr`, the `etcd-migrate` binary, every migration script, vendored etcd CRDs | | `ENTRYPOINT` | `run-migrations.sh` | Two consequences worth naming. A tenant-adjacent backup Pod carries cluster tooling it has no use for. And because the image's own entrypoint is the migration runner, the `command:` override in the strategy template is load-bearing — drop that one line and the Pod runs platform migrations with `kubectl` and `helm` already on the PATH. It also creates the coupling that caused the bug this PR fixes: bumping `ETCD_OPERATOR_VERSION` or editing any migration script re-digests the image, forcing the backup client's pin to move for reasons that have nothing to do with backups. ## The release blocker `platform-migrations` was pinned at two different digests in the committed tree: | Location | Pin | Producer | | --- | --- | --- | | `packages/core/platform/values.yaml` `.migrations.image` | `v1.5.0@sha256:8bf61f17…` | `packages/core/platform/Makefile` | | `backupstrategy-controller/values.yaml` `.chBackupClientImage` | `v1.4.0-rc.2@sha256:17390197…` | **none** | Promotion retags **by digest**: `hack/promote-retag.sh` copies every collected `<repo>@<digest>` to `<repo>:<stable-version>`. Two digests under one repository produce two copies competing for the same destination tag — the first wins, the second hits the write-once guard, and the promotion fails. A dry run against `main` emits exactly that pair: ``` ▸ ghcr.io/cozystack/cozystack/platform-migrations sha256:17390197… → docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9 ▸ ghcr.io/cozystack/cozystack/platform-migrations sha256:8bf61f17… → docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9 ``` `v1.6.0` is the first release cut through promotion rather than a full rebuild. Every earlier release ran `make build`, which restamped both copies as a side effect and hid the missing producer. ## What this PR does `packages/core/platform/Makefile` now stamps both keys with the same ref, so they move in lockstep by construction rather than by instruction, and the committed value is aligned to the digest `core/platform` already carried. Moving the runner forward two minors is safe: it is used only as a curl + jq container, and the image built at `v1.5.0` carries them (`git show v1.5.0:…/migrations/Dockerfile` → `apk add … jq ca-certificates bash curl`). ## What should replace it A dedicated `ch-backup-client` image built by **this package's own Makefile** — measured at 23 MB, 15× smaller, with none of the cluster tooling and none of the entrypoint hazard. That also deletes the cross-package stamp this PR adds, because the pin would be produced by the package that owns it. The objection that has kept this reuse in place — "a second tag release CI would have to re-digest each cut" — does not survive contact with the actual defect. Every other first-party image already pays that cost, automatically. What broke here was not an extra tag; it was **a pin with no producer**, which a package-owned image cannot reproduce. I am deliberately not doing that here: it means a new image, a new build target and a new digest introduced during a release cut. It belongs in a follow-up. ## The guard `hack/image-pin-consistency.bats` asserts the general invariant — no repository pinned at more than one digest — driven through the real promotion selector so it tracks whatever set promotion actually acts on, rather than a reimplementation that could drift from it. Plus a specific lockstep assertion so a recurrence names its cause. This check could not have been added before now: it fails on the very duplicate this PR removes. ## Testing - `hack/image-pin-consistency.bats` — both tests pass; verified they fail when the drifted pin is restored. - `make bats-unit-tests` green (294 tests, exit 0). - `make -C packages/system/backupstrategy-controller test` — 11 tests, 3 suites, green. - `helm template … --set backupStorage.bucketNameOverride=test-bucket` renders the strategy Pod with the new ref. - The Makefile's `yq` step was simulated directly to confirm the relative path and key path resolve and that comments survive the in-place edit. - Merged locally with #3404 onto `main` and re-run: the expanded ref collector in that PR surfaces no additional duplicate pins (314 tests, exit 0). ## Relationship to other work Relates to #3143 — this is one instance of its unowned-pin class, the one its discussion flagged as having no producer. The four never-built packages there remain untouched. Independent of #3404 (promote/retag/mirror storage-shape coverage); the two touch no common files and can merge in either order. Both are needed before the `v1.6.0` promote is re-cut. ## Release note ```release-note Fixed a release-blocking duplicate pin: the ClickHouse backup client image is now stamped from the same source as the platform migrations image it reuses, so both carry one digest. Previously they could drift, which made promotion attempt to retag two digests to the same stable tag and fail. Reusing the migrations image here remains a documented workaround pending a dedicated backup-client image. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Ensured the platform migrations image digest is pinned consistently between the platform and backup strategy components. * Updated the backup strategy component to reference the new pinned `platform-migrations` digest. * **Tests** * Added promotion/retag consistency checks to fail on conflicting source digests for the same stable tag. * Added drift detection to ensure `platform-migrations` image references match across consumer configuration files. * **Documentation** * Expanded in-chart comments and values documentation clarifying the coupling and workaround behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 8ea5892 + a302bd0 commit bbfe9f9

4 files changed

Lines changed: 154 additions & 15 deletions

File tree

‎hack/image-pin-consistency.bats‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
#!/usr/bin/env bats
2+
# Asserts that no first-party image repository is pinned at more than one
3+
# digest across the committed tree.
4+
#
5+
# Promotion retags by digest: hack/promote-retag.sh copies every collected
6+
# <repo>@<digest> to <repo>:<stable-version>. Two different digests under one
7+
# repository therefore produce two copies competing for the same destination
8+
# tag. The first wins, the second hits the write-once guard, and the promotion
9+
# fails — on release day, in a workflow that runs once per release.
10+
#
11+
# This is not hypothetical. platform-migrations was pinned twice: once in
12+
# packages/core/platform/values.yaml (stamped by its Makefile) and once as
13+
# .backupStrategyController.chBackupClientImage, which had no producer at all
14+
# and so froze at v1.4.0-rc.2 while the other advanced to v1.5.0. v1.6.0 is the
15+
# first release cut through the promote path rather than a full rebuild, so it
16+
# would have been the first to hit it.
17+
#
18+
# The general invariant is cheaper to hold than the specific one: a duplicate
19+
# pin can arise from any package reusing another's image, which the tree does
20+
# deliberately (backupstrategy-controller reuses platform-migrations as a
21+
# curl+jq runner rather than shipping a second one-binary tag).
22+
#
23+
# Harness note: the CI path is hack/cozytest.sh, NOT real bats. There is no
24+
# `run`, `$status`, `$output`, `skip`, or setup()/teardown(); each test runs as
25+
# a shell function under `set -eu -x`, so a non-zero exit is the failure.
26+
# Paths are repo-root-relative: BATS_TEST_DIRNAME is unset and would abort the
27+
# whole suite under `set -u`.
28+
#
29+
# Run with: hack/cozytest.sh hack/image-pin-consistency.bats
30+
31+
@test "no image repository is pinned at more than one digest" {
32+
tmp=$(mktemp -d)
33+
34+
# Drive the real promotion selector rather than a reimplementation of it, so
35+
# this tracks whatever set promotion actually acts on. `env -u REGISTRY`:
36+
# the CI workflow exports REGISTRY=<OCIR build registry> for every job, but
37+
# the committed tree vendors its digests under the script's default
38+
# ghcr.io/cozystack/cozystack — inheriting the ambient value would filter for
39+
# the wrong registry and match nothing.
40+
rc=0
41+
env -u REGISTRY hack/promote-retag.sh v9.9.9 --dry-run \
42+
>"$tmp/out" 2>"$tmp/err" || rc=$?
43+
if [ "$rc" -ne 0 ]; then
44+
echo "promote-retag.sh exited $rc" >&2
45+
cat "$tmp/err" >&2
46+
rm -rf "$tmp"
47+
return "$rc"
48+
fi
49+
50+
# Every planned copy ends in docker://<repo>:v9.9.9. A repository appearing
51+
# twice means two source digests aimed at one destination tag.
52+
sed -n 's|.*docker://\(.*\):v9\.9\.9$|\1|p' "$tmp/out" | sort > "$tmp/dests"
53+
dupes=$(uniq -d < "$tmp/dests")
54+
55+
if [ -n "$dupes" ]; then
56+
echo "these repositories are pinned at more than one digest, so promotion" >&2
57+
echo "would try to retag several digests to the same stable tag and fail:" >&2
58+
printf '%s\n' "$dupes" >&2
59+
echo >&2
60+
echo "Offending pins:" >&2
61+
for repo in $dupes; do
62+
grep -rn --include='*.yaml' --include='*.tag' --exclude-dir=charts \
63+
-- "${repo#ghcr.io/cozystack/cozystack/}" packages/ >&2 || true
64+
done
65+
echo >&2
66+
echo "Give the duplicate key a producer that stamps the same ref (see" >&2
67+
echo "packages/core/platform/Makefile), or drop the duplicate pin." >&2
68+
rm -rf "$tmp"
69+
return 1
70+
fi
71+
rm -rf "$tmp"
72+
}
73+
74+
@test "platform-migrations is pinned identically in both consumers" {
75+
# The specific instance, pinned separately so a regression names its cause
76+
# rather than only tripping the general check above. The two must match
77+
# exactly, not merely resolve to the same digest: backupstrategy-controller
78+
# renders its copy into a Pod spec, so a stale tag string there is what an
79+
# operator reads when asking what is running.
80+
a=$(yq -r '.migrations.image' packages/core/platform/values.yaml)
81+
b=$(yq -r '.backupStrategyController.chBackupClientImage' \
82+
packages/system/backupstrategy-controller/values.yaml)
83+
84+
if [ "$a" != "$b" ]; then
85+
echo "platform-migrations pins have drifted:" >&2
86+
echo " packages/core/platform/values.yaml .migrations.image" >&2
87+
echo " $a" >&2
88+
echo " backupstrategy-controller/values.yaml .chBackupClientImage" >&2
89+
echo " $b" >&2
90+
echo >&2
91+
echo "packages/core/platform/Makefile stamps both; do not hand-edit either." >&2
92+
return 1
93+
fi
94+
}

‎packages/core/platform/Makefile‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,22 @@ image-migrations:
4242
--metadata-file images/migrations.json \
4343
$(BUILDX_ARGS)
4444
rm -rf images/migrations/etcd-operator-crds
45+
# Stamp BOTH consumers of this image. backupstrategy-controller renders it
46+
# into the Altinity strategy Pod as a curl+jq runner — a WORKAROUND, not a
47+
# design: that Pod needs only curl/jq/sleep/date but gets 349 MB including
48+
# kubectl, helm and etcd-migrate, and this image's ENTRYPOINT is the
49+
# migration runner itself. It should get a dedicated ~23 MB image owned by
50+
# its own package, which would delete this cross-package stamp; see the TODO
51+
# in ../../system/backupstrategy-controller/values.yaml.
52+
#
53+
# Until then the two pins MUST move together. The second copy had no
54+
# producer, so it silently froze at v1.4.0-rc.2 while this one advanced, and
55+
# a promotion then tried to retag two different digests to the same stable
56+
# tag — which the write-once guard in hack/promote-retag.sh turns into a
57+
# failed release. hack/image-pin-consistency.bats keeps them in lockstep.
4558
IMAGE="$(REGISTRY)/platform-migrations:$(IMAGE_TAG)@$$(yq -e -o json -r '.["containerimage.digest"]' images/migrations.json)" \
4659
yq --inplace '.migrations.image = strenv(IMAGE)' values.yaml
60+
IMAGE="$(REGISTRY)/platform-migrations:$(IMAGE_TAG)@$$(yq -e -o json -r '.["containerimage.digest"]' images/migrations.json)" \
61+
yq --inplace '.backupStrategyController.chBackupClientImage = strenv(IMAGE)' \
62+
../../system/backupstrategy-controller/values.yaml
4763
rm -f images/migrations.json

‎packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -23,13 +23,15 @@ spec:
2323
restartPolicy: Never
2424
containers:
2525
- name: ch-backup-client
26-
# platform-migrations is the pre-built, digested image release
27-
# CI maintains in packages/core/platform/values.yaml. It bundles
28-
# curl + jq + kubectl + helm + ca-certificates on alpine, so the
29-
# strategy Pod does not depend on dl-cdn.alpinelinux.org at run
30-
# time — works on air-gapped clusters. The image is reused
31-
# rather than a single-purpose ch-backup-client to avoid a
32-
# second tag that release CI would have to re-digest each cut.
26+
# WORKAROUND: this resolves to platform-migrations, which bundles
27+
# curl + jq + ca-certificates on alpine — so the Pod needs no
28+
# `apk add` and works air-gapped. That is the only reason it is
29+
# here. It is a 349 MB migration image standing in for a curl+jq
30+
# runner, and its own ENTRYPOINT is run-migrations.sh, so the
31+
# `command:` override below is load-bearing: drop it and this Pod
32+
# runs platform migrations with kubectl and helm on the PATH.
33+
# Replace with a dedicated ch-backup-client image owned by this
34+
# package; see the TODO in ../values.yaml.
3335
image: "{{ .Values.backupStrategyController.chBackupClientImage }}"
3436
imagePullPolicy: IfNotPresent
3537
# The clickhouse-backup HTTP API on :7171 enforces basic auth;

‎packages/system/backupstrategy-controller/values.yaml‎

Lines changed: 35 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,41 @@
11
backupStrategyController:
22
image: "ghcr.io/cozystack/cozystack/backupstrategy-controller:v1.5.0@sha256:d965283f935e4b24b655bc76e37dcb3b731bec832bb5c2e3e407b0284c7526a3"
33
# chBackupClientImage is the image rendered into the Altinity strategy
4-
# Pod: it drives clickhouse-backup's HTTP API via curl + jq. We reuse
5-
# platform-migrations (already pre-built and digested by release CI in
6-
# packages/core/platform/values.yaml) because it bundles curl + jq +
7-
# ca-certificates — avoids shipping a one-binary tiny image whose tag
8-
# would have to be re-digested on every release. Bumping the
9-
# platform-migrations version requires updating BOTH this value and
10-
# packages/core/platform/values.yaml so they remain in lockstep.
11-
chBackupClientImage: "ghcr.io/cozystack/cozystack/platform-migrations:v1.4.0-rc.2@sha256:17390197a9b11d76a8dd1b24c6d1512adb7d3226b28b5bcba8551b14b1ec4be0"
4+
# Pod: it drives clickhouse-backup's HTTP API via curl + jq.
5+
#
6+
# WORKAROUND — platform-migrations is the wrong image for this job and
7+
# should be replaced by a dedicated one (see the TODO below). It is
8+
# reused here only because it happens to bundle curl + jq, and this
9+
# release could not absorb a new image. What that reuse actually costs:
10+
# - the strategy script needs exactly curl, jq, sleep and date, yet
11+
# this image is 349 MB carrying kubectl, helm, git, cozyhr and the
12+
# etcd-migrate binary into a tenant-adjacent backup Pod;
13+
# - its ENTRYPOINT is run-migrations.sh, so only the `command:`
14+
# override in strategy-altinity-default.yaml stands between this Pod
15+
# and running platform migrations with kubectl and helm on the PATH;
16+
# - it couples the backup client to migrations: bumping
17+
# ETCD_OPERATOR_VERSION or editing any migration script re-digests
18+
# this image, forcing this pin to move for reasons that have nothing
19+
# to do with backups. That coupling is what broke — see below.
20+
#
21+
# TODO: replace with a dedicated ch-backup-client image built by THIS
22+
# package's own Makefile (alpine + curl + jq + ca-certificates measures
23+
# 23 MB, 15x smaller). That also removes the cross-package stamp
24+
# entirely, because the pin would then be produced by the package that
25+
# owns it. The often-cited objection — "a second tag release CI must
26+
# re-digest each cut" — is not the real cost: every other first-party
27+
# image already pays it, automatically. The real defect was a pin with
28+
# NO producer, which a package-owned image cannot reproduce.
29+
#
30+
# DO NOT hand-edit: packages/core/platform/Makefile stamps this key with
31+
# the same ref it writes to .migrations.image there, so the two stay in
32+
# lockstep by construction. They previously did not — this key had no
33+
# producer at all and froze at v1.4.0-rc.2 while the other advanced to
34+
# v1.5.0. Two digests under one repository means a promotion tries to
35+
# retag both to the same stable tag, and the write-once guard in
36+
# hack/promote-retag.sh fails the release. hack/image-pin-consistency.bats
37+
# asserts no repository is pinned at more than one digest.
38+
chBackupClientImage: "ghcr.io/cozystack/cozystack/platform-migrations:v1.5.0@sha256:8bf61f17e99eb4a3365394e3d97ac52a1d68cc84aa104894ecb09de25721dc1d"
1239
replicas: 2
1340
debug: false
1441
metrics:

0 commit comments

Comments
 (0)