Repository navigation
Commit bbfe9f9
authored
fix(backupstrategy-controller): give chBackupClientImage a producer (#3405)
## This is a workaround
`backupstrategy-controller` should not be running the platform
**migrations** image. This PR does not fix that — it makes the existing
reuse *safe* so the v1.6.0 promote can proceed, and documents in the
tree exactly what is wrong and what should replace it. Treat it as a
stopgap with a TODO, not as an endorsement of the design.
## What is actually wrong
The Altinity strategy Pod drives clickhouse-backup's HTTP API. Its
script uses exactly four binaries — `curl`, `jq`, `sleep`, `date`. For
that it currently pulls `platform-migrations`:
| | |
| --- | --- |
| Size | **349 MB** (a dedicated `alpine + curl + jq + ca-certificates`
measures **23 MB**) |
| Also contains | `kubectl`, `helm`, `git`, `cozyhr`, the `etcd-migrate`
binary, every migration script, vendored etcd CRDs |
| `ENTRYPOINT` | `run-migrations.sh` |
Two consequences worth naming. A tenant-adjacent backup Pod carries
cluster tooling it has no use for. And because the image's own
entrypoint is the migration runner, the `command:` override in the
strategy template is load-bearing — drop that one line and the Pod runs
platform migrations with `kubectl` and `helm` already on the PATH.
It also creates the coupling that caused the bug this PR fixes: bumping
`ETCD_OPERATOR_VERSION` or editing any migration script re-digests the
image, forcing the backup client's pin to move for reasons that have
nothing to do with backups.
## The release blocker
`platform-migrations` was pinned at two different digests in the
committed tree:
| Location | Pin | Producer |
| --- | --- | --- |
| `packages/core/platform/values.yaml` `.migrations.image` |
`v1.5.0@sha256:8bf61f17…` | `packages/core/platform/Makefile` |
| `backupstrategy-controller/values.yaml` `.chBackupClientImage` |
`v1.4.0-rc.2@sha256:17390197…` | **none** |
Promotion retags **by digest**: `hack/promote-retag.sh` copies every
collected `<repo>@<digest>` to `<repo>:<stable-version>`. Two digests
under one repository produce two copies competing for the same
destination tag — the first wins, the second hits the write-once guard,
and the promotion fails. A dry run against `main` emits exactly that
pair:
```
▸ ghcr.io/cozystack/cozystack/platform-migrations sha256:17390197…
→ docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9
▸ ghcr.io/cozystack/cozystack/platform-migrations sha256:8bf61f17…
→ docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9
```
`v1.6.0` is the first release cut through promotion rather than a full
rebuild. Every earlier release ran `make build`, which restamped both
copies as a side effect and hid the missing producer.
## What this PR does
`packages/core/platform/Makefile` now stamps both keys with the same
ref, so they move in lockstep by construction rather than by
instruction, and the committed value is aligned to the digest
`core/platform` already carried. Moving the runner forward two minors is
safe: it is used only as a curl + jq container, and the image built at
`v1.5.0` carries them (`git show v1.5.0:…/migrations/Dockerfile` → `apk
add … jq ca-certificates bash curl`).
## What should replace it
A dedicated `ch-backup-client` image built by **this package's own
Makefile** — measured at 23 MB, 15× smaller, with none of the cluster
tooling and none of the entrypoint hazard. That also deletes the
cross-package stamp this PR adds, because the pin would be produced by
the package that owns it.
The objection that has kept this reuse in place — "a second tag release
CI would have to re-digest each cut" — does not survive contact with the
actual defect. Every other first-party image already pays that cost,
automatically. What broke here was not an extra tag; it was **a pin with
no producer**, which a package-owned image cannot reproduce.
I am deliberately not doing that here: it means a new image, a new build
target and a new digest introduced during a release cut. It belongs in a
follow-up.
## The guard
`hack/image-pin-consistency.bats` asserts the general invariant — no
repository pinned at more than one digest — driven through the real
promotion selector so it tracks whatever set promotion actually acts on,
rather than a reimplementation that could drift from it. Plus a specific
lockstep assertion so a recurrence names its cause. This check could not
have been added before now: it fails on the very duplicate this PR
removes.
## Testing
- `hack/image-pin-consistency.bats` — both tests pass; verified they
fail when the drifted pin is restored.
- `make bats-unit-tests` green (294 tests, exit 0).
- `make -C packages/system/backupstrategy-controller test` — 11 tests, 3
suites, green.
- `helm template … --set backupStorage.bucketNameOverride=test-bucket`
renders the strategy Pod with the new ref.
- The Makefile's `yq` step was simulated directly to confirm the
relative path and key path resolve and that comments survive the
in-place edit.
- Merged locally with #3404 onto `main` and re-run: the expanded ref
collector in that PR surfaces no additional duplicate pins (314 tests,
exit 0).
## Relationship to other work
Relates to #3143 — this is one instance of its unowned-pin class, the
one its discussion flagged as having no producer. The four never-built
packages there remain untouched.
Independent of #3404 (promote/retag/mirror storage-shape coverage); the
two touch no common files and can merge in either order. Both are needed
before the `v1.6.0` promote is re-cut.
## Release note
```release-note
Fixed a release-blocking duplicate pin: the ClickHouse backup client image is now stamped from the same source as the platform migrations image it reuses, so both carry one digest. Previously they could drift, which made promotion attempt to retag two digests to the same stable tag and fail. Reusing the migrations image here remains a documented workaround pending a dedicated backup-client image.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Ensured the platform migrations image digest is pinned consistently
between the platform and backup strategy components.
* Updated the backup strategy component to reference the new pinned
`platform-migrations` digest.
* **Tests**
* Added promotion/retag consistency checks to fail on conflicting source
digests for the same stable tag.
* Added drift detection to ensure `platform-migrations` image references
match across consumer configuration files.
* **Documentation**
* Expanded in-chart comments and values documentation clarifying the
coupling and workaround behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->4 files changed
Lines changed: 154 additions & 15 deletions
File tree
- hack
- packages
- core/platform
- system/backupstrategy-controller
- templates
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
45 | 58 | | |
46 | 59 | | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
47 | 63 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
33 | 35 | | |
34 | 36 | | |
35 | 37 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | | - | |
6 | | - | |
7 | | - | |
8 | | - | |
9 | | - | |
10 | | - | |
11 | | - | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
12 | 39 | | |
13 | 40 | | |
14 | 41 | | |
| |||
0 commit comments