Repository navigation
backups: CNPG BackupJob fails with a message removed before the pinned controller tag #4258
Description
Describe the bug
postgres-2-backup-roundtrip fails with a BackupJob message that the pinned controller release cannot print. The wording was replaced a week before the tag the lane installs, so whatever reconciled that BackupJob was not built from the commit the pin names.
Environment
- Cozystack:
main - In-tree e2e lane, run 34891318171
To Reproduce
Run the full chainsaw suite. postgres-2-backup-roundtrip fails at its run step.
Expected behaviour
The BackupJob succeeds, or fails with a message the deployed code can print.
Actual behaviour
The BackupJob carries Reason: BackupFailed, started and completed at 21:14:10 in this run:
failed to apply barmanObjectStore to Cluster: admission webhook "vcluster.cnpg.io" denied the request: Cluster.cluster.cnpg.io "postgres-pg-src" is invalid: spec.plugins: Invalid value: ["barman-cloud.cloudnative-pg.io"]: Cannot enable a WAL archiver plugin when barmanObjectStore is configured
Logs
The outer wording is the controller's. It is in neither the tree nor the pinned tag:
$ git grep -c "apply barmanObjectStore" v1.6.0; echo "exit $?"
exit 1
At v1.6.0 the same call site reads differently:
$ git grep -n "to Cluster: %v" v1.6.0 -- internal/backupcontroller/cnpgstrategy_controller.go
231: return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to attach barman-cloud plugin to Cluster: %v", err))
The old wording went away in 13ff34027 on 2026-07-15, and that commit is an ancestor of the tag:
$ git merge-base --is-ancestor 13ff34027 v1.6.0; echo "exit $?"
exit 0
The pin the lane installs:
$ head -2 packages/system/backupstrategy-controller/values.yaml
backupStrategyController:
image: "ghcr.io/cozystack/cozystack/backupstrategy-controller:v1.6.0@sha256:e8bf0521..."
cmd/backupstrategy-controller is the only binary here importing internal/backupcontroller, so nothing else in this repo writes that status.
Additional context
The supported-Kind list cannot narrow this down. It is byte-identical at 13ff34027^ and at v1.6.0, so the seven kinds the binary named in its other failures fit a pre-migration build as well as the tagged one.
Two readings fit and the log separates neither: the image tagged v1.6.0 is not built from v1.6.0, or the lane deployed something other than the pinned digest. To settle it, read the image reference off the running Deployment in this run's crust-gather snapshot, or pull sha256:e8bf0521... and grep the binary for both wordings.
Not the same as #4257. That one is about this controller refusing strategy Kinds added after the tag, which covers the four round-trip suites it names. It does not cover this failure: the wording here was removed before the tag, so a correctly built v1.6.0 could not print it.
Checklist
- I have checked the documentation
- I have searched for similar issues
- I have included all required information
- I have provided clear steps to reproduce
- I have included relevant logs
Activity
scooby87 commentedon Sep 15, 2026
Investigated the provenance question this issue raises. Confirmed reading #1: the image published as
v1.6.0was not built from the source the tag names — it carries pre-migration code, which is why the BackupJob prints a message that was removed before the tag.Pin under investigation (
mainHEAD 070dcb1,packages/system/backupstrategy-controller/values.yaml):backupstrategy-controller:v1.6.0@sha256:e8bf0521….Provenance chain (git):
e8bf0521was first stamped ind95685c42("Prepare release v1.6.0-rc.4"), taggedv1.6.0-rc.4.235d38dae/764e30418) rewrote only the tag stringrc.4→v1.6.0and kept the digest (by-digest promotion, perdocs/agents/image-refs.md). Sov1.6.0@e8bf0521is literally therc.4build.d95685c42still carries the old wording (cnpgstrategy_controller.go:224), and the barman-cloud migration13ff34027is not an ancestor ofd95685c42. So the image was built from a pre-migration tree.v1.6.0tag's own source (764e30418) does contain13ff34027(the new wording), so the tag's source and the tagged binary diverge — exactly the split this issue reports.Binary proof (pulled the image, no crust-gather needed): downloaded layer
e8bf0521from ghcr and grepped the binary.So the tagged
v1.6.0binary predates the migration by about six hours.Blast radius: only
v1.6.0is affected.v1.6.1/v1.6.2/v1.6.3pin different, newer digests, so a natural rebuild already carries the post-migration code.Why
main's e2e lane still fails: committed first-party digests onmainadvance only through release-prep commits;mainstill carries the last prep'sv1.6.0digeste8bf0521while its source is post-migration, so the lane installs a stale pin.Not a repo defect, and no honest in-repo edit fixes it. The stamping (root
Makefilebuild list, by-digest promotion) is correct by design; the defect is release timing — a behaviour-changing source change merged after the last RC (rc.4) and norc.5was cut, so by-digest promotion shipped the stale binary underv1.6.0. Fixingmainmeans a rebuild + re-stamp through CI (the digest cannot be hand-written), and re-pinning to av1.6.1digest would create a tag/digest mismatch of its own.Suggested resolution (needs registry/release access): re-cut and re-stamp
v1.6.0, or bumpmain's pin to a rebuilt digest. Optionally add a provenance gate tohack/verify-promoted-packages.shthat refuses a pinned first-party digest whose build diverges from the promoted source, so this class cannot recur.Same root cause as #4262. Related to #4257 (the frozen
v1.6.0pin lagging the tree), though that issue's angle is the supported-Kind drift, not the message provenance.