You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Helm lookup-gated backup objects are skipped once and never created, which makes migration 50 fail-closed #3518
Several objects in packages/system/backupstrategy-controller and packages/system/bucket are gated on a Helm lookup of an object that the same release (or a sibling release) is still creating. When the lookup comes back
empty the object is silently skipped — and because helm-controller does not re-render without a chart or values
change (drift detection is off by default), it is never created at all.
On our cluster, months after install, the following were all missing:
the 7 default backup Strategy CRs (cozy-default-etcd, -cnpg, -mariadb, -altinity, -foundationdb, -velero-vmdisk, -velero-vminstance), gated on {{- if $bucketName -}} where bucketName comes from a lookup of BucketClaim;
the Velero BackupStorageLocation (velero-bsl.yaml), same gate;
the <bucket>-<user>-credentials Secret (packages/system/bucket/templates/user-credentials.yaml), gated on a lookup of the COSI Secret.
Only BackupClass cozy-default existed, because its template is rendered unconditionally — its own comment even
predicts this situation and assumes "Flux re-renders the chart", which does not happen.
Why it matters
Migration 50 (etcd adoption, v1.6.0) is fail-closed on the snapshot target and reboots its Job forever:
if the target cannot be resolved we FAIL LOUDLY rather than adopt without a snapshot
So a cluster that hit this race cannot be upgraded to v1.6.0 at all, and the documented escape hatch
(migrations.etcdAdoptSkipBackup) means adopting a live etcd — potentially a Kamaji datastore — with no snapshot.
The install timing that caused it was a 4-second window: backupstrategy-controller deployed at 14:10:24, the cozy-backups Bucket at 14:10:28.
Recovering required forcing a real Helm upgrade (reconcile.fluxcd.io/forceAtandrequestedAt; a plain
reconcile does nothing) on two releases, in order, because the race is nested: first backupstrategy-controller, then bucket-<name>-system for the credentials Secret. That second one is easy to
miss: the Secret is rendered by the -system release, not by bucket-<name>.
Suggested fix
Do not gate object creation on a render-time lookup. Either have backupstrategy-controller reconcile these
objects itself (it is a controller and already watches the relevant types), or make the Bucket provisioning a dependsOn so ordering is guaranteed instead of hoped for.
If the templating approach is kept, the chart should at least fail loudly (fail) rather than silently emit
nothing, so the condition is visible at install time instead of years later during an upgrade.
Activity
fix(backupstrategy-controller): repair lookup-gated backup objects (#…