Skip to content

Commit 51ad226

Browse files
fix(backupstrategy-controller): repair lookup-gated backup objects
The default Strategy CRs and the Velero BackupStorageLocation are Helm-templated behind a `lookup` of the BucketClaim the same chart creates, and the <bucket>-<user>-credentials Secret behind a `lookup` of the COSI Secret. When those lookups are empty at install time the objects were silently skipped, and helm-controller does not re-render a release whose chart and values did not change (drift detection is off on operator-generated HelmReleases), so the skip was permanent: clusters ran for months with only BackupClass cozy-default and no Strategy CRs at all. That also fail-closes the pre-adoption snapshot in the v1.6.0 etcd migration, which reads the projected credentials Secret. Add a DefaultObjectsGate runnable to backupstrategy-controller. Once the bucket name is resolvable from the projector's source Secret, it checks that every object cozy-default routes to exists and forces one real Helm upgrade (reconcile.fluxcd.io/forceAt + requestedAt) when any is missing. Helm remains the objects' only author; the gate only makes sure the render that produces them actually happens. It cannot be a render-time `fail` in this chart: the chart is the producer of the Bucket its own lookup reads, so a failed render would deadlock the condition. The bucket chart's user-credentials template CAN fail safely — the BucketAccess that produces the COSI Secret belongs to the parent release and that release retries forever — so make it fail loudly instead of skipping, with requireUserCredentials=false as the offline-render escape. Also correct the chart and doc comments that claimed Flux re-renders on its interval, and document the two-release manual recovery for clusters already affected. Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent f1f3836 commit 51ad226

13 files changed

Lines changed: 1122 additions & 17 deletions

File tree

‎cmd/backupstrategy-controller/main.go‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ import (
2727

2828
corev1 "k8s.io/api/core/v1"
2929
"k8s.io/apimachinery/pkg/runtime"
30+
"k8s.io/apimachinery/pkg/types"
3031
utilruntime "k8s.io/apimachinery/pkg/util/runtime"
3132
clientgoscheme "k8s.io/client-go/kubernetes/scheme"
3233
ctrl "sigs.k8s.io/controller-runtime"
@@ -203,6 +204,29 @@ func main() {
203204
}
204205
}
205206

207+
// The default Strategy CRs and the Velero BSL are Helm-templated behind
208+
// a lookup of the BucketClaim the same chart creates, so a fresh install
209+
// renders them empty — permanently, because helm-controller does not
210+
// re-render an unchanged, successful release. The gate detects that and
211+
// forces the one real Helm upgrade that materialises them. Disabled when
212+
// the HelmRelease coordinates are not plumbed (e.g. a chart-less local run).
213+
if hrName := os.Getenv("BACKUP_DEFAULT_OBJECTS_HELMRELEASE_NAME"); hrName != "" {
214+
gate := &backupcontroller.DefaultObjectsGate{
215+
Client: mgr.GetClient(),
216+
Config: credentialsConfig,
217+
BackupClassName: os.Getenv("BACKUP_DEFAULT_OBJECTS_BACKUPCLASS"),
218+
HelmRelease: types.NamespacedName{
219+
Namespace: os.Getenv("BACKUP_DEFAULT_OBJECTS_HELMRELEASE_NAMESPACE"),
220+
Name: hrName,
221+
},
222+
VeleroNamespace: os.Getenv("BACKUP_DEFAULT_OBJECTS_VELERO_NAMESPACE"),
223+
}
224+
if err := gate.SetupWithManager(mgr); err != nil {
225+
setupLog.Error(err, "unable to add DefaultObjectsGate runnable")
226+
os.Exit(1)
227+
}
228+
}
229+
206230
if err = (&backupcontroller.BackupJobReconciler{
207231
Client: mgr.GetClient(),
208232
Scheme: mgr.GetScheme(),

‎docs/operations/backup-classes.md‎

Lines changed: 44 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,9 +93,46 @@ On a fresh-cluster install, the Velero `BackupStorageLocation` `cozy-default` is
9393

9494
### Cozy-default Bucket bootstrap
9595

96-
`cozy-default` ships an `apps.cozystack.io/Bucket cozy-backups` CR in `tenant-root`, which the bucket-application chart turns into a `BucketClaim`; the COSI driver then assigns the real S3 bucket name and writes it to the BucketClaim's `.status.bucketName`. The strategy templates and the Velero BSL all read that real bucket name (Helm `lookup` against the BucketClaim). On a fresh install the BucketClaim takes a short reconcile cycle to populate its status — until it does, the strategy templates render empty and only the `Bucket` CR + `BackupClass` are present in the cluster. The HelmRelease re-reconciles on its interval (5 minutes by default — set by the cozystack operator's `helmrelease-interval` flag, not a Flux default), at which point the populated BucketClaim status causes the missing strategy templates to materialise.
96+
`cozy-default` ships an `apps.cozystack.io/Bucket cozy-backups` CR in `tenant-root`, which the bucket-application chart turns into a `BucketClaim`; the COSI driver then assigns the real S3 bucket name (`bucket-<claim-UID>`, so it cannot be computed in advance) and writes it to the BucketClaim's `.status.bucketName`. The strategy templates and the Velero BSL all read that real bucket name (Helm `lookup` against the BucketClaim). On a fresh install the BucketClaim takes a reconcile cycle to populate its status — until it does, the strategy templates render empty and only the `Bucket` CR + `BackupClass` are present in the cluster.
9797

98-
If you need the BackupClass functional immediately (e.g. an e2e), trigger a Flux reconcile (`flux reconcile helmrelease backupstrategy-controller -n cozy-backup-controller`) once you see `kubectl get bucketclaim -n tenant-root bucket-cozy-backups -o jsonpath='{.status.bucketName}'` non-empty.
98+
**That skip does not repair itself on a reconcile.** helm-controller re-renders a release only when its chart or values change; the interval reconcile is a no-op for a healthy release, and drift detection is off on operator-generated HelmReleases. A cluster that lost the race at install time therefore keeps `BackupClass cozy-default` with **no** `Strategy` CRs and **no** Velero BSL indefinitely — which also fail-closes the pre-adoption snapshot in the v1.6.0 etcd migration.
99+
100+
Convergence is driven instead by the controller's default-objects gate (`backupStorage.reconcileDefaultObjects`, on by default). Once the bucket name is resolvable it checks that every object `cozy-default` routes to exists, and stamps `reconcile.fluxcd.io/forceAt` + `requestedAt` on the `backupstrategy-controller` HelmRelease to force the real Helm upgrade that re-runs the lookups. Expect the objects within one minute of the bucket becoming ready. Watch it with:
101+
102+
```bash
103+
kubectl -n cozy-backup-controller logs deploy/backupstrategy-controller | grep default-objects-gate
104+
```
105+
106+
#### Manual recovery on an affected cluster
107+
108+
Only needed on a cluster running a version without the gate (or with `reconcileDefaultObjects: false`). A plain `flux reconcile helmrelease` does **nothing** here — it does not re-render. You need a forced upgrade, and **both** annotations: `forceAt` is what makes helm-controller run a real Helm upgrade, and it is only honoured together with `requestedAt`.
109+
110+
First confirm the bucket name is actually resolvable — forcing before that just re-runs the same empty lookup:
111+
112+
```bash
113+
kubectl -n tenant-root get bucketclaim bucket-cozy-backups -o jsonpath='{.status.bucketName}'
114+
```
115+
116+
Then force the two releases, **in this order**:
117+
118+
```bash
119+
# 1. The credentials Secret. It is rendered by the -system release, not by
120+
# bucket-cozy-backups — easy to miss, and the projector (hence every
121+
# strategy and Velero) has no source without it.
122+
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
123+
kubectl -n tenant-root annotate helmrelease bucket-cozy-backups-system \
124+
reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite
125+
kubectl -n tenant-root get secret bucket-cozy-backups-system-credentials
126+
127+
# 2. The Strategy CRs and the Velero BSL.
128+
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
129+
kubectl -n cozy-backup-controller annotate helmrelease backupstrategy-controller \
130+
reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite
131+
kubectl get $(kubectl get crd -o name | grep strategy.backups.cozystack.io) 2>/dev/null
132+
kubectl -n cozy-velero get backupstoragelocation cozy-default
133+
```
134+
135+
The race is nested: step 2's `lookup` resolves from the BucketClaim status, but the projector — and the v1.6.0 etcd migration — read the Secret from step 1, so a cluster missing both needs both.
99136

100137
### Observability
101138

@@ -106,6 +143,11 @@ The credentials projector emits two Prometheus counters labelled by `namespace`
106143

107144
Alert on `rate(cozystack_backup_credentials_projection_failures_total[5m]) > 0` or `absent_over_time(cozystack_backup_credentials_projection_successes_total[10m])` to catch a stale BSL credential or a malformed source Secret without log scraping.
108145

146+
The default-objects gate emits two more:
147+
148+
- `cozystack_backup_default_objects_missing{backupclass="cozy-default"}` — how many objects `cozy-default` routes to are absent. **This is the alert that would have caught the missing Strategy CRs**: it is non-zero whatever the HelmRelease's `Ready` condition says. Alert on `min_over_time(cozystack_backup_default_objects_missing[15m]) > 0`.
149+
- `cozystack_backup_default_objects_force_reconciles_total` — forced Helm upgrades issued. A counter that keeps climbing means the forced render is not producing the objects (a missing CRD, for instance), which is a different problem from the install-time race.
150+
109151
## Admin overrides for `cozy-default`
110152

111153
`cozy-default` is rendered by the `backupstrategy-controller` chart and owned by Flux's helm-controller. **Direct `kubectl edit backupclass cozy-default` is overwritten on the next helm reconcile** — the same applies to its companion `strategy.backups.cozystack.io/*` CRs (`cozy-default-cnpg`, `cozy-default-etcd`, `cozy-default-mariadb`, `cozy-default-altinity`, `cozy-default-foundationdb`, the two `cozy-default-velero-*`). The supported override path is the `backupStorage` block on the **`platform` component** of the `cozystack.cozystack-platform` Package CR:

0 commit comments

Comments
 (0)