Skip to content

Commit cfaf2e0

Browse files
authored
fix(backupstrategy-controller): repair lookup-gated backup objects (#3524)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Fixes #3518. Objects in `packages/system/backupstrategy-controller` and `packages/system/bucket` are gated on a Helm `lookup` of an object that is still being created. When the lookup is empty the object is skipped — and since helm-controller does not re-render a release whose chart and values did not change (drift detection is off on operator-generated HelmReleases), **the skip is permanent**, not a bootstrap window. On the reporter's cluster the install-timing window was 4 seconds and the objects were still missing months later: - the 7 default backup `Strategy` CRs and the Velero `BackupStorageLocation`, gated on `include "backupstrategy-controller.bucketName"` → a `lookup` of `BucketClaim`; - the `<bucket>-<user>-credentials` Secret (`packages/system/bucket/templates/user-credentials.yaml`), gated on a `lookup` of the COSI Secret. Only `BackupClass cozy-default` was present, because it renders unconditionally. Its own comment predicted this case while assuming "Flux re-renders the chart", which does not happen — the same trap `packages/core/platform/sources/cozystack-basics.yaml` already documents: *"Drift detection is off on operator-generated HelmReleases, so a capability gate in the templates would render the policy out at first install and never add it back."* Consequence: migration 50 (etcd adoption, v1.6.0) is fail-closed on the snapshot target. It resolves the target from the projected credentials Secret, which has no source without `<bucket>-<user>-credentials`, so its Job reboots forever and the cluster cannot be upgraded at all. ### 1. `backupstrategy-controller` reconciles the objects into existence New leader-elected `DefaultObjectsGate` runnable (`internal/backupcontroller/default_objects_gate.go`), modelled on the existing `SystemCredentialsProjector`. Every minute: - resolves the bucket name from the **projector's source Secret** (`parseSourceSecret`) — no new dependency, no `lookup`, and it covers the external-S3 path too. While it is unresolvable the gate does nothing: forcing then would only re-run the same empty lookup; - reads `BackupClass cozy-default` as the manifest of what must exist (its `strategyRef`s), plus the `cozy-default` BSL when `velero.bslEnabled`; - when any is missing, stamps `reconcile.fluxcd.io/forceAt` **and** `requestedAt` on its own HelmRelease. Both are required: `requestedAt` alone only requests a reconcile, which is a no-op for an unchanged release; `forceAt` is what triggers a real Helm upgrade, and it is only honoured together with `requestedAt`; - throttled to one forced upgrade per 5 minutes, and a no-op in the steady state. A `strategyRef` whose CRD is not installed is not counted as missing, so a missing CRD cannot drive an endless force loop. Two new metrics: `cozystack_backup_default_objects_missing{backupclass}` — the alert that would have caught this, since it is non-zero regardless of the HelmRelease's `Ready` condition — and `cozystack_backup_default_objects_force_reconciles_total`. RBAC delta is two lines: `patch` on `helmreleases` (`get` was already there) and `get` on `backupstoragelocations`. Both accesses are point Gets through the dynamic client, so no cluster-wide informer and no `list`/`watch`. Off switch: `backupStorage.reconcileDefaultObjects: false`. **Why not a render-time `fail` in this chart:** it would deadlock. `templates/bucket.yaml` in this same chart is the producer of the `Bucket` whose `BucketClaim` the lookup reads, so a failed render never applies it and the condition can never resolve. **Why not template the objects in Go:** their bodies are values-driven (endpoint, region, `forcePathStyle`, the Altinity strategy's whole PodTemplateSpec). Helm stays their single author; the gate only makes sure the render that produces them actually happens. **Why not `dependsOn`:** an edge guarantees the `Bucket` CR is *applied*, not that its claim is *reconciled*, so it does not close the race. ### 2. The gate also repairs the credentials Secret, one release earlier The `<bucket>-<user>-credentials` Secret is the same trap as the Strategy CRs, one release up the chain: `packages/system/bucket/templates/user-credentials.yaml` renders it behind a `lookup` of the COSI Secret, and skips it just as permanently. It is also the *first* domino — the projector reads it, so without it there is no `cozy-backups-creds`, no strategy, no Velero, and no migration 50. An earlier revision of this PR made that render `fail` so the release would retry and converge. [@IvanHunters pointed out](#3524 (comment)) that Helm cannot render a partial set, so the `fail` aborts the **whole** `<bucket>-system` release — the other users' Secrets and the bucket UI Deployment/Service/Ingress/HTTPRoute with them. One declared user whose `BucketAccess` never provisions would take down every other user of that bucket, and on an installed release park it in `Failed`, blocking every later upgrade. That was a real blast-radius regression on a universal path, and it is gone. The repair now lives in the gate, where it is per-object: - while the source Secret is absent, or carries no bucket name, the gate stamps `forceAt` + `requestedAt` on the bucket's `bucket-<name>-system` HelmRelease instead of doing nothing; - **no self-deadlock:** the `BucketClaim` and the `BucketAccess` whose COSI Secret that lookup reads are rendered unconditionally by the **parent** release (`packages/apps/bucket/templates/bucketclaim.yaml`), not by the one being forced; - **no RBAC delta:** `patch` on `helmreleases` is already cluster-scoped; - the two releases are throttled **independently**. They are forced in sequence on a bootstrap (credentials, then the objects the resolved bucket unblocks), so one shared timestamp would delay the second by a full `MinForceInterval` for no reason; - coordinates come from the chart under `provisionBucket`, and are omitted on external S3, where the Secret is admin-managed and no release renders it. The bucket chart keeps its partial render. Its comment now explains why the skip is not self-healing and why failing is the wrong lever, and a test renders the UI with *every* user unresolvable so a reintroduced `fail` fails the suite. **Scope this narrows, deliberately:** tenant buckets keep the pre-existing silent skip. The gate owns only the platform bucket, which is the cluster-breaking path. Generalising the repair to every bucket needs a controller that owns tenant buckets and is a separate change. Two more corrections from the same review: - `cozystack_backup_default_objects_missing` was only written on the happy path, so the state this PR exists to catch reported `0` and the documented alert never fired. The credentials Secret now counts in `missing`, so an absent or rotated-away Secret moves the gauge. On an API error the gauge is deliberately left alone rather than flapping, and a new `cozystack_backup_default_objects_check_errors_total` marks it stale; the doc pairs the two instead of overstating the gauge. `force_reconciles_total` gained `namespace`/`name`, since two releases can now be forced. - `forceHelmRelease` does a point `Get` and skips a release with `spec.suspend: true`. helm-controller ignores both annotations while suspended (`cozyhr suspend` sets exactly that), so the gate was re-stamping every `MinForceInterval` for the whole suspension and climbing the force counter — which the runbook attributes to a render that is not producing the objects. The skip is logged and not counted. ### 3. Corrected comments and docs `_helpers.tpl`, `backupclass-default.yaml`, `values.yaml` and `docs/operations/backup-classes.md` all claimed the skip self-heals on the HelmRelease interval, and the doc suggested a `flux reconcile` that does not re-render. They now say it does not, and why. The two-release manual recovery for already-affected clusters is documented, including the easy-to-miss part: the credentials Secret is rendered by the `bucket-<name>-system` release, not by `bucket-<name>`. ### Manual recovery for already-affected clusters Only needed on a version without the gate. A plain `flux reconcile helmrelease` does **nothing** — it does not re-render. ```bash # 0. Confirm the bucket name is resolvable first — forcing before that # just re-runs the same empty lookup. kubectl -n tenant-root get bucketclaim bucket-cozy-backups -o jsonpath='{.status.bucketName}' # 1. The credentials Secret. Rendered by the -system release, NOT by # bucket-cozy-backups — easy to miss, and the projector (hence every # strategy, Velero, and migration 50) has no source without it. ts=$(date -u +%Y-%m-%dT%H:%M:%SZ) kubectl -n tenant-root annotate helmrelease bucket-cozy-backups-system \ reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite # 2. The Strategy CRs and the Velero BSL. ts=$(date -u +%Y-%m-%dT%H:%M:%SZ) kubectl -n cozy-backup-controller annotate helmrelease backupstrategy-controller \ reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite ``` ### Validation - `go test ./internal/... -count=1` — pass, including 15 cases for the gate: force on missing, no-op in the steady state, throttling, unmapped kinds ignored, BSL skipped when disabled, patch failure surfaced without advancing the throttle, absent HelmRelease tolerated, plus force-on-absent-Secret, force-on-empty-bucket-name, external-S3 reports-but-does-not-force, independent throttles, suspend skip on each of the two releases, gauge written on the unresolved path, and errors counter on a failed check. - `make helm-unit-tests` — pass, with 2 new suites: 3 cases for the bucket chart (partial render preserved, and the UI still renders with every user unresolvable — the guard against reintroducing the `fail`), 7 for the gate's env + RBAC wiring. Those include that `.Release.Name` and `.Release.Namespace` match the HelmRelease the operator generates (`releaseName` becomes the HR's `metadata.name`, and no `spec.releaseName`/`targetNamespace` is set, so the Helm release name equals the HR name), that the bucket release coordinates track a renamed bucket together with `systemSecretName`, and that they are unwired on `provisionBucket: false`. - `make generate` — no diff. - `make go-unit-tests rd-presets-check test-check-readiness migrations-target-check` — pass. - `make bats-unit-tests` — every suite passes except `hack/migration-seaweedfs-db-adopt.bats`, which needs a Docker daemon not available in this environment ("docker is required: these tests run the migrations inside alpine"). Unrelated to this diff. **Not verified — please read this before approving.** There was still no live-cluster run. What that gap covers is smaller than it was: dropping the chart `fail` removed the universal behaviour change on every bucket's install path, so there is no longer a render-blind failure mode whose convergence has to be observed. What remains is one mechanism — `forceAt` + `requestedAt` on an unchanged release — applied to two releases instead of one, argued from read code rather than an observed install: `Strategy.Name = RetryOnFailure` on operator-generated HelmReleases (`internal/operator/package_reconciler.go`), and `bucketName = "bucket-" + claim.UID` in `packages/system/objectstorage-controller/images/objectstorage/patches/92-bucketclaim-propagate-ready.diff`. It is the same procedure as the manual recovery below, which has been run by hand on the reporter's cluster. Someone with a bootstrap cluster should still confirm that the gate fires within a minute or two of the bucket becoming ready, and that forcing `bucket-<name>-system` materialises the credentials Secret. ### Screenshots Not applicable: no UI change. This PR touches a controller, two system charts, and docs. <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> Walked file by file against the diff of this PR. **`website` — affected, follow-up issue opened.** `content/en/docs/next/operations/services/backup-classes.md` mirrors this repo's `docs/operations/backup-classes.md` and carries the exact stale claim this PR corrects, plus a `flux reconcile helmrelease` recovery suggestion that has no effect. `operations/configuration/platform-package.md` enumerates the forwarded `backupStorage` keys, and `reconcileDefaultObjects` is admin-settable through `spec.components.platform.values.backupStorage` (the whole block is deep-merged), so that list becomes incomplete. Filed as an issue rather than a PR because the wording depends on the shape this PR lands in, and whether to touch the frozen `v1.5`/`v1.6` copies is a maintainer call. No existing website issue or PR covered it. **`terraform-provider-cozystack` — not affected.** Checked, not assumed: `packages/apps/**` is untouched by this diff (`git show --stat HEAD -- packages/apps/` is empty), so no app `values.schema.json`, no `values.yaml` default, no version enum, and no `ApplicationDefinition` `kind`/`plural`/`release.prefix` changed. `reconcileDefaultObjects` is a key on a **system** chart, which has no `values.schema.json` and is not modelled by the provider. (An earlier revision added `requireUserCredentials` to the `bucket` system chart; that is gone, and no key was added to `apps/bucket`, so the app API is untouched.) The `<bucket>-<user>-credentials` Secret the provider looks up by literal name is **not renamed** — this PR only changes whether it gets created, which if anything removes a case where the provider silently read null from a Secret that never existed. Everything else: no `hack/` layout or make-target change (`ccp`, `external-apps-example`, `cozyhr`), no `ApplicationDefinition` CRD change, no namespace rename, no node prerequisites in `hack/e2e-prepare-cluster.bats` (`talm`, `ansible-cozystack`), no `cozyhr.cozystack.io/values-files` annotation or chart-source-kind change, no `service-proxy-name` / `wholeIP` / `allowICMP` rename (`cozy-proxy`), no telemetry metric or label rename (`cozystack-telemetry-server` — the two metrics here are new, not renamed), no node or network requirement change (`examples`). - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#639 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(backupstrategy-controller): the default backup Strategy CRs, the Velero BackupStorageLocation and the bucket user-credentials Secret were skipped permanently when their install-time Helm `lookup` came back empty, leaving clusters with a BackupClass but no strategies and blocking the v1.6.0 etcd migration. The controller now detects the missing objects and forces the Helm upgrade that creates them, including on the bucket release that renders the credentials Secret. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added automatic detection and recovery for missing default backup resources. - Missing resources trigger a throttled Helm upgrade once bucket configuration is available. - Added configuration to enable or disable default-object reconciliation. - Added validation requiring user credential Secrets during bucket installation by default, with an offline rendering option. - **Bug Fixes** - Improved recovery from incomplete backup and Velero storage configuration. - **Documentation** - Expanded operational guidance, monitoring metrics, alerts, and manual recovery steps. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 9807152 + f859b71 commit cfaf2e0

12 files changed

Lines changed: 1790 additions & 20 deletions

File tree

‎cmd/backupstrategy-controller/main.go‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ import (
2727

2828
corev1 "k8s.io/api/core/v1"
2929
"k8s.io/apimachinery/pkg/runtime"
30+
"k8s.io/apimachinery/pkg/types"
3031
utilruntime "k8s.io/apimachinery/pkg/util/runtime"
3132
clientgoscheme "k8s.io/client-go/kubernetes/scheme"
3233
ctrl "sigs.k8s.io/controller-runtime"
@@ -203,6 +204,39 @@ func main() {
203204
}
204205
}
205206

207+
// The default Strategy CRs and the Velero BSL are Helm-templated behind
208+
// a lookup of the BucketClaim the same chart creates, so a fresh install
209+
// renders them empty — permanently, because helm-controller does not
210+
// re-render an unchanged, successful release. The gate detects that and
211+
// forces the one real Helm upgrade that materialises them. Disabled when
212+
// the HelmRelease coordinates are not plumbed (e.g. a chart-less local run).
213+
if hrName := os.Getenv("BACKUP_DEFAULT_OBJECTS_HELMRELEASE_NAME"); hrName != "" {
214+
gate := &backupcontroller.DefaultObjectsGate{
215+
Client: mgr.GetClient(),
216+
Config: credentialsConfig,
217+
BackupClassName: os.Getenv("BACKUP_DEFAULT_OBJECTS_BACKUPCLASS"),
218+
HelmRelease: types.NamespacedName{
219+
Namespace: os.Getenv("BACKUP_DEFAULT_OBJECTS_HELMRELEASE_NAMESPACE"),
220+
Name: hrName,
221+
},
222+
VeleroNamespace: os.Getenv("BACKUP_DEFAULT_OBJECTS_VELERO_NAMESPACE"),
223+
// The platform bucket's <bucket>-system release renders the
224+
// credentials Secret the projector reads, behind the same kind
225+
// of install-time lookup — and while that Secret is missing
226+
// nothing downstream can resolve. Empty when the bucket is not
227+
// provisioned by Cozystack (external S3), where the Secret is
228+
// admin-managed and no release renders it.
229+
CredentialsHelmRelease: types.NamespacedName{
230+
Namespace: os.Getenv("BACKUP_CREDENTIALS_HELMRELEASE_NAMESPACE"),
231+
Name: os.Getenv("BACKUP_CREDENTIALS_HELMRELEASE_NAME"),
232+
},
233+
}
234+
if err := gate.SetupWithManager(mgr); err != nil {
235+
setupLog.Error(err, "unable to add DefaultObjectsGate runnable")
236+
os.Exit(1)
237+
}
238+
}
239+
206240
if err = (&backupcontroller.BackupJobReconciler{
207241
Client: mgr.GetClient(),
208242
Scheme: mgr.GetScheme(),

‎docs/operations/backup-classes.md‎

Lines changed: 51 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,9 +93,52 @@ On a fresh-cluster install, the Velero `BackupStorageLocation` `cozy-default` is
9393

9494
### Cozy-default Bucket bootstrap
9595

96-
`cozy-default` ships an `apps.cozystack.io/Bucket cozy-backups` CR in `tenant-root`, which the bucket-application chart turns into a `BucketClaim`; the COSI driver then assigns the real S3 bucket name and writes it to the BucketClaim's `.status.bucketName`. The strategy templates and the Velero BSL all read that real bucket name (Helm `lookup` against the BucketClaim). On a fresh install the BucketClaim takes a short reconcile cycle to populate its status — until it does, the strategy templates render empty and only the `Bucket` CR + `BackupClass` are present in the cluster. The HelmRelease re-reconciles on its interval (5 minutes by default — set by the cozystack operator's `helmrelease-interval` flag, not a Flux default), at which point the populated BucketClaim status causes the missing strategy templates to materialise.
96+
`cozy-default` ships an `apps.cozystack.io/Bucket cozy-backups` CR in `tenant-root`, which the bucket-application chart turns into a `BucketClaim`; the COSI driver then assigns the real S3 bucket name (`bucket-<claim-UID>`, so it cannot be computed in advance) and writes it to the BucketClaim's `.status.bucketName`. The strategy templates and the Velero BSL all read that real bucket name (Helm `lookup` against the BucketClaim). On a fresh install the BucketClaim takes a reconcile cycle to populate its status — until it does, the strategy templates render empty and only the `Bucket` CR + `BackupClass` are present in the cluster.
9797

98-
If you need the BackupClass functional immediately (e.g. an e2e), trigger a Flux reconcile (`flux reconcile helmrelease backupstrategy-controller -n cozy-backup-controller`) once you see `kubectl get bucketclaim -n tenant-root bucket-cozy-backups -o jsonpath='{.status.bucketName}'` non-empty.
98+
**That skip does not repair itself on a reconcile.** helm-controller re-renders a release only when its chart or values change; the interval reconcile is a no-op for a healthy release, and drift detection is off on operator-generated HelmReleases. A cluster that lost the race at install time therefore keeps `BackupClass cozy-default` with **no** `Strategy` CRs and **no** Velero BSL indefinitely — which also fail-closes the pre-adoption snapshot in the v1.6.0 etcd migration.
99+
100+
The same trap sits one release earlier, on the Secret everything else depends on. `bucket-cozy-backups-system-credentials` is rendered by the `bucket-cozy-backups-system` release behind a `lookup` of the COSI Secret, and is skipped just as permanently when that lookup is empty. Without it the credentials projector has no source at all, so no strategy, no Velero and no v1.6.0 etcd migration can resolve — and the gate cannot even determine the bucket name, which it reads from that Secret.
101+
102+
Convergence is driven instead by the controller's default-objects gate (`backupStorage.reconcileDefaultObjects`, on by default), which repairs both, in order. While the credentials Secret is absent or carries no bucket name, the gate stamps `reconcile.fluxcd.io/forceAt` + `requestedAt` on the **bucket's** `bucket-<name>-system` HelmRelease. Once the bucket name resolves it checks that every object `cozy-default` routes to exists and stamps the same pair on the **`backupstrategy-controller`** HelmRelease, forcing the real Helm upgrade that re-runs the lookups. The two are throttled independently (one force per 5 minutes each), so the second is not delayed by the first. Expect the objects within a minute or two of the bucket becoming ready.
103+
104+
The gate does not force a suspended HelmRelease — helm-controller ignores both annotations while `spec.suspend` is true, so it logs the skip and leaves the force counter alone. A release left suspended (`cozyhr suspend`) is therefore never repaired until it is resumed.
105+
106+
Watch it with:
107+
108+
```bash
109+
kubectl -n cozy-backup-controller logs deploy/backupstrategy-controller | grep default-objects-gate
110+
```
111+
112+
#### Manual recovery on an affected cluster
113+
114+
Only needed on a cluster running a version without the gate (or with `reconcileDefaultObjects: false`). A plain `flux reconcile helmrelease` does **nothing** here — it does not re-render. You need a forced upgrade, and **both** annotations: `forceAt` is what makes helm-controller run a real Helm upgrade, and it is only honoured together with `requestedAt`.
115+
116+
First confirm the bucket name is actually resolvable — forcing before that just re-runs the same empty lookup:
117+
118+
```bash
119+
kubectl -n tenant-root get bucketclaim bucket-cozy-backups -o jsonpath='{.status.bucketName}'
120+
```
121+
122+
Then force the two releases, **in this order**:
123+
124+
```bash
125+
# 1. The credentials Secret. It is rendered by the -system release, not by
126+
# bucket-cozy-backups — easy to miss, and the projector (hence every
127+
# strategy and Velero) has no source without it.
128+
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
129+
kubectl -n tenant-root annotate helmrelease bucket-cozy-backups-system \
130+
reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite
131+
kubectl -n tenant-root get secret bucket-cozy-backups-system-credentials
132+
133+
# 2. The Strategy CRs and the Velero BSL.
134+
ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
135+
kubectl -n cozy-backup-controller annotate helmrelease backupstrategy-controller \
136+
reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite
137+
kubectl get $(kubectl get crd -o name | grep strategy.backups.cozystack.io) 2>/dev/null
138+
kubectl -n cozy-velero get backupstoragelocation cozy-default
139+
```
140+
141+
The race is nested: step 2's `lookup` resolves from the BucketClaim status, but the projector — and the v1.6.0 etcd migration — read the Secret from step 1, so a cluster missing both needs both.
99142

100143
### Observability
101144

@@ -106,6 +149,12 @@ The credentials projector emits two Prometheus counters labelled by `namespace`
106149

107150
Alert on `rate(cozystack_backup_credentials_projection_failures_total[5m]) > 0` or `absent_over_time(cozystack_backup_credentials_projection_successes_total[10m])` to catch a stale BSL credential or a malformed source Secret without log scraping.
108151

152+
The default-objects gate emits three more:
153+
154+
- `cozystack_backup_default_objects_missing{backupclass="cozy-default"}` — how many of the objects the platform default backups depend on are absent: the credentials Secret, the Strategy CRs `cozy-default` routes to, and the Velero BSL. **This is the alert that would have caught the missing Strategy CRs**: it is non-zero whatever the HelmRelease's `Ready` condition says. Alert on `min_over_time(cozystack_backup_default_objects_missing[15m]) > 0`.
155+
- `cozystack_backup_default_objects_check_errors_total{backupclass="cozy-default"}` — checks that could not reach a conclusion (an API error reading the source Secret, the BackupClass, or one of the routed objects). The gauge above is deliberately **not** written on those ticks, so that it does not flap on a transient API error — which means that while this counter climbs, the gauge is stale and a `0` on it proves nothing. Pair the two: `min_over_time(cozystack_backup_default_objects_missing[15m]) > 0 or rate(cozystack_backup_default_objects_check_errors_total[15m]) > 0`.
156+
- `cozystack_backup_default_objects_force_reconciles_total{namespace,name}` — forced Helm upgrades issued, labelled by the release forced (this chart's own, or the bucket's `-system` release). A counter that keeps climbing means the forced render is not producing the objects (a missing CRD, for instance), which is a different problem from the install-time race. A suspended release is skipped before the patch and is **not** counted here, so a paused release cannot masquerade as a render that keeps failing — look for the `skipped forcing a suspended HelmRelease` log line instead.
157+
109158
## Admin overrides for `cozy-default`
110159

111160
`cozy-default` is rendered by the `backupstrategy-controller` chart and owned by Flux's helm-controller. **Direct `kubectl edit backupclass cozy-default` is overwritten on the next helm reconcile** — the same applies to its companion `strategy.backups.cozystack.io/*` CRs (`cozy-default-cnpg`, `cozy-default-etcd`, `cozy-default-mariadb`, `cozy-default-altinity`, `cozy-default-foundationdb`, the two `cozy-default-velero-*`). The supported override path is the `backupStorage` block on the **`platform` component** of the `cozystack.cozystack-platform` Package CR:

0 commit comments

Comments
 (0)