You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 64097dc
Browse filesBrowse the repository at this point in the historyBrowse files
fix(platform): forward backupStorage overrides to the backupstrategy-controller Package (#3333)
## What this PR does
Fixes#3245.
The documented admin override for the `cozy-default` backup S3
coordinates — `spec.components.backupstrategy-controller.values` on the
`Package cozystack.cozystack-platform`
(`docs/operations/backup-classes.md`, "Admin overrides for
`cozy-default`") — had no effect: the `cozystack-platform` PackageSource
exposes exactly one component (`platform`), so the operator silently
ignored the block, and the platform chart rendered the child `Package
cozystack.backupstrategy-controller` via the `.default` helper with no
components argument, so nothing forwarded `backupStorage` either. Admins
running against an external or TLS S3 had no working supported path to
set `backupStorage.endpoint` / `region` / `systemSecretName`.
This PR wires the override through the platform chart, the same way
`bundles/iaas.yaml` forwards kubevirt and gpu-operator values:
- `packages/core/platform/values.yaml` gains a top-level `backupStorage:
{}` value.
- `templates/bundles/system.yaml` renders it into the emitted `Package
cozystack.backupstrategy-controller` as
`components.backupstrategy-controller.values.backupStorage`, from where
the cozystack operator merges it into the `backupstrategy-controller`
HelmRelease over the chart defaults. When `backupStorage` is unset the
components block is omitted and the emitted Package is byte-identical to
the historical render.
- `docs/operations/backup-classes.md` now documents the working path
(`spec.components.platform.values.backupStorage`) and explicitly calls
out the two plausible-looking paths that do not work (the old documented
one, and patching the child Package directly, which the platform
helm-reconcile reverts).
- A new helm-unittest suite
(`packages/core/platform/tests/bundles_backupstorage_wiring_test.yaml`)
is the behavioural regression test: it proves the unset case emits no
components block and the set case lands the values on the child Package,
on both `isp-full` and `isp-full-generic`.
Verified with `make test` in `packages/core/platform` (22 suites, 91
tests, all green). `make show` requires a live cluster (`cozyhr` +
`lookup` in `templates/repository.yaml`), so the unittest render of
`templates/bundles/system.yaml` is the offline verification path.
### Screenshots
Not a UI change.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#622
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
Walking the trigger map against the diff: the change to
`packages/core/platform/values.yaml` reaches **cozystack/website** twice
— `content/en/docs/next/operations/configuration/platform-package.md`
(hand-written table of `spec.components.platform.values.*` keys gains
`backupStorage`) and
`content/en/docs/next/operations/services/backup-classes.md` (mirrors
the broken override example this PR corrects). The follow-up is
cozystack/website#622, which also carries the superseded website#547
content and covers both next/ and v1.4/ docs, marked not to merge before
this PR lands; its v1.4 copy assumes this fix is backported to
release-1.4. No other downstream repository restates the touched files:
the templates change is chart-internal, no app `values.schema.json`,
CRD, variant list, `hack/` file, or namespace changed.
### Release note
```release-note
fix(platform): the documented admin override for the cozy-default backup S3 coordinates now works — set spec.components.platform.values.backupStorage on the cozystack.cozystack-platform Package and the platform chart forwards it to the backupstrategy-controller component (previously the documented spec.components.backupstrategy-controller path was silently ignored)
```
Copy file name to clipboardExpand all lines: docs/operations/backup-classes.md
+15-12Lines changed: 15 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Backup Classes
2
2
3
-
Cozystack ships a single platform-managed `BackupClass` named `cozy-default`. It is provisioned automatically when the `backupstrategy-controller` package is installed and references the system-managed S3 bucket `cozy-backups` in the `tenant-root` namespace.
3
+
Cozystack ships a single platform-managed `BackupClass` named `cozy-default`. It is provisioned automatically when the `backupstrategy-controller` package is installed and references the system-managed bucket provisioned through the `apps.cozystack.io/Bucket` CR `cozy-backups` in the `tenant-root` namespace (the real S3 bucket name is the COSI-assigned one from `BucketClaim.status.bucketName`).
4
4
5
5
Tenants reference `cozy-default` from `BackupJob`, `Plan`, and `RestoreJob` resources — they do **not** supply S3 credentials, endpoints, or paths. The platform projects the system-managed credentials Secret into the tenant namespace per BackupJob (or, for long-lived references like Velero's `BackupStorageLocation`, into a fixed list of system namespaces on a periodic tick), and the default strategy templates encode `<namespace>/<application>` into every S3 path so two tenants with the same application name never collide.
6
6
@@ -46,7 +46,7 @@ VM-driven (Velero) backups land in the same `cozy-backups` bucket under the `vel
46
46
47
47
The strategy CR `cozy-default-foundationdb` is shipped, but it is **not** bound by `cozy-default` yet. Restore runs `fdbrestore` from inside the `cozy-foundationdb-operator` Deployment, which does not yet mount `cozy-backups-creds`. Until the operator deployment is updated to mount the projected Secret, FDB platform-default restore silently fails — admins who need it today should keep using a per-app `Bucket` plus a custom `BackupClass`, or wire the credentials file into the operator deployment themselves.
48
48
49
-
**Cleanup gotcha (zombie backup_agent).** Unlike CNPG/MariaDB/Altinity (one-shot operator-side Backup CRs), the FoundationDB driver creates a `foundationdb.org/FoundationDBBackup` CR that drives a **long-lived**`backup_agent` Deployment streaming continuously to S3. Deleting a Cozystack `Backup` (e.g. via retention sweeping) does NOT stop that Deployment — the agent keeps writing until the next BackupJob's `stopOtherFoundationDBBackups` call swaps it out, until an admin invokes `examples/backups/foundationdb/cleanup.sh`, or until the operator-side CR is deleted by hand. If a tenant deletes their last Cozystack Backup and never submits another BackupJob, the agent pods will continue running indefinitely and accumulate S3 PUTs. This is intentional today (the driver has no RBAC verb to stop the operator-side CR on Cozystack-Backup deletion) but admins should be aware of it.
49
+
**Cleanup gotcha (zombie backup_agent).** Unlike CNPG/MariaDB/Altinity (one-shot operator-side Backup CRs), the FoundationDB driver creates an `apps.foundationdb.org/FoundationDBBackup` CR that drives a **long-lived**`backup_agent` Deployment streaming continuously to S3. Deleting a Cozystack `Backup` (e.g. via retention sweeping) does NOT stop that Deployment — the agent keeps writing until the next BackupJob's `stopOtherFoundationDBBackups` call swaps it out, until an admin invokes `examples/backups/foundationdb/cleanup.sh`, or until the operator-side CR is deleted by hand. If a tenant deletes their last Cozystack Backup and never submits another BackupJob, the agent pods will continue running indefinitely and accumulate S3 PUTs. This is intentional today (the driver has no RBAC verb to stop the operator-side CR on Cozystack-Backup deletion) but admins should be aware of it.
50
50
51
51
## ClickHouse: opt-in to the system bucket
52
52
@@ -76,7 +76,7 @@ kubectl -n tenant-root get secret bucket-cozy-backups-system-credentials
76
76
kubectl -n cozy-velero get backupstoragelocation cozy-default
77
77
```
78
78
79
-
The bucket lives in `tenant-root` and is provisioned through the `apps.cozystack.io/Bucket` CR. The system-managed credentials Secret never leaves that namespace. The backupstrategy-controller projects a copy under the name `cozy-backups-creds` into a tenant namespace right before each BackupJob runs, and refreshes the same Secret in `cozy-velero` (and any other namespace listed in `backupStorage.systemNamespaces`) on a 1-minute tick. The projected Secret carries multiple key formats so each driver finds what it needs in one place:
79
+
The bucket lives in `tenant-root` and is provisioned through the `apps.cozystack.io/Bucket` CR. The system-managed credentials Secret never leaves that namespace. The backupstrategy-controller projects a copy under the name `cozy-backups-creds` into a tenant namespace right before each BackupJob or RestoreJob runs, and refreshes the same Secret in `cozy-velero` (and any other namespace listed in `backupStorage.systemNamespaces`) on a 1-minute tick. The projected Secret carries multiple key formats so each driver finds what it needs in one place:
@@ -87,15 +87,15 @@ The bucket lives in `tenant-root` and is provisioned through the `apps.cozystack
87
87
88
88
### Bootstrap window
89
89
90
-
On a fresh-cluster install, the Velero `BackupStorageLocation` `cozy-default` is rendered before the credentials projector has had a chance to copy `cozy-backups-creds` into `cozy-velero`. The BSL reports `Unavailable` until the projector's first synchronous round completes (which happens immediately when the `backupstrategy-controller` Pod becomes Ready — typically tens of seconds after `helm install` returns, not minutes). Velero rejects new `Backup` AND `Restore` requests against `storageLocation: cozy-default` during that window. Plan VM backup automation accordingly, or wait for `kubectl -n cozy-velero get bsl cozy-default -o jsonpath='{.status.phase}' = Available` before submitting backups.
90
+
On a fresh-cluster install, the Velero `BackupStorageLocation` `cozy-default` is rendered before the credentials projector has had a chance to copy `cozy-backups-creds` into `cozy-velero`. The BSL reports `Unavailable` until the projector's first synchronous round completes (which runs as soon as the `backupstrategy-controller` acquires leadership — in practice moments after the Pod becomes Ready, typically tens of seconds after `helm install` returns, not minutes). Velero rejects new `Backup` AND `Restore` requests against `storageLocation: cozy-default` during that window. Plan VM backup automation accordingly, or wait for the BSL to become ready before submitting backups: `kubectl -n cozy-velero wait backupstoragelocation cozy-default --for=jsonpath='{.status.phase}'=Available --timeout=5m`.
91
91
92
92
**Note on controller restarts.** The BSL flickers `Unavailable` on every `backupstrategy-controller` pod restart while the projector replays its first synchronous round. The window is short (single-digit seconds) but operators who alert on BSL availability should suppress alerts during the controller's `kube_pod_container_status_restarts_total{container=backupstrategy-controller}` events or use a longer evaluation window than the projector tick (60s).
93
93
94
94
### Cozy-default Bucket bootstrap
95
95
96
-
`cozy-default`ships an `apps.cozystack.io/Bucket cozy-backups` CR in `tenant-root`, which the bucket-application chart turns into a `BucketClaim`; the COSI driver then assigns the real S3 bucket name and writes it to the BucketClaim's `.status.bucketName`. The strategy templates and the Velero BSL all read that real bucket name (Helm `lookup` against the BucketClaim). On a fresh install the BucketClaim takes a short reconcile cycle to populate its status — until it does, the strategy templates render empty and only the `Bucket` CR + `BackupClass` are present in the cluster. Flux re-renders the HelmRelease on its standard interval (default 10 minutes), at which point the populated BucketClaim status causes the missing strategy templates to materialise.
96
+
`cozy-default`ships an `apps.cozystack.io/Bucket cozy-backups` CR in `tenant-root`, which the bucket-application chart turns into a `BucketClaim`; the COSI driver then assigns the real S3 bucket name and writes it to the BucketClaim's `.status.bucketName`. The strategy templates and the Velero BSL all read that real bucket name (Helm `lookup` against the BucketClaim). On a fresh install the BucketClaim takes a short reconcile cycle to populate its status — until it does, the strategy templates render empty and only the `Bucket` CR + `BackupClass` are present in the cluster. The HelmRelease re-reconciles on its interval (5 minutes by default — set by the cozystack operator's `helmrelease-interval` flag, not a Flux default), at which point the populated BucketClaim status causes the missing strategy templates to materialise.
97
97
98
-
If you need the BackupClass functional immediately (e.g. an e2e), trigger a Flux reconcile (`flux reconcile helmrelease backupstrategy-controller`) once you see `kubectl get bucketclaim -n tenant-root bucket-cozy-backups -o jsonpath='{.status.bucketName}'` non-empty.
98
+
If you need the BackupClass functional immediately (e.g. an e2e), trigger a Flux reconcile (`flux reconcile helmrelease backupstrategy-controller -n cozy-backup-controller`) once you see `kubectl get bucketclaim -n tenant-root bucket-cozy-backups -o jsonpath='{.status.bucketName}'` non-empty.
99
99
100
100
### Observability
101
101
@@ -104,11 +104,11 @@ The credentials projector emits two Prometheus counters labelled by `namespace`
Alert on `rate(failures_total) > 0` or `absent_over_time(successes_total[10m])` to catch a stale BSL credential or a malformed source Secret without log scraping.
107
+
Alert on `rate(cozystack_backup_credentials_projection_failures_total[5m]) > 0` or `absent_over_time(cozystack_backup_credentials_projection_successes_total[10m])` to catch a stale BSL credential or a malformed source Secret without log scraping.
108
108
109
109
## Admin overrides for `cozy-default`
110
110
111
-
`cozy-default` is rendered by the `backupstrategy-controller` chart and owned by Flux's helm-controller. **Direct `kubectl edit backupclass cozy-default` is overwritten on the next helm reconcile** — the same applies to its companion `strategy.backups.cozystack.io/*` CRs (`cozy-default-cnpg`, `cozy-default-etcd`, `cozy-default-mariadb`, `cozy-default-altinity`, `cozy-default-foundationdb`, the two `cozy-default-velero-*`). The supported override path is the cozystack `Package` CR, which lets admins inject Helm values into platform components:
111
+
`cozy-default` is rendered by the `backupstrategy-controller` chart and owned by Flux's helm-controller. **Direct `kubectl edit backupclass cozy-default` is overwritten on the next helm reconcile** — the same applies to its companion `strategy.backups.cozystack.io/*` CRs (`cozy-default-cnpg`, `cozy-default-etcd`, `cozy-default-mariadb`, `cozy-default-altinity`, `cozy-default-foundationdb`, the two `cozy-default-velero-*`). The supported override path is the `backupStorage` block on the **`platform` component** of the `cozystack.cozystack-platform` Package CR:
112
112
113
113
```yaml
114
114
apiVersion: cozystack.io/v1alpha1
@@ -117,7 +117,7 @@ metadata:
117
117
name: cozystack.cozystack-platform
118
118
spec:
119
119
components:
120
-
backupstrategy-controller:
120
+
platform:
121
121
values:
122
122
backupStorage:
123
123
provisionBucket: true # default; set false for external S3
@@ -130,10 +130,13 @@ spec:
130
130
- cozy-velero
131
131
```
132
132
133
+
The platform chart forwards this block into the child `Package cozystack.backupstrategy-controller` as `components.backupstrategy-controller.values.backupStorage` (`packages/core/platform/templates/bundles/system.yaml`), from where the cozystack operator merges it into the `backupstrategy-controller` HelmRelease over the chart defaults. Two paths that look plausible do **not** work: `spec.components.backupstrategy-controller`on the `cozystack.cozystack-platform` Package is silently ignored (the only component under that PackageSource is `platform`), and patching the child `Package cozystack.backupstrategy-controller` directly is reverted whenever the platform helm-reconcile re-renders it.
134
+
133
135
| Knob | Effect |
134
136
|---|---|
135
137
| `provisionBucket` | Toggle creation of the in-cluster `apps.cozystack.io/Bucket` CR. Set `false` for external S3 (see [Disabling the platform-managed bucket](#disabling-the-platform-managed-bucket)). |
136
-
| `bucketName` | K8s name of the Bucket CR + lookup key for the COSI BucketClaim. The actual S3 bucket name is the COSI-assigned UUID, surfaced through `BucketClaim.status.bucketName`. |
138
+
| `bucketName` | Two modes. With `provisionBucket: true` (default): K8s name of the Bucket CR + lookup key for the COSI BucketClaim — the actual S3 bucket name is the COSI-assigned UUID, surfaced through `BucketClaim.status.bucketName`. With `provisionBucket: false`: taken **verbatim as the real S3 bucket name** and baked into every strategy CR + the Velero BSL. |
139
+
| `namespace` | Namespace the Bucket CR (and its system-credentials Secret) lives in — `tenant-root` by default. Must be a tenant namespace (`tenant-*`): the Bucket chart's RBAC helper fails the Helm render for any other prefix. |
137
140
| `bucketNameOverride` | Escape hatch for offline `helm template` renders — bypasses the live-cluster BucketClaim lookup. Leave empty in production. |
138
141
| `endpoint` | **Fallback** S3 endpoint. For a provisioned bucket the strategy CRs + Velero BSL derive the endpoint from the COSI system Secret (external ACME ingress, forced `https://`) instead; this value is used only for external S3 (`provisionBucket: false`) and offline renders. For external S3, switching it to `https://` enables TLS in the MariaDB/FoundationDB strategies — ensure the CA bundle is reachable to the relevant operator/driver Pods first. |
139
142
| `region` | Re-projected into `cozy-backups-creds` on the next reconcile. Pod-restart required for chart-emitted clients consuming the region via env (ClickHouse sidecar today). |
@@ -158,7 +161,7 @@ The system-managed credentials Secret is the **only** way for in-cluster strateg
158
161
159
162
## Disabling the platform-managed bucket
160
163
161
-
If a deployment runs against an external S3 (no SeaweedFS), set `backupStorage.provisionBucket: false` in the `backupstrategy-controller` values and create the source credentials Secret in `tenant-root` manually (flat-key format: `accessKey` / `secretKey` / `endpoint` / `bucketName`; or the raw COSI `BucketInfo` JSON). Update `backupStorage.endpoint`, `backupStorage.region`, and (for VM backups) the chart's Velero BSL settings to point at the external S3.
164
+
If a deployment runs against an external S3 (no SeaweedFS), set `backupStorage.provisionBucket: false` via the `platform` component override described above and create the source credentials Secret in `tenant-root` manually (flat-key format: `accessKey` / `secretKey` / `endpoint` / `bucketName`; or the raw COSI `BucketInfo` JSON). In the same `backupStorage` block, update `endpoint`, `region`, **and `bucketName`**: with `provisionBucket: false` the strategies and the Velero BSL take `bucketName` verbatim as the real S3 bucket name (no COSI lookup), so it must name the actual bucket on the external S3 — the `bucketName` key inside the Secret alone is not enough. The Velero `BackupStorageLocation` picks the same values up automatically (the chart renders it from the same `backupStorage` block), so no separate BSL configuration is needed. Note that disabling the cluster-default BSL itself (the chart's `velero.bslEnabled` value) is **not** carried by the `backupStorage` override path — the platform Package forwards only the `backupStorage` block.
162
165
163
166
## Upgrade notes from chart-managed backups
164
167
@@ -170,7 +173,7 @@ If a deployment runs against an external S3 (no SeaweedFS), set `backupStorage.p
170
173
>
171
174
> **Postgres `backup.enabled: true` with placeholder credentials no longer wires backups on upgrade.**
172
175
>
173
-
> The pre-PR defaults for `backup.s3AccessKey` / `backup.s3SecretKey` in `packages/apps/postgres/values.yaml` were the literal `"<your-access-key>"` / `"<your-secret-key>"` placeholders, so the Postgres chart still rendered a backup block on the `cnpg.io/Cluster` (with junk credentials, WAL archiving failing at runtime). After this PR those defaults are empty strings and the chart NO LONGER renders any backup wiring when the placeholders are unmodified. Tenants on the legacy chart-managed flow who relied on those placeholders see `spec.plugins` and the `ObjectStore` disappear from the live `Cluster` on `helm upgrade`. Action — pick one:
176
+
> The pre-v1.5 defaults for `backup.s3AccessKey` / `backup.s3SecretKey` in `packages/apps/postgres/values.yaml` were the literal `"<your-access-key>"` / `"<your-secret-key>"` placeholders, so the Postgres chart still rendered a backup block on the `cnpg.io/Cluster` (with junk credentials, WAL archiving failing at runtime). Starting with v1.5 those defaults are empty strings and the chart NO LONGER renders any backup wiring when the placeholders are unmodified. Tenants on the legacy chart-managed flow who relied on those placeholders see `spec.plugins` and the `ObjectStore` disappear from the live `Cluster` on `helm upgrade`. Action — pick one:
174
177
>
175
178
> - **Move to the platform flow (recommended).** Set `backup.useSystemBucket: true`; the chart leaves `spec.plugins` unset and the CNPG backup driver SSA-applies the `ObjectStore` and patches `spec.plugins` onto the live `Cluster` at first BackupJob time. No tenant-side keys required.
176
179
> - **Stay on the legacy chart-managed flow.** Supply real `backup.s3AccessKey` / `backup.s3SecretKey` (or a pre-existing `backup.s3CredentialsSecret.name`); the chart renders `spec.plugins` + the `ObjectStore` from those coordinates.
0 commit comments