You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1c69a40
Browse filesBrowse the repository at this point in the historyBrowse files
fix(mongodb): apply the storage once per job, and pin the terms that decide ownership
The system-bucket storage was applied on every reconcile of a job, not
once per job: with the operator CR already minted nothing returned before
the apply, so a streaming dump saw an SSA patch every poll for up to a
day, and the only thing keeping that patch off a cluster that had
re-acquired the chart's tasks and pitr since the mint was a predicate
term no test held. The storage a CR names is fixed for its dump, so the
apply now runs only before the CR exists; the legacy-render hold covers
that window, and a test pins that an existing CR is never injected over.
The other predicate that decides whether a Backup owns an archive
compared the live entry's bucket and credentialsSecret against the
strategy, and only the bucket half had a fixture. The credential is what
tells a tenant's entry from the platform's when a bucket name collides,
so a case varying only that term joins the foreign-bucket one. The
cleanup shortcut's snapshot-kind term and the 24h running ceiling are
pinned the same way, the latter as an absolute boundary pair.
The generated field description said the schema rejects useSystemBucket
without backup.enabled. The aggregated apiserver does not evaluate CEL on
application schemas yet, so the render-time refusal is the one that
holds; the description and the runbook now say so. The runbook's restore
paragraph still described the unconditional credential swap this branch
narrowed, exempted MariaDB from a retention warning its shipped strategy
does not earn, and pointed at a Plan retention field that does not
exist; the example README and a dispatcher test's comment still carried
the pre-branch claim that the driver never injects and never deletes.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <[email protected]>
Copy file name to clipboardExpand all lines: docs/operations/backup-classes.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -98,7 +98,7 @@ spec:
98
98
99
99
The chart declares that storage as `s3-storage`, which the `cozy-default-mongodb` strategy names (`storageName: s3-storage`, `type: logical`). The driver then drives on-demand backups and restores through the unified `BackupJob` / `RestoreJob` / `Plan` interface on top of the operator's storage config — adding restore-to-differently-named instances (via `PerconaServerMongoDBRestore.spec.backupSource`) and point-in-time recovery that the raw scheduled-task / `mongodump` paths do not offer. When a target cluster has backups disabled, the driver surfaces a clear `Ready=False` precondition on the BackupJob/RestoreJob rather than hanging until the deadline.
100
100
101
-
For a to-copy restore the operator reads the dump from the **source** backup's bucket/endpoint, but authenticates with the **target** cluster's own S3 credentials (the source release's Secret dies with it in a DR scenario). So the target application's credentials must be able to read the source bucket — the common case where both share the one platform `cozy-backups` bucket. A target whose credentials are scoped to a different S3 account or bucket cannot read the source archive, and the restore fails loudly with `AccessDenied` (RestoreJob `Failed`) rather than silently restoring nothing.
101
+
For a to-copy restore the operator reads the dump from the **source** backup's bucket/endpoint. The driver re-points the restore at the **target** cluster's own S3 credentials only when the target declares a storage on that same bucket (the common case where both share one bucket), since the source release's Secret dies with it in a DR scenario; a system-bucket source keeps its projected `cozy-backups-creds` reference, which outlives the source app. A target whose storages are all on a different bucket gets no credential swap: the restore keeps the source's Secret reference and is held at `Ready=False RestoreCredentialsMissing` naming that Secret until it exists in the restore namespace, so a cross-bucket restore never runs with a credential for the wrong bucket. Re-create the source release's `-s3-creds` Secret by name to let it proceed.
102
102
103
103
A full, scripted example (write a marker document, back up, restore to a copy, assert the round-trip while the source stays untouched) is in [`examples/backups/mongodb/`](../../examples/backups/mongodb/) — driven by `run-all.sh`.
104
104
@@ -116,13 +116,13 @@ When `useSystemBucket: true`:
116
116
117
117
- The chart-emitted `<release>-s3-creds` Secret is no longer rendered, and the chart leaves `spec.backup.storages`, `spec.backup.tasks` and `spec.backup.pitr` unset on the `PerconaServerMongoDB`.
118
118
- On every BackupJob the driver SSA-injects the `s3-storage` entry from the strategy coordinates (bucket/endpoint from the platform system bucket, `credentialsSecret: cozy-backups-creds`, prefix `<namespace>/<application>`) under its own field manager, so a Flux re-render never reverts it and a later change to the coordinates is picked up on the next backup rather than frozen at the first.
119
-
- The injection waits for the release to actually render the flag. The app's `backup.useSystemBucket` is a desired value (it reads `true` as soon as it is written, before helm-controller renders the revision), while the tenant's own storage, scheduled tasks and PITR stream from the previous render stay on the `PerconaServerMongoDB` until it does. As long as the cluster still carries `spec.backup.tasks` or `spec.backup.pitr`, the driver does not inject and holds the BackupJob at `Ready=False` (`PerconaServerMongoDBLegacyRender`), failing it at the backup deadline; a release wedged on an old revision therefore never has its nightly dump and oplog redirected into the shared bucket. The schema also rejects `useSystemBucket: true` together with `enabled: false` (the chart refuses the same pair at render time), so a release cannot wedge on that combination.
119
+
- The injection waits for the release to actually render the flag. The app's `backup.useSystemBucket` is a desired value (it reads `true` as soon as it is written, before helm-controller renders the revision), while the tenant's own storage, scheduled tasks and PITR stream from the previous render stay on the `PerconaServerMongoDB` until it does. As long as the cluster still carries `spec.backup.tasks` or `spec.backup.pitr`, the driver does not inject and holds the BackupJob at `Ready=False` (`PerconaServerMongoDBLegacyRender`), failing it at the backup deadline; a release wedged on an old revision therefore never has its nightly dump and oplog redirected into the shared bucket. `useSystemBucket: true` together with `enabled: false` is refused by the chart at render time; the schema carries the same rule as an `x-kubernetes-validations` marker, which the aggregated apiserver does not evaluate yet ([#2657](https://github.com/cozystack/cozystack/issues/2657)), so until then the render-time refusal is what stops such a release, and a release wedged on it shows up as a failed HelmRelease rather than a redirected backup.
120
120
121
121
`destinationPath`, `endpointURL`, `insecureSkipTLSVerify`, `s3AccessKey` and `s3SecretKey` are ignored in this mode.
122
122
123
123
Because the chart drops `spec.backup.tasks` and `spec.backup.pitr`, the psmdb operator's own scheduled backups and oplog PITR do **not** run on this flow — `backup.schedule` and `backup.retentionPolicy` stay in the schema for the default flow but have no effect here, and the `recoveryTime` PITR restores described below are not available. Migrate scheduled backups to a `backups.cozystack.io/Plan` against `cozy-default` instead. **Flipping this flag on a running app** removes the storage, tasks and pitr the app had, so a release that was taking scheduled backups + PITR before the flip stops until a `Plan` takes over.
124
124
125
-
Dropping `spec.backup.tasks` also drops the psmdb `keep` retention it carried, so on this flow **the driver owns the archive** the way the Rabbitmq/Redis drivers own theirs (an on-demand `PerconaServerMongoDBBackup` is not covered by psmdb task retention). The driver stamps the operator's `percona.com/delete-backup` finalizer onto each backup it mints against a storage that carries the platform coordinates (the strategy's bucket and `credentialsSecret`, as read back from the live cluster after the injection; a strategy without an `s3` block, or a storage pointing anywhere else, holds the job at `Ready=False` instead of minting an unowned dump), and deleting a Cozystack `Backup` (a retention-pruned `Plan`, or a manual delete) deletes that `PerconaServerMongoDBBackup` CR — whose finalizer removes the pbm object from the shared bucket before the `Backup` is released, so a Plan does not accumulate objects in `cozy-backups`. A legacy backup writes to the tenant's own bucket and keeps the pre-existing hands-off contract (its lifecycle stays the tenant's / operator's; nothing is deleted on `Backup` removal). Note this prunes only when a `Backup` is deleted: unlike CNPG and MariaDB, which prune engine-side on their own schedule, MongoDB has no engine-side scheduled trim on this flow, so keeping `cozy-backups` bounded depends on a `Plan`'s retention actually deleting old `Backup` objects. **A `Plan` has no retention field yet** (`PlanSpec` carries only `applicationRef`, `backupClassName` and `schedule`), so today a nightly `useSystemBucket` Plan grows the shared bucket unbounded — `schedule` and `retentionPolicy` on the app are inert here (see [Opt-in to the system bucket](#opt-in-to-the-system-bucket)), and platform-level retention is tracked in the [`BackupRetentionPolicy` design (#4192)](https://github.com/cozystack/cozystack/pull/4192). Teardown GC of orphaned `Backup`s / S3 data is [#3966](https://github.com/cozystack/cozystack/issues/3966). Until those land, bound `cozy-backups` with an S3 lifecycle policy on the object store. Two cases release the `Backup` without waiting for the object to be pruned, so a wedged prune never pins a namespace in `Terminating`: a terminating namespace (where the operator's delete-backup step cannot run) and the `backups.cozystack.io/skip-artifact-cleanup: "true"` annotation. In both the driver strips its own finalizer and leaves the object for a bucket lifecycle policy to reclaim. A down operator or unreachable storage in a live namespace is NOT released automatically — the `Backup` stays `Terminating` as a visible signal until the operator recovers or you set that annotation to force the release.
125
+
Dropping `spec.backup.tasks` also drops the psmdb `keep` retention it carried, so on this flow **the driver owns the archive** the way the Rabbitmq/Redis drivers own theirs (an on-demand `PerconaServerMongoDBBackup` is not covered by psmdb task retention). The driver stamps the operator's `percona.com/delete-backup` finalizer onto each backup it mints against a storage that carries the platform coordinates (the strategy's bucket and `credentialsSecret`, as read back from the live cluster after the injection; a strategy without an `s3` block, or a storage pointing anywhere else, holds the job at `Ready=False` instead of minting an unowned dump), and deleting a Cozystack `Backup` (a retention-pruned `Plan`, or a manual delete) deletes that `PerconaServerMongoDBBackup` CR — whose finalizer removes the pbm object from the shared bucket before the `Backup` is released, so a Plan does not accumulate objects in `cozy-backups`. A legacy backup writes to the tenant's own bucket and keeps the pre-existing hands-off contract (its lifecycle stays the tenant's / operator's; nothing is deleted on `Backup` removal). Note this prunes only when a `Backup` is deleted: unlike CNPG, whose shipped strategy sets a `retentionPolicy` the engine trims on, MongoDB has no engine-side scheduled trim on this flow, so keeping `cozy-backups` bounded depends on a `Plan`'s retention actually deleting old `Backup` objects. **A `Plan` has no retention field yet** (`PlanSpec` carries only `applicationRef`, `backupClassName` and `schedule`), so today a nightly `useSystemBucket` Plan grows the shared bucket unbounded — `schedule` and `retentionPolicy` on the app are inert here (see [Opt-in to the system bucket](#opt-in-to-the-system-bucket)), and platform-level retention is tracked in the [`BackupRetentionPolicy` design (#4192)](https://github.com/cozystack/cozystack/pull/4192). Teardown GC of orphaned `Backup`s / S3 data is [#3966](https://github.com/cozystack/cozystack/issues/3966). Until those land, bound `cozy-backups` with an S3 lifecycle policy on the object store. Two cases release the `Backup` without waiting for the object to be pruned, so a wedged prune never pins a namespace in `Terminating`: a terminating namespace (where the operator's delete-backup step cannot run) and the `backups.cozystack.io/skip-artifact-cleanup: "true"` annotation. In both the driver strips its own finalizer and leaves the object for a bucket lifecycle policy to reclaim. A down operator or unreachable storage in a live namespace is NOT released automatically — the `Backup` stays `Terminating` as a visible signal until the operator recovers or you set that annotation to force the release.
126
126
127
127
A dump that has started streaming on this flow is not cancelled on the 30-minute backup deadline (a real dataset routinely outruns it, and deleting the CR mid-stream would prune a partial), but it is not unbounded either: after 24 hours in `running` the `BackupJob` fails with a `BackupAbandoned` warning naming the `PerconaServerMongoDBBackup`, which is left in place. If that dump later completes, its archive is reachable only through that CR; delete the CR to prune it.
128
128
@@ -300,7 +300,7 @@ The defaults aim at a reasonable middle (30-day retention, gzip compression wher
- **Rabbitmq strategy**: `artifactURITemplate` (the `<namespace>/<application>/<backup-name>/definitions.json` object-key layout) and the pod `template` (image, resources). The backup is a definitions export over the management API — vhosts, users, permissions, queues, exchanges, bindings, policies, parameters — so message payloads are out of scope and there is no point-in-time recovery; a full message-data backup is a Velero volume snapshot instead. **Restore is a merge, not a reset**: importing definitions (`POST /api/definitions`) creates or updates what the export contains and never deletes, so anything created after the backup survives the restore and the broker is not returned to its exact backup-time state. A to-copy restore imports the source's users (with password hashes) into the target, so those source credentials become valid logins there. Unlike the operator-backed strategies (whose engine owns archive retention), the Rabbitmq driver owns its object outright and deletes it from the bucket when its `Backup` is deleted — via a one-shot Job the Backup's removal waits on — so a retention-pruned `Plan` does not normally accumulate objects. The delete is **best-effort**: when it cannot run from here the Backup is released without deleting the object (with a `Warning` Event naming what was left behind), so the Backup — and any namespace being torn down — never wedges. The named give-up conditions are: the namespace is terminating (`CREATE` is forbidden there), the strategy CR or the projected credentials are unavailable, the rendered Job name is invalid, or the skip annotation below is set. Otherwise a genuinely failing delete (object storage unreachable) keeps the `Backup` in `Terminating` (a visible signal to act on) and retries. To release such a stuck `Backup`, annotate it `backups.cozystack.io/skip-artifact-cleanup: "true"` — cleanup then skips the delete and lets the `Backup` go, leaving the object in the bucket to be reclaimed manually or by a bucket lifecycle policy (`kubectl annotate backup <name> -n <namespace> backups.cozystack.io/skip-artifact-cleanup=true`).
- **Etcd strategy**: today the strategy is path-only; combine with `Plan.spec.retentionPolicy` for trim cadence.
303
+
- **Etcd strategy**: today the strategy is path-only, and a `Plan` carries no retention field yet (see [#4192](https://github.com/cozystack/cozystack/pull/4192)), so nothing trims etcd snapshots on the shared bucket automatically either.
304
304
305
305
The system-managed credentials Secret is the **only** way for in-cluster strategies to reach `cozy-backups`. Do not embed access keys in `BackupClass.parameters` — the security model relies on Secret references, and `parameters` end up in `Backup.status.underlyingResources`, which tenants can read.
Copy file name to clipboardExpand all lines: examples/backups/mongodb/README.md
+1-9Lines changed: 1 addition & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -23,15 +23,7 @@ Files are numbered so that `kubectl apply -f` order matches the dependency graph
23
23
24
24
## How MongoDB backups differ from the other drivers
25
25
26
-
The psmdb operator only services on-demand `PerconaServerMongoDBBackup` CRs when
27
-
the cluster has the pbm agents running and a storage declared — which is what
28
-
`backup.enabled: true` on the MongoDB application wires up. Unlike the MariaDB /
29
-
CNPG drivers, the strategy does **not** inject the storage into the cluster; it
30
-
only names it (`storageName: s3-storage`, the storage the chart declares) via
31
-
the platform's `cozy-default-mongodb` Strategy. So the application must opt into
32
-
backups (`backup.enabled: true`) with the bucket coordinates in its `backup.*`
33
-
values. The driver surfaces a clear `Ready=False` precondition (rather than a
34
-
silent hang) when the target cluster has backups disabled.
26
+
The psmdb operator runs the pbm agents when the cluster has `backup.enabled: true`, and services an on-demand `PerconaServerMongoDBBackup` only once the storage the CR names is declared on the cluster. Where that storage comes from depends on the flow the application picked. On the default flow (this example) the chart declares it from the application's own `backup.*` values, and the `cozy-default-mongodb` Strategy only names it (`storageName: s3-storage`); the driver injects nothing. With `backup.useSystemBucket: true` the chart leaves the storage out and the driver injects the platform bucket's coordinates onto the cluster before each backup. Either way the driver surfaces a clear `Ready=False` precondition (rather than a silent hang) when the target cluster has backups disabled; see `docs/operations/backup-classes.md` for both flows.
35
27
36
28
Restores use `PerconaServerMongoDBRestore.spec.backupSource`, carrying the S3
37
29
destination read back from the source backup, so the same artifact restores
0 commit comments