Skip to content

Commit 501bc4f

Browse files
committed
fix(mongodb): key system-bucket injection off the app flag, not live storage
Review of the useSystemBucket flow surfaced three defects rooted in the driver deciding what to do from the live cluster's storage set rather than from the app's opt-in. The snapshot fallback fired for legacy apps too. Its gate was `snap.S3 == nil && rendered.S3 != nil`, but the coordinates moved onto the shared cozy-default strategy, so rendered.S3 is now non-nil for every MongoDB app. A legacy backup written to the tenant's own bucket got its snapshot rewritten to the platform bucket, sending restore to the wrong place. Gate the fallback on the app's backup.useSystemBucket flag instead, threaded through createMongoDBBackupArtifact, and pin the legacy case. The injected storage never caught up with the strategy. Injection was gated on the named storage being absent, so it ran once per cluster and froze the coordinates at the first backup; a later endpoint/region change or bucket re-provision was a silent no-op, and Helm could not repair it because the useSystemBucket chart renders no storages. Key injection off the app flag and re-apply on every BackupJob (the prefix is deterministic, so this never splits the archive the way CNPG's serverName would). Add UseSystemBucket to mongodbapp.MongoDBSpec so a legacy app is protected by the flag, not by whatever happens to be on the cluster. Docs said the opposite of what the driver does. backup-classes.md claimed "the MongoDB driver cannot inject its S3 target per-backup"; correct it and add an opt-in-to-the-system-bucket section that names which values stop being read and that scheduled backups + PITR must move to a Plan. Fix the endpoint table row, the tuning reference, the RBAC comment that said the driver never patches, the strategy header/inline comments, and the example comment. Also assert notExists on spec.backup.pitr in the helm test, and lift the duplicated cozy-backups-creds default into a const. Signed-off-by: Andrey Kolkov <[email protected]> Assisted-By: LLM
1 parent d528a8d commit 501bc4f

8 files changed

Lines changed: 166 additions & 95 deletions

File tree

‎docs/operations/backup-classes.md‎

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ Different operators expect different endpoint shapes; the strategy templates ren
3535
| CNPG (Postgres) | `barmanObjectStore.endpointURL` | full URL (scheme preserved) |
3636
| Etcd | `destination.s3.endpoint` | full URL (scheme preserved) |
3737
| MariaDB | `storage.s3.endpoint` | bare host:port (scheme stripped); `tls.enabled` derived from the scheme |
38-
| MongoDB | n/a — storage lives on the app (`backup.endpointURL`) | full URL (scheme preserved), configured on the MongoDB application, not the strategy |
38+
| MongoDB | `s3.endpointUrl` | full URL (scheme preserved); on the default flow the app's `backup.endpointURL`, on `useSystemBucket` the strategy coordinate injected by the driver |
3939
| FoundationDB | `blobStoreConfiguration.accountName` + `urlParameters.secure_connection` | bare host:port + derived secure flag |
4040
| Velero | `BackupStorageLocation.spec.config.s3Url` | full URL (scheme preserved) |
4141
| ClickHouse sidecar | `S3_ENDPOINT` env | bare host:port (from projected Secret) |
@@ -68,9 +68,13 @@ When `useSystemBucket: true`:
6868

6969
`s3Region`, `s3Bucket`, `endpoint`, `s3AccessKey`, `s3SecretKey`, and `s3CredentialsSecret` are ignored in this mode.
7070

71-
## MongoDB: the application owns the backup storage
71+
## MongoDB: backup storage
7272

73-
Unlike CNPG / MariaDB / Altinity, the MongoDB driver cannot inject its S3 target per-backup: the Percona psmdb operator only runs the percona-backup-mongodb (pbm) agents and services `PerconaServerMongoDBBackup` CRs when the `PerconaServerMongoDB` cluster has `spec.backup.enabled: true` and a storage declared, and a `PerconaServerMongoDBBackup` references that storage only by name. So the MongoDB application must **opt into backups** and point at the bucket in its own chart values:
73+
MongoDB supports two storage flows. By default the S3 target lives on the app CR and the tenant supplies its own credentials; alternatively the tenant opts into the platform system bucket and the driver injects the storage onto the live cluster at backup time.
74+
75+
### Default: the application owns the backup storage
76+
77+
The Percona psmdb operator only runs the percona-backup-mongodb (pbm) agents and services `PerconaServerMongoDBBackup` CRs when the `PerconaServerMongoDB` cluster has `spec.backup.enabled: true` and a storage declared, and a `PerconaServerMongoDBBackup` references that storage only by name. On the default flow the MongoDB application **opts into backups** and points at the bucket in its own chart values:
7478

7579
```yaml
7680
apiVersion: apps.cozystack.io/v1alpha1
@@ -93,6 +97,25 @@ For a to-copy restore the operator reads the dump from the **source** backup's b
9397

9498
A full, scripted example (write a marker document, back up, restore to a copy, assert the round-trip while the source stays untouched) is in [`examples/backups/mongodb/`](../../examples/backups/mongodb/) — driven by `run-all.sh`.
9599

100+
### Opt-in to the system bucket
101+
102+
Unlike ClickHouse's sidecar, PBM takes the bucket/endpoint/prefix as static fields on the `PerconaServerMongoDB` CR, and the platform bucket name is only known at BackupJob time — so the chart cannot render them. Instead the `cozy-default-mongodb` driver injects the storage onto the live cluster. To back up to the platform bucket without supplying S3 credentials, set:
103+
104+
```yaml
105+
backup:
106+
enabled: true
107+
useSystemBucket: true
108+
```
109+
110+
When `useSystemBucket: true`:
111+
112+
- The chart-emitted `<release>-s3-creds` Secret is no longer rendered, and the chart leaves `spec.backup.storages`, `spec.backup.tasks` and `spec.backup.pitr` unset on the `PerconaServerMongoDB`.
113+
- On every BackupJob the driver SSA-injects the `s3-storage` entry from the strategy coordinates (bucket/endpoint from the platform system bucket, `credentialsSecret: cozy-backups-creds`, prefix `<namespace>/<release>`) under its own field manager, so a Flux re-render never reverts it and a later change to the coordinates is picked up on the next backup rather than frozen at the first.
114+
115+
`destinationPath`, `endpointURL`, `insecureSkipTLSVerify`, `s3AccessKey` and `s3SecretKey` are ignored in this mode.
116+
117+
Because the chart drops `spec.backup.tasks` and `spec.backup.pitr`, the psmdb operator's own scheduled backups and oplog PITR do **not** run on this flow — `backup.schedule` and `backup.retentionPolicy` stay in the schema for the default flow but have no effect here, and the `recoveryTime` PITR restores described below are not available. Migrate scheduled backups to a `backups.cozystack.io/Plan` against `cozy-default` instead. **Flipping this flag on a running app** removes the storage, tasks and pitr the app had, so a release that was taking scheduled backups + PITR before the flip stops until a `Plan` takes over.
118+
96119
### Point-in-time recovery (MongoDB)
97120

98121
psmdb records an oplog stream between logical backups. A MongoDB `RestoreJob` recovers to a timestamp via `spec.options.recoveryTime` — the same option name and RFC3339 format the Postgres/CNPG driver uses, so the two are uniform:
@@ -240,7 +263,7 @@ The defaults aim at a reasonable middle (30-day retention, gzip compression wher
240263

241264
- **CNPG strategy**: `barmanObjectStore.retentionPolicy`, `data.compression`, `wal.compression`.
242265
- **MariaDB strategy**: `compression`, `maxRetention`, `databases[]`.
243-
- **MongoDB strategy**: `storageName` (which `spec.backup.storages` entry on the psmdb cluster to use), `type` (`logical`), `compressionType` / `compressionLevel`. The S3 target itself is tuned via `backup.*` values on the MongoDB release (see [MongoDB: the application owns the backup storage](#mongodb-the-application-owns-the-backup-storage)).
266+
- **MongoDB strategy**: `storageName` (which `spec.backup.storages` entry on the psmdb cluster to use), `type` (`logical`), `compressionType` / `compressionLevel`, and — for the `useSystemBucket` flow — the `s3` coordinates the driver injects. On the default flow the S3 target is tuned via `backup.*` values on the MongoDB release instead (see [MongoDB: backup storage](#mongodb-backup-storage)).
244267
- **Altinity strategy**: tune the `clickhouse-backup` sidecar via `backup.*` values on the ClickHouse release; the strategy Pod is a thin HTTP client. When the S3 endpoint's certificate is signed by a private CA rather than a publicly-trusted one — SeaweedFS's in-cluster `:8333` being the case in point — point `backup.endpointCA` at a Secret holding that CA bundle; the chart mounts it into the sidecar and adds it to the trust store via `SSL_CERT_DIR`, which supplements the system CA set rather than replacing it.
245268
- **FoundationDB strategy**: `snapshotPeriodSeconds`, `agentCount`, `urlParameters[]`.
246269
- **Velero strategy (VMInstance / VMDisk)**: `ttl`, `includedResources[]`, `excludedResources[]`.

‎examples/backups/mongodb/05-mongodb-src.yaml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,11 @@
44
# when the cluster has the percona-backup-mongodb agents running and a storage
55
# declared — that is what `backup.enabled: true` wires up. The cozystack
66
# backup-strategy driver names that storage ("s3-storage", the one the chart
7-
# declares) via the cozy-default-mongodb Strategy; it does NOT inject storage
8-
# into the cluster, so unlike MariaDB the application must opt into backups.
7+
# declares) via the cozy-default-mongodb Strategy. This example uses the default
8+
# flow, where the application owns the storage and the driver does not inject it
9+
# — so, unlike MariaDB, the application must opt into backups. (Set
10+
# backup.useSystemBucket: true instead to have the driver inject the platform
11+
# system-bucket storage; see docs/operations/backup-classes.md.)
912
#
1013
# The bucket coordinates (destinationPath / endpointURL / s3AccessKey /
1114
# s3SecretKey) point the storage at the Bucket provisioned in 00-bucket.yaml.

‎internal/backupcontroller/mongodbapp/types.go‎

Lines changed: 17 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -54,13 +54,20 @@ type MongoDBList struct {
5454
Items []MongoDB `json:"items"`
5555
}
5656

57-
// MongoDBSpec is intentionally empty: the driver does not currently read or
58-
// patch any apps.cozystack.io/MongoDB spec field. It reads the downstream
59-
// psmdb.percona.com/PerconaServerMongoDB CR (rendered by the chart's
60-
// HelmRelease) for existence + backup-enabled gating, and drives
61-
// PerconaServerMongoDBBackup / PerconaServerMongoDBRestore CRs; it never
62-
// mutates the app CR.
63-
//
64-
// Reserved for future fields the driver might need to read. Add fields here
65-
// only when the driver genuinely needs them.
66-
type MongoDBSpec struct{}
57+
// MongoDBSpec mirrors only the apps.cozystack.io/MongoDB spec fields the driver
58+
// reads. It drives the downstream psmdb.percona.com CRs directly, so it keeps
59+
// no other spec state here.
60+
type MongoDBSpec struct {
61+
Backup MongoDBBackupSpec `json:"backup,omitempty"`
62+
}
63+
64+
// MongoDBBackupSpec carries the backup fields the driver reads off the app CR.
65+
// UseSystemBucket is the tenant's opt-in to the platform system bucket: on that
66+
// flow the chart leaves spec.backup.storages unset and the driver injects the
67+
// storage from the strategy coordinates at BackupJob time. The driver keys its
68+
// injection and its snapshot fallback off this flag, not off whatever storage
69+
// happens to be on the live cluster, so a legacy app that ships its own static
70+
// storage (UseSystemBucket=false) is never touched.
71+
type MongoDBBackupSpec struct {
72+
UseSystemBucket bool `json:"useSystemBucket,omitempty"`
73+
}

‎internal/backupcontroller/psmdbstrategy_controller.go‎

Lines changed: 38 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,12 @@ const (
4848
// storage; when it leaves StorageName empty the driver falls back to this.
4949
psmdbDefaultStorageName = "s3-storage"
5050

51+
// psmdbDefaultCredentialsSecret is the Secret the useSystemBucket flow
52+
// authenticates against: the BackupJob reconciler projects it into the
53+
// namespace before dispatch. Both the storage injection and the snapshot
54+
// fallback default to it when the strategy names none.
55+
psmdbDefaultCredentialsSecret = "cozy-backups-creds"
56+
5157
// psmdbFieldManager is the server-side-apply field owner for the driver's
5258
// storage injection on the useSystemBucket flow. A dedicated manager (with
5359
// ForceOwnership) keeps the injected spec.backup.storages entry from being
@@ -199,6 +205,8 @@ func (r *BackupJobReconciler) reconcileMongoDB(ctx context.Context, j *backupsv1
199205
return ctrl.Result{}, err
200206
}
201207

208+
useSystemBucket := app.Spec.Backup.UseSystemBucket
209+
202210
rendered, err := renderMongoDBTemplate(strategy.Spec.Template, app, resolved.Parameters)
203211
if err != nil {
204212
return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to template MongoDB strategy: %v", err))
@@ -231,10 +239,15 @@ func (r *BackupJobReconciler) reconcileMongoDB(ctx context.Context, j *backupsv1
231239
// the storage from the strategy's coordinates before the precondition looks
232240
// for it. Owned by a dedicated field manager with ForceOwnership so a Flux
233241
// re-render of the app never reverts it (mirrors the CNPG driver's
234-
// spec.plugins patch). Gated on the storage being ABSENT: rendered.S3==nil
235-
// (a custom strategy) or a legacy app that already ships its own static
236-
// s3-storage is left untouched, so this never clobbers a manual bucket.
237-
if shouldInjectMongoDBSystemStorage(cluster, storageName, rendered) {
242+
// spec.plugins patch). Keyed off the app's useSystemBucket flag, not off
243+
// whatever storage is on the live cluster: when the flag is set the apply
244+
// runs on every BackupJob so a later change to the strategy coordinates
245+
// (endpoint, region, bucket re-provision) catches up rather than being
246+
// frozen at the first backup; when it is unset the driver never touches the
247+
// cluster, so a legacy app that ships its own static storage is left alone.
248+
// The path prefix is deterministic (<ns>/<release>), so re-applying the
249+
// whole entry never splits the archive the way CNPG's serverName would.
250+
if shouldInjectMongoDBSystemStorage(useSystemBucket, rendered) {
238251
if err := r.applyMongoDBSystemStorage(ctx, j.Namespace, psmdbName, storageName, rendered.S3); err != nil {
239252
return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to inject system-bucket storage onto psmdb.percona.com/PerconaServerMongoDB %s/%s: %v", j.Namespace, psmdbName, err))
240253
}
@@ -271,7 +284,7 @@ func (r *BackupJobReconciler) reconcileMongoDB(ctx context.Context, j *backupsv1
271284
if j.Status.BackupRef != nil {
272285
return ctrl.Result{}, nil
273286
}
274-
artifact, err := r.createMongoDBBackupArtifact(ctx, j, resolved, mdbBackup, rendered, storageName)
287+
artifact, err := r.createMongoDBBackupArtifact(ctx, j, resolved, mdbBackup, rendered, storageName, useSystemBucket)
275288
if err != nil {
276289
return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to create Backup artifact: %v", err))
277290
}
@@ -396,18 +409,15 @@ func psmdbStorageS3CredentialsSecret(raw runtime.RawExtension) string {
396409
}
397410

398411
// shouldInjectMongoDBSystemStorage reports whether the driver must SSA-inject
399-
// the system-bucket storage onto the live cluster: only when the strategy
400-
// carries S3 coordinates (the useSystemBucket flow) AND the cluster does not
401-
// already declare the named storage. The absence gate is the load-bearing
402-
// guarantee that a legacy app shipping its own static storage — or a custom
403-
// strategy with no S3 coordinates — is left untouched, so injection never
404-
// clobbers a manually configured bucket.
405-
func shouldInjectMongoDBSystemStorage(cluster *psmdbtypes.PerconaServerMongoDB, storageName string, rendered *strategyv1alpha1.MongoDBTemplate) bool {
406-
if rendered.S3 == nil {
407-
return false
408-
}
409-
_, hasStorage := cluster.Spec.Backup.Storages[storageName]
410-
return !hasStorage
412+
// the system-bucket storage onto the live cluster: only when the app opted in
413+
// via backup.useSystemBucket AND the strategy actually carries S3 coordinates
414+
// to inject. Keying off the app flag (not the live cluster's storage set) is
415+
// the load-bearing guarantee: a legacy app leaves the flag false and is never
416+
// touched, so injection never clobbers a manually configured bucket; a
417+
// useSystemBucket app is injected on every BackupJob, so a later change to the
418+
// strategy coordinates catches up instead of being frozen at the first backup.
419+
func shouldInjectMongoDBSystemStorage(useSystemBucket bool, rendered *strategyv1alpha1.MongoDBTemplate) bool {
420+
return useSystemBucket && rendered.S3 != nil
411421
}
412422

413423
// buildMongoDBSystemStorageEntry constructs the psmdb spec.backup.storages
@@ -419,7 +429,7 @@ func shouldInjectMongoDBSystemStorage(cluster *psmdbtypes.PerconaServerMongoDB,
419429
func buildMongoDBSystemStorageEntry(s3 *strategyv1alpha1.MongoDBStorageS3) map[string]interface{} {
420430
cred := s3.CredentialsSecret
421431
if cred == "" {
422-
cred = "cozy-backups-creds"
432+
cred = psmdbDefaultCredentialsSecret
423433
}
424434
s3Cfg := map[string]interface{}{
425435
"bucket": s3.Bucket,
@@ -551,6 +561,7 @@ func (r *BackupJobReconciler) createMongoDBBackupArtifact(
551561
mdbBackup *psmdbtypes.PerconaServerMongoDBBackup,
552562
rendered *strategyv1alpha1.MongoDBTemplate,
553563
storageName string,
564+
useSystemBucket bool,
554565
) (*backupsv1alpha1.Backup, error) {
555566
takenAt := metav1.Now()
556567
if mdbBackup.Status.Completed != nil && !mdbBackup.Status.Completed.IsZero() {
@@ -565,7 +576,7 @@ func (r *BackupJobReconciler) createMongoDBBackupArtifact(
565576
driverMD[psmdbDestinationKey] = mdbBackup.Status.Destination
566577
}
567578

568-
underlyingResources, err := marshalMongoDBBackupSnapshot(mdbBackup, rendered, storageName, resolved.Parameters)
579+
underlyingResources, err := marshalMongoDBBackupSnapshot(mdbBackup, rendered, storageName, resolved.Parameters, useSystemBucket)
569580
if err != nil {
570581
return nil, fmt.Errorf("encode source snapshot for Backup.status.underlyingResources: %w", err)
571582
}
@@ -984,7 +995,7 @@ type mongodbBackupSnapshot struct {
984995
Parameters map[string]string `json:"parameters,omitempty"`
985996
}
986997

987-
func marshalMongoDBBackupSnapshot(mdbBackup *psmdbtypes.PerconaServerMongoDBBackup, rendered *strategyv1alpha1.MongoDBTemplate, storageName string, parameters map[string]string) (*runtime.RawExtension, error) {
998+
func marshalMongoDBBackupSnapshot(mdbBackup *psmdbtypes.PerconaServerMongoDBBackup, rendered *strategyv1alpha1.MongoDBTemplate, storageName string, parameters map[string]string, useSystemBucket bool) (*runtime.RawExtension, error) {
988999
snap := mongodbBackupSnapshot{
9891000
Kind: psmdbBackupSnapshotKind,
9901001
APIVersion: psmdbBackupSnapshotAPIVersion,
@@ -997,11 +1008,15 @@ func marshalMongoDBBackupSnapshot(mdbBackup *psmdbtypes.PerconaServerMongoDBBack
9971008
// On the useSystemBucket flow the driver injected the storage, so fall back
9981009
// to those coordinates when the operator's status echo is empty - the
9991010
// restore path rebuilds backupSource from this snapshot and needs the
1000-
// endpoint + credentialsSecret (cozy-backups-creds) to be present.
1001-
if snap.S3 == nil && rendered.S3 != nil {
1011+
// endpoint + credentialsSecret (cozy-backups-creds) to be present. Gated on
1012+
// the app's flag, not on rendered.S3 alone: the coordinates now live on the
1013+
// shared cozy-default strategy, so rendered.S3 is non-nil for a legacy app
1014+
// too, and recording the platform bucket for an archive written to the
1015+
// tenant's own bucket would send restore to the wrong place.
1016+
if useSystemBucket && snap.S3 == nil && rendered.S3 != nil {
10021017
cred := rendered.S3.CredentialsSecret
10031018
if cred == "" {
1004-
cred = "cozy-backups-creds"
1019+
cred = psmdbDefaultCredentialsSecret
10051020
}
10061021
snap.S3 = &psmdbtypes.BackupStorageS3{
10071022
Bucket: rendered.S3.Bucket,

0 commit comments

Comments
 (0)