Skip to content

Commit b9efc6a

Browse files
committed
fix(backups): harden MongoDB restore auth + PITR option handling from review
Address the two blocking findings in PR #3562: - Restore into a differently-named target now authenticates with the TARGET cluster's own S3 credentials instead of the source's. The backupSource carried the source release's -s3-creds Secret, which is deleted together with the source app — the disaster-recovery case backups exist for — leaving a to-copy restore unable to reach S3. The restore replays into the target (gated on backup.enabled just above), whose own -s3-creds Secret is present and grants access to the same bucket; only the credential reference is swapped, bucket/prefix/endpoint still come from the source archive. Falls back to the source reference when the target declares no discoverable s3 credentials (in-place restore resolves to the same Secret, so it is a no-op there). - A non-string recoveryTime (spec.options is free-form runtime.RawExtension, so {"recoveryTime": 20260805} passes admission) no longer silently degrades a PITR request to a plain snapshot restore reported Succeeded. When the typed decode fails and the blob carries a recoveryTime key, the RestoreJob now fails terminally — the same guarantee pitrSpec already makes for a malformed RFC3339 string. Minor findings also addressed: - BackupJob tolerates a not-yet-present MongoDB app CR with the same bounded grace the psmdb-CR existence gate uses, instead of failing terminally on a GitOps single-commit ordering race. - examples: the to-copy target writes into its OWN S3 prefix (mongodb-target/) rather than the source's, so the two clusters' oplog and backup metadata no longer interleave under shared keys (both run replset rs0). - examples: wait_psmdb_ready no longer treats a transient state=error as terminal — the cluster state is recomputed each reconcile and can blip through error during replset init before converging to ready. Unit tests cover the target-credential selector (preferred/default/sole storage) and the load-bearing recoveryTime-key detection. Signed-off-by: Andrey Kolkov <[email protected]>
1 parent 6f25225 commit b9efc6a

4 files changed

Lines changed: 223 additions & 8 deletions

File tree

‎examples/backups/mongodb/00-helpers.sh‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -124,11 +124,15 @@ wait_hr_ready() {
124124

125125
# Wait until the psmdb.percona.com/PerconaServerMongoDB CR reports state=ready
126126
# (the operator's own readiness signal — the rs0 members are up and initialised
127-
# and, when backup.enabled, the pbm agents are running).
127+
# and, when backup.enabled, the pbm agents are running). No fail-fast value: the
128+
# cluster state is recomputed every operator reconcile and can pass through
129+
# "error" transiently during replset init before it converges to "ready", so a
130+
# terminal-on-error gate would fail the whole run on one blip. Poll for the
131+
# positive value only, bounded by the timeout.
128132
wait_psmdb_ready() {
129133
local cr="$1" timeout="${2:-600}"
130134
wait_for_field perconaservermongodbs.psmdb.percona.com "$cr" \
131-
'{.status.state}' ready "$NAMESPACE" "$timeout" error
135+
'{.status.state}' ready "$NAMESPACE" "$timeout"
132136
}
133137

134138
# Name of the first data-bearing pod of a PerconaServerMongoDB CR's rs0 replica

‎examples/backups/mongodb/20-mongodb-target.yaml‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,12 @@
11
# Empty MongoDB target for the to-copy restore. The psmdb operator replays the
22
# logical dump into this running cluster, so it needs the percona-backup-mongodb
3-
# agents too — hence backup.enabled: true with the SAME bucket coordinates as
4-
# the source (the restore reads the backup from that bucket via backupSource).
5-
# No "app" user or database is declared: the restore recreates the demo database
6-
# from the dump.
3+
# agents too — hence backup.enabled: true. It uses the SAME bucket as the source
4+
# but its OWN prefix (mongodb-target/): the restore reads the source archive via
5+
# backupSource, so the target never needs to write into the source's prefix, and
6+
# a shared prefix would interleave the two clusters' oplog/backup metadata (both
7+
# run replset rs0) — a source PITR could then resolve oplog chunks the target
8+
# wrote. No "app" user or database is declared: the restore recreates the demo
9+
# database from the dump.
710
---
811
apiVersion: apps.cozystack.io/v1alpha1
912
kind: MongoDB
@@ -14,7 +17,7 @@ spec:
1417
size: 1Gi
1518
backup:
1619
enabled: true
17-
destinationPath: "s3://REPLACE_WITH_COSI_BUCKET_NAME/mongodb-src/"
20+
destinationPath: "s3://REPLACE_WITH_COSI_BUCKET_NAME/mongodb-target/"
1821
endpointURL: "REPLACE_WITH_S3_ENDPOINT"
1922
insecureSkipTLSVerify: true
2023
s3AccessKey: "REPLACE_WITH_S3_ACCESS_KEY"

‎internal/backupcontroller/psmdbstrategy_controller.go‎

Lines changed: 114 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -175,7 +175,19 @@ func (r *BackupJobReconciler) reconcileMongoDB(ctx context.Context, j *backupsv1
175175
app, err := r.getMongoDBApp(ctx, j.Namespace, j.Spec.ApplicationRef.Name)
176176
if err != nil {
177177
if apierrors.IsNotFound(err) {
178-
return r.markBackupJobFailed(ctx, j, fmt.Sprintf("MongoDB application not found: %s/%s", j.Namespace, j.Spec.ApplicationRef.Name))
178+
// A GitOps apply that lands the MongoDB app and the BackupJob in one
179+
// commit can reconcile the BackupJob before the app CR is visible in
180+
// the cache. Tolerate a not-yet-present app with the same bounded
181+
// grace the psmdb-CR existence gate below uses, instead of failing
182+
// terminally on a transient ordering race. StartedAt was persisted
183+
// above, so the deadline clock is already running.
184+
if psmdbBackupDeadlineExceeded(j.Status.StartedAt) {
185+
return r.markBackupJobFailed(ctx, j, fmt.Sprintf(
186+
"MongoDB application %s/%s not found within %s",
187+
j.Namespace, j.Spec.ApplicationRef.Name, psmdbDefaultBackupDeadline))
188+
}
189+
return r.requeueMongoDBBackupWaiting(ctx, j, "MongoDBApplicationNotReady",
190+
fmt.Sprintf("waiting for MongoDB application %s/%s to exist", j.Namespace, j.Spec.ApplicationRef.Name))
179191
}
180192
return ctrl.Result{}, err
181193
}
@@ -312,6 +324,54 @@ func psmdbBackupPrecondition(cluster *psmdbtypes.PerconaServerMongoDB, storageNa
312324
return ""
313325
}
314326

327+
// psmdbTargetCredentialsSecret returns the s3 credentialsSecret name the target
328+
// cluster declares for its backup storage, so a restore authenticates with the
329+
// TARGET's own credentials rather than the (possibly deleted) source's. It
330+
// prefers the storage whose name the source backup used, then the chart's
331+
// default storage, then — only when the cluster declares exactly one storage —
332+
// that sole storage. Returns "" when nothing is resolvable, in which case the
333+
// caller keeps the source reference. spec.backup.storages is decoded on demand
334+
// from runtime.RawExtension (the partial type keeps only the key set for the
335+
// precondition gate), so no upstream storage shape is mirrored here.
336+
func psmdbTargetCredentialsSecret(cluster *psmdbtypes.PerconaServerMongoDB, preferredStorage string) string {
337+
storages := cluster.Spec.Backup.Storages
338+
if len(storages) == 0 {
339+
return ""
340+
}
341+
if preferredStorage != "" {
342+
if cred := psmdbStorageS3CredentialsSecret(storages[preferredStorage]); cred != "" {
343+
return cred
344+
}
345+
}
346+
if cred := psmdbStorageS3CredentialsSecret(storages[psmdbDefaultStorageName]); cred != "" {
347+
return cred
348+
}
349+
if len(storages) == 1 {
350+
for _, raw := range storages {
351+
return psmdbStorageS3CredentialsSecret(raw)
352+
}
353+
}
354+
return ""
355+
}
356+
357+
// psmdbStorageS3CredentialsSecret extracts .s3.credentialsSecret from one
358+
// psmdb spec.backup.storages entry. Returns "" for an empty/non-object entry,
359+
// a non-s3 storage, or an entry that names no secret.
360+
func psmdbStorageS3CredentialsSecret(raw runtime.RawExtension) string {
361+
if len(raw.Raw) == 0 {
362+
return ""
363+
}
364+
var st struct {
365+
S3 struct {
366+
CredentialsSecret string `json:"credentialsSecret"`
367+
} `json:"s3"`
368+
}
369+
if err := json.Unmarshal(raw.Raw, &st); err != nil {
370+
return ""
371+
}
372+
return st.S3.CredentialsSecret
373+
}
374+
315375
// psmdbBackupDeadlineExceeded reports whether enough wall-clock time elapsed
316376
// since the BackupJob started that we should give up on a stuck operator
317377
// backup. Returns false when StartedAt is nil so the first reconcile does not
@@ -518,6 +578,17 @@ func (r *RestoreJobReconciler) reconcileMongoDBRestore(ctx context.Context, rest
518578
// than failing — so a bad recoveryTime fails terminally below.
519579
options, unknownKeys, err := parseMongoDBRestoreOptions(restoreJob.Spec.Options)
520580
if err != nil {
581+
// Falling back to defaults is safe for most malformed options, but not
582+
// when the blob carries a recoveryTime key: silently degrading a PITR
583+
// request to a plain snapshot restore is exactly the wrong-point restore
584+
// pitrSpec fails terminally to prevent. spec.options is free-form
585+
// runtime.RawExtension, so a non-string recoveryTime (e.g. {"recoveryTime":
586+
// 20260805}) passes admission yet fails the typed decode here — fail
587+
// terminally rather than restore to the snapshot and report Succeeded.
588+
if restoreOptionsCarryRecoveryTimeKey(restoreJob.Spec.Options) {
589+
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf(
590+
"restoreJob.spec.options carries a recoveryTime but is malformed (%v); refusing to fall back to a plain snapshot restore, which would silently ignore the requested point-in-time", err))
591+
}
521592
logger.Info("malformed restoreJob.spec.options; falling back to defaults", "error", err)
522593
r.Recorder.Eventf(restoreJob, corev1.EventTypeWarning, "MalformedOptions",
523594
"spec.options is not valid JSON; falling back to defaults: %v", err)
@@ -575,6 +646,27 @@ func (r *RestoreJobReconciler) reconcileMongoDBRestore(ctx context.Context, rest
575646
fmt.Sprintf("waiting for target psmdb.percona.com/PerconaServerMongoDB %s/%s to enable backups", target.Namespace, targetPSMDBName))
576647
}
577648

649+
// Re-point the restore at the TARGET cluster's own S3 credentials. The
650+
// backupSource carries the SOURCE cluster's credentialsSecret (read back from
651+
// the source backup status / snapshot), which names the source release's
652+
// -s3-creds Secret. That Secret is deleted together with the source app —
653+
// exactly the disaster-recovery case backups exist for — so a restore into a
654+
// fresh, differently-named target must not depend on it. The restore replays
655+
// into the target, whose own -s3-creds Secret is guaranteed present
656+
// (backup.enabled was gated just above) and grants access to the backup's
657+
// bucket. The backupSource's bucket/prefix/endpoint still come from the
658+
// source (that is where the archive lives); only the credential reference is
659+
// swapped. Falls back to the source reference when the target declares no
660+
// discoverable s3 credentials (in-place restore into the still-present source
661+
// resolves to the same Secret, so this is a no-op there).
662+
if source.S3 != nil {
663+
if cred := psmdbTargetCredentialsSecret(targetCluster, source.StorageName); cred != "" && cred != source.S3.CredentialsSecret {
664+
logger.Debug("re-pointing MongoDB restore credentials at target cluster storage",
665+
"restorejob", restoreJob.Name, "from", source.S3.CredentialsSecret, "to", cred)
666+
source.S3.CredentialsSecret = cred
667+
}
668+
}
669+
578670
mdbRestore, err := r.ensureMongoDBRestore(ctx, restoreJob, targetPSMDBName, source, pitr)
579671
if err != nil {
580672
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("failed to ensure psmdb.percona.com/PerconaServerMongoDBRestore: %v", err))
@@ -649,6 +741,7 @@ func (r *RestoreJobReconciler) resolveMongoDBBackupSource(ctx context.Context, b
649741
return &psmdbtypes.BackupSource{
650742
Type: psmdbBackupTypeOrDefault(live.Status.Type),
651743
Destination: live.Status.Destination,
744+
StorageName: live.Status.StorageName,
652745
S3: live.Status.S3.DeepCopy(),
653746
}, nil
654747
}
@@ -678,6 +771,7 @@ func (r *RestoreJobReconciler) resolveMongoDBBackupSource(ctx context.Context, b
678771
if snap.Type != "" {
679772
src.Type = snap.Type
680773
}
774+
src.StorageName = snap.StorageName
681775
src.S3 = snap.S3.DeepCopy()
682776
}
683777
return src, nil
@@ -911,6 +1005,25 @@ func parseMongoDBRestoreOptions(opts *runtime.RawExtension) (MongoDBRestoreOptio
9111005
return out, unknown, nil
9121006
}
9131007

1008+
// restoreOptionsCarryRecoveryTimeKey reports whether spec.options parses as a
1009+
// JSON object with a "recoveryTime" key present, regardless of that value's
1010+
// type. The caller uses it to decide whether a typed-decode failure is
1011+
// load-bearing (a malformed recoveryTime must fail terminally) or benign (a
1012+
// malformed unrelated field can fall back to defaults). A blob that is not a
1013+
// JSON object at all returns false — there is no recoveryTime to honour, so the
1014+
// permissive fall-back applies.
1015+
func restoreOptionsCarryRecoveryTimeKey(opts *runtime.RawExtension) bool {
1016+
if opts == nil || len(opts.Raw) == 0 {
1017+
return false
1018+
}
1019+
var obj map[string]json.RawMessage
1020+
if err := json.Unmarshal(opts.Raw, &obj); err != nil {
1021+
return false
1022+
}
1023+
_, ok := obj["recoveryTime"]
1024+
return ok
1025+
}
1026+
9141027
// unknownJSONKeys returns the top-level object keys in raw that are not in
9151028
// known. Returns nil when raw is not a JSON object (a malformed blob is handled
9161029
// by the typed decode's error path, not here).

‎internal/backupcontroller/psmdbstrategy_controller_test.go‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ package backupcontroller
33

44
import (
55
"context"
6+
"fmt"
67
"testing"
78

89
corev1 "k8s.io/api/core/v1"
@@ -111,6 +112,74 @@ func TestPsmdbBackupPrecondition(t *testing.T) {
111112
})
112113
}
113114

115+
// ---------------------------------------------------------------------------
116+
// Restore credentials re-pointed at the target cluster (DR after source delete)
117+
// ---------------------------------------------------------------------------
118+
119+
// TestPsmdbTargetCredentialsSecret pins the fix for the disaster-recovery gap:
120+
// a restore into a differently-named target must authenticate with the TARGET
121+
// cluster's own S3 credentials, because the source release's -s3-creds Secret is
122+
// deleted together with the source app. The selector prefers the source's
123+
// storage name, then the chart default, then a sole storage.
124+
func TestPsmdbTargetCredentialsSecret(t *testing.T) {
125+
s3Storage := func(secret string) runtime.RawExtension {
126+
return runtime.RawExtension{Raw: []byte(fmt.Sprintf(`{"type":"s3","s3":{"credentialsSecret":%q}}`, secret))}
127+
}
128+
clusterWith := func(storages map[string]runtime.RawExtension) *psmdbtypes.PerconaServerMongoDB {
129+
return &psmdbtypes.PerconaServerMongoDB{
130+
Spec: psmdbtypes.PerconaServerMongoDBSpec{
131+
Backup: psmdbtypes.PerconaServerMongoDBBackupConfig{Storages: storages},
132+
},
133+
}
134+
}
135+
cases := []struct {
136+
name string
137+
storages map[string]runtime.RawExtension
138+
preferred string
139+
want string
140+
}{
141+
{
142+
name: "preferred storage wins",
143+
storages: map[string]runtime.RawExtension{"s3-storage": s3Storage("target-creds"), "other": s3Storage("other-creds")},
144+
preferred: "s3-storage",
145+
want: "target-creds",
146+
},
147+
{
148+
name: "falls back to chart default when preferred absent",
149+
storages: map[string]runtime.RawExtension{"s3-storage": s3Storage("target-creds")},
150+
preferred: "nonexistent",
151+
want: "target-creds",
152+
},
153+
{
154+
name: "sole non-default storage is used",
155+
storages: map[string]runtime.RawExtension{"custom": s3Storage("custom-creds")},
156+
want: "custom-creds",
157+
},
158+
{
159+
name: "ambiguous (multiple, none default, no preferred) yields empty",
160+
storages: map[string]runtime.RawExtension{"a": s3Storage("a-creds"), "b": s3Storage("b-creds")},
161+
want: "",
162+
},
163+
{
164+
name: "no storages yields empty",
165+
storages: nil,
166+
want: "",
167+
},
168+
{
169+
name: "non-s3 storage yields empty",
170+
storages: map[string]runtime.RawExtension{"s3-storage": {Raw: []byte(`{"type":"azure","azure":{"container":"c"}}`)}},
171+
want: "",
172+
},
173+
}
174+
for _, tc := range cases {
175+
t.Run(tc.name, func(t *testing.T) {
176+
if got := psmdbTargetCredentialsSecret(clusterWith(tc.storages), tc.preferred); got != tc.want {
177+
t.Errorf("psmdbTargetCredentialsSecret: got %q want %q", got, tc.want)
178+
}
179+
})
180+
}
181+
}
182+
114183
// ---------------------------------------------------------------------------
115184
// Backup-side ensure idempotency + CR shape
116185
// ---------------------------------------------------------------------------
@@ -339,6 +408,32 @@ func TestParseMongoDBRestoreOptionsAndPITR(t *testing.T) {
339408
t.Errorf("deadline: got %v want 120s", o.effectiveRestoreDeadline())
340409
}
341410
})
411+
// A non-string recoveryTime (spec.options is free-form runtime.RawExtension,
412+
// so {"recoveryTime": 20260805} passes admission) must be treated as a
413+
// load-bearing decode failure: the caller fails the RestoreJob terminally
414+
// rather than silently degrading the PITR request to a snapshot restore. The
415+
// two conditions the caller ANDs are (a) the typed decode errors and (b) the
416+
// recoveryTime key is present.
417+
t.Run("non-string recoveryTime is a load-bearing decode failure (major-2 guard)", func(t *testing.T) {
418+
opts := &runtime.RawExtension{Raw: []byte(`{"recoveryTime":20260805}`)}
419+
if _, _, err := parseMongoDBRestoreOptions(opts); err == nil {
420+
t.Fatal("expected a typed-decode error for a numeric recoveryTime")
421+
}
422+
if !restoreOptionsCarryRecoveryTimeKey(opts) {
423+
t.Fatal("expected the recoveryTime key to be detected so the caller fails terminally")
424+
}
425+
})
426+
t.Run("recoveryTime-key detection is scoped to object blobs carrying the key", func(t *testing.T) {
427+
if restoreOptionsCarryRecoveryTimeKey(nil) {
428+
t.Error("nil options must report no recoveryTime key")
429+
}
430+
if restoreOptionsCarryRecoveryTimeKey(&runtime.RawExtension{Raw: []byte(`{"restoreTimeoutSeconds":30}`)}) {
431+
t.Error("a blob without recoveryTime must report no key (benign fall-back applies)")
432+
}
433+
if restoreOptionsCarryRecoveryTimeKey(&runtime.RawExtension{Raw: []byte(`"garbage`)}) {
434+
t.Error("a non-object blob must report no key (no recoveryTime to honour)")
435+
}
436+
})
342437
}
343438

344439
// ---------------------------------------------------------------------------

0 commit comments

Comments
 (0)