Skip to content

Commit a7240ba

Browse files
authored
fix(backups): delete a MariaDB dump from S3 when its Backup is deleted (#5233)
## What this PR does Deleting a MariaDB `Backup` removed only the Cozystack object: the dump stayed in the bucket and the `k8s.mariadb.com/Backup` stayed in the namespace. Nothing else prunes such a dump on the platform bucket. mariadb-operator's `maxRetention` runs only inside the next backup run of the same prefix, and a `Plan` never deletes a `Backup`. Deleting a MariaDB `Backup` now deletes its dump, then the operator `Backup` that wrote it, and only then lets the `Backup` go. Every run of an application writes under the same prefix, so the only key ever deleted is one proven to belong to this `Backup`: - a `Backup` taken with #5227 records its dump in `driverMetadata` (`k8s.mariadb.com/dump-file`), and that key is deleted unless another run of the same prefix, finished or not, could have written it. The size probe that records the name lets only runs finished by then contest it, so a run that uploaded its dump and pruned this one through `maxRetention` before completing can have its dump recorded here; the delete contests the recorded dump again against every run as it stands, keeps it when any could own it, and releases the `Backup` with a `Warning`; - a `Backup` taken before that is matched with the same ownership-proven attribution #5223 introduced for the size probe and #5227 uses for legacy restores: one dump in the operator `Backup`'s lifetime on both clocks (the timestamp in the name and `LastModified`), and in the lifetime of no other run of the same prefix, finished or not: a run that has not completed yet may already have uploaded its dump and pruned this one through `maxRetention`, so its window is treated as still open. The match is recorded on the `Backup` before the delete, so a retry deletes the same key by name; - when no dump is proven, the `Backup` is released with a `Warning` event and nothing is deleted, the operator `Backup` included, since it is what any later attribution starts from. The delete runs inside the controller through the minio client #5223 added, the same library mariadb-operator writes with, so TLS is verified the same way. Bucket, prefix and dump name come from the operator `Backup`. When the operator `Backup` writes to the platform bucket, which it names by the bucket and by the projected `cozy-backups-creds` Secret, the endpoint and keys are the platform's current ones (read from the Secret `cozy-backups-creds` is projected from, TLS from the scheme of the platform endpoint, as the default strategy decides it, still trusting a CA the strategy pinned), so a `Backup` taken before the endpoint moved is still deleted. Any other bucket is reached as the operator `Backup` recorded it. Other cases: - a restore from the `Backup` holds the deletion while it runs: a `RestoreJob` of the `Backup` that has not finished, or an operator `Restore` of its operator `Backup` that mariadb-operator has not marked `Complete` (it does so once the Restore's Job has completed or failed). The second check matters because a `RestoreJob` can fail on its deadline while its operator `Restore` keeps reading the dump. Both are listed from the API server rather than the informer cache, so a restore that started just before the delete is not missed; - a restore does not start from a `Backup` that is being deleted or is gone: the MariaDB restore reads the `Backup` from the API server before creating the operator `Restore` and fails the `RestoreJob` otherwise, since mariadb-operator restores the nearest other dump once the requested one is deleted. A failed lookup of an existing operator `Restore` is retried rather than failing the `RestoreJob`; - a failed dump delete keeps the `Backup` in `Terminating` and retries every 30 seconds with a `Warning` naming the `backups.cozystack.io/skip-artifact-cleanup` annotation, which releases the `Backup` and deletes nothing; a failed delete of the operator `Backup` is retried with the controller's backoff; - a missing or malformed platform credentials Secret falls back to the credentials the operator `Backup` names, so it never releases a `Backup` on its own: a delete that then fails holds it as above. A failed read of that Secret (an API error) is retried instead, because the recorded endpoint may be the one the platform moved off; - a recorded dump that is not in the listing is not deleted again: the operator `Backup` is deleted and a `Normal` `ArtifactAlreadyAbsent` event names the key and the endpoint that was listed, so a dump left on storage the platform moved off can still be found; - in a terminating namespace a failed delete releases the `Backup` with a `Warning`, so it does not hold the namespace; - an operator `Backup` that is gone, that stores on a PVC or volume, or whose Job failed (`Complete=True` with reason `JobFailed`, which wrote no dump) releases the `Backup` with a `Warning` and nothing is deleted, as does a cluster without the mariadb-operator CRDs. The controller gains the `delete` verb on `k8s.mariadb.com/backups`. No Job, image or flag is added. Known limitation: when a namespace is deleted with its `Backup`s, the namespace controller may delete the operator `Backup` first, and the `Backup` is then released with a `Warning` and its dump left behind, because the bucket and prefix are read from the operator `Backup`. Known limitation: an operator `Backup` that never completes, for example one whose Job pod never starts, contests every later dump its lifetime covers on that prefix, so those `Backup`s are released with a `Warning` and their dumps left behind. That leaks storage rather than deleting a dump another `Backup` may restore from. The contest reads the runs from the operator `Backup`s that still exist, so a run whose operator `Backup` was deleted by hand no longer contests a dump. Likewise, an operator `Restore` that never completes holds its `Backup`'s deletion until the skip annotation releases it. These and the namespace-teardown limitation above are tracked in #5291. Stacked on #5227 and targets its branch until that merges; only the commits after #5227's head belong here. Tests: controller tests on the fake client cover each path above, including the window bound (`takenAt`, not the operator's `Complete` transition) of the legacy match, ordering (the operator `Backup` survives a failed dump delete, and a legacy `Backup` retried after a failed operator `Backup` delete finishes by its recorded name) and the finalizer at `Reconcile` level. The minio adapter is tested against an HTTP server for the escaped key and an error response. A helm-unittest pins the RBAC verbs. An unfinished neighbour run is covered both ways: one that started inside the `Backup`'s lifetime keeps its candidate dump, one that started after it does not contest an earlier dump. The coordination with restores is covered from both sides: a `RestoreJob` or operator `Restore` that has not finished, including one only the API server has seen yet, holds the deletion, and a restore from a `Backup` being deleted or gone is refused without creating an operator `Restore`. A recorded dump is contested the same way, down to the full sequence of the size probe recording a dump an unfinished run uploaded and the `Backup` then being deleted, and a recorded dump already gone is not contested. Every condition was mutated, each mutation turns a test red, including the timeout that bounds an unreachable endpoint. Checked on a test cluster against SeaweedFS with the controller function and its real minio client: over the in-cluster `seaweedfs-s3.<ns>:8333` endpoint with its wildcard certificate a recorded dump and a legacy dump were deleted with their operator `Backup`s, a neighbouring dump was kept, and a recorded dump already gone completed; with a dead recorded endpoint the dump on the platform bucket was deleted through the platform's current endpoint; on a foreign bucket behind a dead endpoint the `Backup` was held with everything kept. The contest of a recorded dump and the coordination with restores were added after that run and are covered by the controller tests only. ### Screenshots Not a UI change. ### Downstream repositories Walked against the diff. It touches no CRD or API type, no app `values.schema.json`, no `packages/core/platform/values.yaml`, no `hack/` tooling the trigger map names, and no ApplicationDefinition. Only `cozystack/website` is affected: `content/en/docs/next/operations/services/backup-classes.md` describes deletion of the artifact on `Backup` removal for Rabbitmq, Redis, Kafka and the MongoDB system bucket but not for MariaDB. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: needed, not opened yet: document MariaDB dump deletion on `Backup` removal and the skip annotation - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): deleting a MariaDB Backup on S3 storage now deletes its dump and its k8s.mariadb.com/Backup before the Backup goes. Only a dump proven to be the Backup's own is deleted: the one it records, unless another run could have written it, or for older Backups the one that can be attributed to its run. Otherwise the Backup is released with a Warning event and nothing is deleted. A failed delete keeps the Backup Terminating and retries; the backups.cozystack.io/skip-artifact-cleanup annotation releases it and keeps the dump. Deletion waits while a RestoreJob or an operator Restore reads the Backup's dump, and a restore from a Backup being deleted is refused. ```
2 parents 195de03 + b3ca798 commit a7240ba

10 files changed

Lines changed: 2002 additions & 41 deletions

File tree

‎internal/backupcontroller/artifact_size.go‎

Lines changed: 38 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,13 @@ type s3ObjectLister interface {
3838
ListObjects(ctx context.Context, bucket, prefix string, recursive bool) ([]s3ObjectInfo, error)
3939
}
4040

41+
type s3ObjectRemover interface {
42+
// RemoveObject deletes key. S3 answers 204 for a key that does not
43+
// exist and minio-go takes 204 as success; a backend answering a
44+
// missing key with any other status makes it an error.
45+
RemoveObject(ctx context.Context, bucket, key string) error
46+
}
47+
4148
// s3Connection is what a driver resolves from its rendered strategy to
4249
// read the bucket with the same identity the backup was written with.
4350
type s3Connection struct {
@@ -63,8 +70,10 @@ type objectStore struct {
6370
// reader reads Secrets, which the manager's client fetches uncached,
6471
// and lists operator Backups, which it serves from the cache.
6572
reader client.Reader
66-
// newLister overrides the S3 client; nil means a real minio client.
67-
newLister func(conn s3Connection) (s3ObjectLister, error)
73+
// newLister and newRemover override the S3 client; nil means a real
74+
// minio client.
75+
newLister func(conn s3Connection) (s3ObjectLister, error)
76+
newRemover func(conn s3Connection) (s3ObjectRemover, error)
6877
}
6978

7079
func (s objectStore) lister(conn s3Connection) (s3ObjectLister, error) {
@@ -74,6 +83,17 @@ func (s objectStore) lister(conn s3Connection) (s3ObjectLister, error) {
7483
return newMinioObjectLister(conn)
7584
}
7685

86+
func (s objectStore) remover(conn s3Connection) (s3ObjectRemover, error) {
87+
if s.newRemover != nil {
88+
return s.newRemover(conn)
89+
}
90+
c, err := newMinioObjectClient(conn)
91+
if err != nil {
92+
return nil, err
93+
}
94+
return c, nil
95+
}
96+
7797
func (s objectStore) secretValue(ctx context.Context, namespace, name, key string) (string, error) {
7898
secret := &corev1.Secret{}
7999
if err := s.reader.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, secret); err != nil {
@@ -94,6 +114,10 @@ func (r *RestoreJobReconciler) objectStore() objectStore {
94114
return objectStore{reader: r.Client, newLister: r.newS3ObjectLister}
95115
}
96116

117+
func (r *BackupReconciler) objectStore() objectStore {
118+
return objectStore{reader: r.Client, newLister: r.newS3ObjectLister, newRemover: r.newS3ObjectRemover}
119+
}
120+
97121
func (r *BackupJobReconciler) s3Lister(conn s3Connection) (s3ObjectLister, error) {
98122
return r.objectStore().lister(conn)
99123
}
@@ -103,6 +127,14 @@ type minioObjectLister struct {
103127
}
104128

105129
func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) {
130+
c, err := newMinioObjectClient(conn)
131+
if err != nil {
132+
return nil, err
133+
}
134+
return c, nil
135+
}
136+
137+
func newMinioObjectClient(conn s3Connection) (*minioObjectLister, error) {
106138
opts := &minio.Options{
107139
Creds: credentials.NewStaticV4(conn.AccessKey, conn.SecretKey, conn.SessionToken),
108140
Region: conn.Region,
@@ -152,6 +184,10 @@ func (l *minioObjectLister) ListObjects(ctx context.Context, bucket, prefix stri
152184
return collectObjects(ctx, l.client.ListObjects(ctx, bucket, minio.ListObjectsOptions{Prefix: prefix, Recursive: recursive}))
153185
}
154186

187+
func (l *minioObjectLister) RemoveObject(ctx context.Context, bucket, key string) error {
188+
return l.client.RemoveObject(ctx, bucket, key, minio.RemoveObjectOptions{})
189+
}
190+
155191
// collectObjects drains a minio listing. minio-go ends a listing whose
156192
// context is cancelled between pages by closing the channel without an
157193
// error item, so a cancelled context is checked once the channel closes:

‎internal/backupcontroller/backup_controller.go‎

Lines changed: 37 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -39,9 +39,22 @@ type BackupReconciler struct {
3939
client.Client
4040
Scheme *runtime.Scheme
4141
Recorder record.EventRecorder
42-
// CredentialsConfig lets the Rabbitmq cleanup path project cozy-backups-creds
43-
// into the Backup's namespace before spawning its best-effort delete Job.
42+
// CredentialsConfig locates the platform backup credentials: the Job-based
43+
// cleanups project them into the Backup's namespace, and the MariaDB
44+
// cleanup reads them to reach the platform bucket.
4445
CredentialsConfig BackupCredentialsConfig
46+
// APIReader is the manager's uncached reader. The MariaDB cleanup lists
47+
// the RestoreJobs and operator Restores of a Backup, and the operator
48+
// Backups that may own its dump, through it: a restore that started
49+
// before the delete, or a run that has already uploaded its dump, may
50+
// not be in the cache yet. Wired in SetupWithManager.
51+
APIReader client.Reader
52+
53+
// newS3ObjectLister and newS3ObjectRemover override the S3 client the
54+
// MariaDB cleanup deletes a dump with; nil means a real minio client.
55+
// Set by unit tests.
56+
newS3ObjectLister func(conn s3Connection) (s3ObjectLister, error)
57+
newS3ObjectRemover func(conn s3Connection) (s3ObjectRemover, error)
4558
}
4659

4760
func (r *BackupReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
@@ -126,22 +139,20 @@ func (r *BackupReconciler) cleanupOnDelete(ctx context.Context, backup *backupsv
126139
// orchestration" for tenant-facing guidance.
127140
return ctrl.Result{}, nil
128141
case strategyv1alpha1.MariaDBStrategyKind:
129-
// Cozystack Backup deletion does NOT delete the operator-side
130-
// k8s.mariadb.com/Backup CR or the backing S3/PVC archive.
131-
// Lifecycle of that CR is owned by mariadb-operator's
132-
// MaxRetention setting and by tenants / explicit teardown flows;
133-
// the controller's RBAC has no delete verb on
134-
// k8s.mariadb.com/backups for that reason. Surfacing this as an
135-
// explicit branch (rather than falling through to the Velero
136-
// default) keeps the contract honest: the MariaDB driver does
137-
// not own the archive, so it does not delete it on CR removal.
138-
// Mirrors the Altinity branch's "we do not own S3" stance.
139-
return ctrl.Result{}, nil
142+
// mariadb-operator's maxRetention prunes only from inside the next
143+
// backup run of the same prefix and a Plan never deletes a Backup, so
144+
// a dump on the platform bucket is deleted here or never. A dump in
145+
// a tenant's own bucket is deleted too: the Backup proves it owns
146+
// it, and tenants cannot delete Backups, so the delete comes from an
147+
// administrator or from namespace teardown. Teardown can remove the
148+
// operator Backup first, and the dump is then left behind with a
149+
// Warning, since its storage is read from it (see cleanupMariaDBBackup).
150+
return r.cleanupMariaDBBackup(ctx, backup)
140151
case strategyv1alpha1.MongoDBStrategyKind:
141152
// Split by flow (see cleanupMongoDBBackup). A legacy backup writes to the
142153
// tenant's own bucket and its lifecycle stays the tenant's / psmdb
143154
// operator's — that branch is a no-op, the same "we do not own the
144-
// archive" contract as Altinity / MariaDB / FDB. A useSystemBucket backup
155+
// archive" contract as Altinity / FDB. A useSystemBucket backup
145156
// lands in the shared cozy-backups bucket with no psmdb task retention to
146157
// prune it, so the driver OWNS it: this deletes the operator
147158
// PerconaServerMongoDBBackup CR and waits for its delete-backup finalizer
@@ -154,7 +165,7 @@ func (r *BackupReconciler) cleanupOnDelete(ctx context.Context, backup *backupsv
154165
// is governed by the next BackupJob's stopOtherFoundationDBBackups
155166
// call, by manual operator action, or by
156167
// examples/backups/foundationdb/cleanup.sh. Same "we do not own
157-
// the archive" contract as Altinity / MariaDB — the explicit
168+
// the archive" contract as Altinity — the explicit
158169
// branch guards the seam against a future driver refactor that
159170
// might incidentally stamp velero.io/backup-name onto FDB
160171
// driverMetadata via a shared helper.
@@ -291,7 +302,18 @@ func (r *BackupReconciler) cleanupCNPGBackup(ctx context.Context, backup *backup
291302

292303
// SetupWithManager registers the BackupReconciler with the Manager.
293304
func (r *BackupReconciler) SetupWithManager(mgr ctrl.Manager) error {
305+
r.APIReader = mgr.GetAPIReader()
294306
return ctrl.NewControllerManagedBy(mgr).
295307
For(&backupsv1alpha1.Backup{}).
296308
Complete(r)
297309
}
310+
311+
// apiReader returns the uncached reader. Reconcilers built by hand in unit
312+
// tests have none and fall back to their fake Client, which has no cache to
313+
// bypass.
314+
func (r *BackupReconciler) apiReader() client.Reader {
315+
if r.APIReader != nil {
316+
return r.APIReader
317+
}
318+
return r.Client
319+
}

‎internal/backupcontroller/backup_controller_test.go‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -176,11 +176,8 @@ func TestBackupCleanup_Altinity_DoesNotFallThroughToVelero(t *testing.T) {
176176

177177
// TestBackupCleanup_MariaDB_DoesNotFallThroughToVelero locks in the same
178178
// invariant as the Altinity test above for the MariaDB strategy. The
179-
// dispatcher must explicitly route MariaDB-strategy Backups to a no-op
180-
// cleanup rather than fall through to the Velero default - the MariaDB
181-
// driver does not own the operator-side k8s.mariadb.com/Backup CR or the
182-
// S3/PVC archive behind it (rbac.yaml grants no delete verb on
183-
// k8s.mariadb.com/backups for that reason). The test seeds the Velero
179+
// dispatcher must route MariaDB-strategy Backups to their own cleanup
180+
// rather than fall through to the Velero default. The test seeds the Velero
184181
// metadata keys plus a matching velero.io/Backup; with the explicit branch
185182
// no DeleteBackupRequest is synthesised, and the Velero Backup survives.
186183
// Without the branch the default falls through to cleanupVeleroBackup,

‎internal/backupcontroller/mariadb_artifact_size.go‎

Lines changed: 33 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -60,11 +60,23 @@ func (r *BackupJobReconciler) attributedMariaDBDump(ctx context.Context, mdbBack
6060
// from can have been edited since, and pointing at another bucket now says
6161
// nothing about where this dump went.
6262
func findMariaDBDump(ctx context.Context, store objectStore, mdbBackup *mariadbtypes.Backup, completedAt time.Time) (mariadbDump, error) {
63-
lister, s3, err := mariadbStorage(ctx, store, mdbBackup)
63+
lister, _, err := mariadbStorage(ctx, store, mdbBackup)
6464
if err != nil {
6565
return mariadbDump{}, err
6666
}
67-
others, err := finishedMariaDBRuns(ctx, store.reader, mdbBackup)
67+
return findMariaDBDumpWith(ctx, lister, store.reader, mdbBackup, completedAt, false)
68+
}
69+
70+
// findMariaDBDumpWith is findMariaDBDump reading the bucket through lister,
71+
// for a caller that reaches the bucket by another endpoint than the one
72+
// mdbBackup names. withUnfinished lets runs that have not completed contest
73+
// the dump too, as neighbourMariaDBRuns describes.
74+
func findMariaDBDumpWith(ctx context.Context, lister s3ObjectLister, reader client.Reader, mdbBackup *mariadbtypes.Backup, completedAt time.Time, withUnfinished bool) (mariadbDump, error) {
75+
s3 := mdbBackup.Spec.Storage.S3
76+
if s3 == nil {
77+
return mariadbDump{}, errors.New("the operator Backup does not store to S3")
78+
}
79+
others, err := neighbourMariaDBRuns(ctx, reader, mdbBackup, withUnfinished)
6880
if err != nil {
6981
return mariadbDump{}, fmt.Errorf("list operator Backups: %w", err)
7082
}
@@ -125,6 +137,10 @@ type mariadbRun struct {
125137
lower, upper time.Time
126138
}
127139

140+
// mariadbUnfinishedRunEnd closes the window of a run that has not completed,
141+
// which can still write its dump at any time.
142+
var mariadbUnfinishedRunEnd = time.Date(9999, time.December, 31, 0, 0, 0, 0, time.UTC)
143+
128144
func newMariaDBRun(b *mariadbtypes.Backup, created, completed time.Time) mariadbRun {
129145
return mariadbRun{
130146
name: b.Namespace + "/" + b.Name,
@@ -150,8 +166,8 @@ func (r mariadbRun) placesOnEitherClock(d mariadbDump) bool {
150166
// dump in the window on one clock only may be this run's own, misplaced by
151167
// a skewed clock, which leaves any other candidate unproven. A neighbour's
152168
// dump is still taken if this run's own is missing from the listing, or
153-
// misplaced on both clocks, and the neighbour's operator Backup is gone or
154-
// has not completed.
169+
// misplaced on both clocks, and the neighbour's operator Backup is gone or,
170+
// unless others includes unfinished runs, has not completed.
155171
func matchMariaDBDump(objects []s3ObjectInfo, prefix string, own mariadbRun, others []mariadbRun) (mariadbDump, error) {
156172
var near []mariadbDump
157173
for _, obj := range objects {
@@ -183,14 +199,17 @@ func matchMariaDBDump(objects []s3ObjectInfo, prefix string, own mariadbRun, oth
183199
return d, nil
184200
}
185201

186-
// finishedMariaDBRuns returns the runs of the other operator Backups that
202+
// neighbourMariaDBRuns returns the runs of the other operator Backups that
187203
// write under the same bucket and prefix as b and have completed, whether
188-
// their Job succeeded or failed. One still running is left out: its window
189-
// has no end yet, and an operator Backup stuck short of Complete, as one
190-
// whose Job pod never starts is, would otherwise contest every later run of
191-
// the application for good. The endpoint is not compared, since one has
192-
// several spellings, and a false neighbour only leaves a size unset.
193-
func finishedMariaDBRuns(ctx context.Context, reader client.Reader, b *mariadbtypes.Backup) ([]mariadbRun, error) {
204+
// their Job succeeded or failed. One still running is left out unless
205+
// withUnfinished is set: its window has no end yet, and an operator Backup
206+
// stuck short of Complete, as one whose Job pod never starts is, would
207+
// otherwise contest every later run of the application for good. That is
208+
// the price of deleting a dump, since a run can upload its dump and prune
209+
// older ones through maxRetention before it completes. The endpoint is not
210+
// compared, since one has several spellings, and a false neighbour only
211+
// keeps a dump unattributed.
212+
func neighbourMariaDBRuns(ctx context.Context, reader client.Reader, b *mariadbtypes.Backup, withUnfinished bool) ([]mariadbRun, error) {
194213
list := &mariadbtypes.BackupList{}
195214
if err := reader.List(ctx, list); err != nil {
196215
return nil, err
@@ -207,6 +226,9 @@ func finishedMariaDBRuns(ctx context.Context, reader client.Reader, b *mariadbty
207226
}
208227
cond := apimeta.FindStatusCondition(other.Status.Conditions, mariadbtypes.ConditionTypeComplete)
209228
if cond == nil || cond.Status != metav1.ConditionTrue {
229+
if withUnfinished {
230+
runs = append(runs, newMariaDBRun(other, other.CreationTimestamp.Time, mariadbUnfinishedRunEnd))
231+
}
210232
continue
211233
}
212234
runs = append(runs, newMariaDBRun(other, other.CreationTimestamp.Time, cond.LastTransitionTime.Time))

‎internal/backupcontroller/mariadb_restore_dump_test.go‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ import (
1414
"k8s.io/client-go/tools/record"
1515
ctrl "sigs.k8s.io/controller-runtime"
1616
"sigs.k8s.io/controller-runtime/pkg/client"
17+
"sigs.k8s.io/controller-runtime/pkg/client/interceptor"
1718

1819
backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1"
1920
"github.com/cozystack/cozystack/internal/backupcontroller/mariadbapp"
@@ -184,6 +185,117 @@ func recordedDump(name string) map[string]string {
184185
return map[string]string{mariadbDumpFileKey: name, mariadbDumpTimeKey: mdbRestoreTime}
185186
}
186187

188+
// A Backup being deleted is deleting its dump, so a restore that starts
189+
// now can find it gone once the operator reads it, and the operator then
190+
// restores the nearest other dump. The Backup is read from the API server,
191+
// since the copy the reconcile was handed can predate the delete.
192+
func TestReconcileMariaDBRestore_RefusesABackupBeingDeleted(t *testing.T) {
193+
creds := s3CredsSecret("tenant", "creds")
194+
present := func() *fakeS3Lister {
195+
return &fakeS3Lister{objects: []s3ObjectInfo{{Key: mdbRestorePrefix + mdbRestoreDump, LastModified: mdbRestoreT0.Add(20 * time.Second)}}}
196+
}
197+
cases := map[string]map[string]string{"recorded dump": recordedDump(mdbRestoreDump), "legacy Backup": nil}
198+
for name, md := range cases {
199+
t.Run(name+", being deleted", func(t *testing.T) {
200+
f := newMariaDBRestoreFixture(t, md, present(), creds)
201+
ctx := context.Background()
202+
live := &backupsv1alpha1.Backup{}
203+
if err := f.c.Get(ctx, client.ObjectKeyFromObject(f.backup), live); err != nil {
204+
t.Fatal(err)
205+
}
206+
live.Finalizers = []string{backupFinalizer}
207+
if err := f.c.Update(ctx, live); err != nil {
208+
t.Fatal(err)
209+
}
210+
if err := f.c.Delete(ctx, live); err != nil {
211+
t.Fatal(err)
212+
}
213+
f.reconcile(t)
214+
f.wantFailed(t, "src-bk", "being deleted")
215+
})
216+
t.Run(name+", already gone", func(t *testing.T) {
217+
f := newMariaDBRestoreFixture(t, md, present(), creds)
218+
if err := f.c.Delete(context.Background(), f.backup.DeepCopy()); err != nil {
219+
t.Fatal(err)
220+
}
221+
f.reconcile(t)
222+
f.wantFailed(t, "src-bk", "is gone")
223+
})
224+
}
225+
// The refusal applies only before the operator Restore exists. One
226+
// created by an earlier reconcile but not yet in the cache must be
227+
// found and followed, or the RestoreJob fails under a running restore
228+
// and the cleanup deletes the dump that restore reads.
229+
t.Run("an operator Restore the cache has not seen yet is followed", func(t *testing.T) {
230+
f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), present(), creds)
231+
ctx := context.Background()
232+
live := &backupsv1alpha1.Backup{}
233+
if err := f.c.Get(ctx, client.ObjectKeyFromObject(f.backup), live); err != nil {
234+
t.Fatal(err)
235+
}
236+
live.Finalizers = []string{backupFinalizer}
237+
if err := f.c.Update(ctx, live); err != nil {
238+
t.Fatal(err)
239+
}
240+
if err := f.c.Delete(ctx, live); err != nil {
241+
t.Fatal(err)
242+
}
243+
if err := f.c.Get(ctx, client.ObjectKeyFromObject(live), live); err != nil || live.DeletionTimestamp.IsZero() {
244+
t.Fatalf("Backup not left being deleted: %v", err)
245+
}
246+
live.ResourceVersion = ""
247+
apiServer := newMariaDBStrategyTestClient(t, live, &mariadbtypes.Restore{ObjectMeta: metav1.ObjectMeta{
248+
Namespace: "tenant", Name: "rj-abcde",
249+
Labels: map[string]string{
250+
backupsv1alpha1.OwningJobNameLabel: "rj",
251+
backupsv1alpha1.OwningJobNamespaceLabel: "tenant",
252+
},
253+
}})
254+
f.r.APIReader = apiServer
255+
f.reconcile(t)
256+
if f.rj.Status.Phase == backupsv1alpha1.RestoreJobPhaseFailed {
257+
t.Fatalf("RestoreJob failed under a running restore: %s", f.rj.Status.Message)
258+
}
259+
if got := f.restores(t); len(got) != 0 {
260+
t.Errorf("created %d operator Restores; want the existing one followed", len(got))
261+
}
262+
})
263+
// A failed lookup says nothing about whether the operator Restore
264+
// exists, so the RestoreJob must not be failed under one that runs.
265+
t.Run("an operator Restore lookup that fails is retried", func(t *testing.T) {
266+
f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), present(), creds)
267+
f.r.APIReader = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{
268+
List: func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error {
269+
if _, ok := list.(*mariadbtypes.RestoreList); ok {
270+
return errors.New("etcdserver: request timed out")
271+
}
272+
return c.List(ctx, list, opts...)
273+
},
274+
})
275+
if _, err := f.r.reconcileMariaDBRestore(context.Background(), f.rj, f.backup); err == nil {
276+
t.Error("err = nil, want the failed lookup returned for a retry")
277+
}
278+
if f.rj.Status.Phase == backupsv1alpha1.RestoreJobPhaseFailed {
279+
t.Fatalf("RestoreJob failed on a lookup error: %s", f.rj.Status.Message)
280+
}
281+
if got := f.restores(t); len(got) != 0 {
282+
t.Errorf("created %d operator Restores without knowing whether one exists", len(got))
283+
}
284+
})
285+
t.Run("a Backup that cannot be read keeps the RestoreJob waiting", func(t *testing.T) {
286+
f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), present(), creds)
287+
f.r.APIReader = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{
288+
Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error {
289+
if _, ok := obj.(*backupsv1alpha1.Backup); ok {
290+
return errors.New("apiserver unavailable")
291+
}
292+
return c.Get(ctx, key, obj, opts...)
293+
},
294+
})
295+
f.wantWaiting(t, f.reconcile(t), "src-bk", "apiserver unavailable")
296+
})
297+
}
298+
187299
func TestReconcileMariaDBRestore_RecordedDump(t *testing.T) {
188300
creds := s3CredsSecret("tenant", "creds")
189301

0 commit comments

Comments
 (0)