backups: a stuck MariaDB run and namespace teardown leave dumps behind #5291

Describe the bug

Two paths leave a MariaDB dump and its k8s.mariadb.com/Backup behind after the dump cleanup added in #5233. Both fail safe (nothing that may belong to another Backup is deleted) but the storage they hold is never reclaimed.

1. A k8s.mariadb.com/Backup that never completes pins every later dump of its prefix. When the operator Backup stays Complete=False (MariaDB not ready, a Job pod that never starts), the BackupJob is marked Failed after mariadbDefaultBackupDeadline (30 minutes, internal/backupcontroller/mariadbstrategy_controller.go), but the operator Backup is left in place and no Cozystack Backup is created for it, so nothing ever deletes it. The dump cleanup treats a run that has not completed as one whose window is still open, because a run can upload its dump and prune older ones through maxRetention before it completes. From then on every dump of that application written after the stuck run was created is contested by it, so deleting any later Backup releases it with a Warning ArtifactNotDeleted naming the stuck run and leaves its dump and operator Backup behind.
2. Namespace teardown leaves dumps behind. Tenant roles grant no delete on backups.backups.cozystack.io (packages/system/backup-controller/templates/tenant-clusterroles.yaml), so apart from an administrator, namespace deletion is how these Backups are deleted. mariadb-operator puts no finalizer on its Backup, so during teardown the operator Backup is usually gone before the Cozystack Backup finalizer runs, and the cleanup, which reads bucket, prefix and credentials from the operator Backup, releases the Backup with a Warning and keeps the dump. This is the MariaDB case of Backups are orphaned on application/tenant deletion (no teardown GC of Backup CRs or S3 data) #3966.

Environment

• Cozystack: main with fix(mariadb): restore the dump a Backup was taken as #5227 and fix(backups): delete a MariaDB dump from S3 when its Backup is deleted #5233
• Provider: any, MariaDB backups on S3 storage

To Reproduce

Path 1:

1. Create a MariaDB application with a backup Plan on S3 storage.
2. Make one backup run unable to complete, for example by making its Job pod unschedulable, and wait for its BackupJob to fail on the deadline.
3. Let later backup runs succeed, then delete one of their Backups.

Path 2:

1. Create a MariaDB application in a tenant namespace and take a backup.
2. Delete the namespace.

Expected behaviour

Path 1: a run the BackupJob has given up on stops contesting later dumps, for example because its operator Backup is deleted when the BackupJob fails on the deadline (which also stops a Job that is still running from uploading a dump nobody tracks).

Path 2: a Backup that records its dump (k8s.mariadb.com/dump-file) on the platform bucket is deleted without the operator Backup: bucket and prefix are also in driverMetadata (k8s.mariadb.com/storage-uri) and status.underlyingResources, and the platform credentials do not depend on the namespace.

Actual behaviour

Path 1: every later Backup of the application is released with Warning ArtifactNotDeleted ... also falls in the lifetime of k8s.mariadb.com/Backup <ns>/<stuck run>, and its dump and operator Backup stay.

Path 2: the Backup is released with Warning ArtifactNotDeleted ... k8s.mariadb.com/Backup <name> is gone, and its dump stays in the bucket.

Logs

Warning  ArtifactNotDeleted  left s3://<bucket>/<prefix>/<dump> behind: the recorded dump also falls in the lifetime of k8s.mariadb.com/Backup <ns>/<stuck run>, which may have written it; the operator Backup is kept

Screenshots

Not applicable.

Additional context

Found in review of #5233 (review threads r4513233611, r4513233620, r4529413835 and r4529413863), where both paths were accepted as follow-ups because the fail-safe outcome is a leak rather than a deleted dump.

Related cases from the same reviews, also fail-safe, worth handling with the above:

• An operator Restore that never reaches Complete (its pods never schedule, the target MariaDB never becomes ready) holds the deletion of its Backup for good, with a Normal ArtifactCleanupWaiting every 30 seconds; only the backups.cozystack.io/skip-artifact-cleanup annotation releases it. A Warning after a bound would surface it.
• The Backup controller runs a single worker, so a few Backups held on an unreachable endpoint (each attempt bounded by a 10 second S3 timeout, retried every 30 seconds) delay the finalizer of every other Backup, other drivers included.
• A backend that answers the DELETE of a missing key with anything but 204 makes the delete fail and retry, since minio-go v7.0.98 accepts only 204; on a versioned bucket the delete writes a delete marker and frees no space.
• With the platform credentials Secret missing, the cleanup falls back to the endpoint recorded on the operator Backup; after an endpoint move it finds no dump there and records only a Normal ArtifactAlreadyAbsent. A Warning when the listed endpoint differs from the platform's would make that visible.

Checklist

• I have checked the documentation
• I have searched for similar issues
• I have included all required information
• I have provided clear steps to reproduce
• I have included relevant logs