Backups are orphaned on application/tenant deletion (no teardown GC of Backup CRs or S3 data) #3966

Related to #3965 (missing configurable backup retention/cleanup). This is a distinct but adjacent gap: #3965 is about pruning live backups by a policy (age / max-count); this issue is about garbage-collecting backups when the application or tenant itself is deleted — which no retention policy can cover, because retention is enforced only by the live operator.

Problem / motivation

When a user deletes a whole managed application (e.g. an apps.cozystack.io Postgres cluster) or an entire tenant namespace, the backup artifacts in the object store are left behind forever, with nothing tracking them. Orphaned backups are effectively the designed default: the backup subsystem intentionally decouples backup artifacts from application lifecycle in both directions, and there is no garbage collection at teardown.

This means:

• The S3 archive under s3://<bucket>/<namespace>/<app>/ (and the barman serverName prefix) survives the app/tenant and is never reclaimed — silent, unbounded storage growth in the shared cozy-backups bucket.
• Dangling Backup CRs point at a spec.applicationRef that no longer resolves; a surviving Plan keeps firing failing BackupJobs against the deleted app.

Findings (current behavior)

1. Backup CRs carry no ownerReference to the application (by design)

• Backup objects are created with a bare ObjectMeta, no ownerReferences — explicit for Velero at internal/backupcontroller/velerostrategy_controller.go:539-545 ("No OwnerReferences set on Backup. The Backup must survive BackupJob deletion so users don't lose their backup artifacts"), and the same for every other driver (e.g. cnpgstrategy_controller.go:607-611). So Backup CRs survive deletion of the BackupJob, the Plan, and the application, becoming dangling records.
• Plan carries no app ownerReference and no finalizer — internal/backupcontroller/plan_controller.go uses a plain For(&Plan{}) with no deletion handling. The only ownerRef the core sets is Plan → BackupJob (plan_controller.go:84-87). Deleting the app does not GC the Plan; it keeps producing failing BackupJobs.
• Nothing watches application / Tenant / namespace deletion — there is no Watches/Owns on apps.cozystack.io, HelmRelease, or Tenant anywhere in internal/backupcontroller/.

2. S3 data is not deleted on application deletion

• The only owner link touched at app deletion is Cluster → barman ObjectStore CR GC (internal/backupcontroller/cnpgstrategy_controller.go:470-482): k8s garbage-collects the ObjectStore config object, but that does not tell barman to purge the S3 archive. No code path deletes the s3://.../<ns>/<name>/ archive when the Cluster/ObjectStore is removed.
• retentionPolicy: "30d" (packages/system/backupstrategy-controller/templates/strategy-cnpg-default.yaml:13) does not help here — it is enforced by the live Cluster's barman sidecar, which no longer exists after deletion. The archive freezes at its last state and is orphaned.
• All other engines are explicitly documented as "we do not own the archive" (Altinity/MariaDB/MongoDB/FoundationDB branches in internal/backupcontroller/backup_controller.go:96-160; CNPG single-Backup deletion note at :227-233). The controller has no delete RBAC on the operator backup CRs (packages/system/backupstrategy-controller/templates/rbac.yaml). Velero is the only engine that purges data, and only when its own Backup CR is deleted — never on app deletion.

3. Shared-bucket (useSystemBucket) prefixes are stranded on tenant teardown

• Default strategy scopes each app by a path prefix: serverName: "{{ .Application.metadata.namespace }}-{{ .Application.metadata.name }}" and destinationPath: "s3://<bucket>/{{ namespace }}/{{ name }}/" (strategy-cnpg-default.yaml:9-11; api/backups/v1alpha1/DESIGN.md:25,30).
• When the whole tenant namespace is deleted, the k8s objects (Cluster, ObjectStore, Backup CRs, Plans) vanish, but the s3://cozy-backups/<namespace>/<app>/ prefix in the shared bucket is left with every reference gone. There is no bucket-level lifecycle/expiration rule (none found in packages/apps/bucket/ or the strategy controller).

4. No orphan detection / GC exists anywhere

• grep of orphan|stale|dangling|garbage|reap|prune|unreferenced across internal/backupcontroller/, api/backups/, and the strategy-controller package returns only: comments about kube-GC of CRs via ownerRefs, informer-cache "stale" usage, and Velero's DeletePropagationOrphan for PVCs — none of it about backup archives.
• There is no controller, CronJob, or script that scans for S3 prefixes / Backup CRs whose application no longer exists. (The only whole-engine cleanup helper is a manual examples/backups/foundationdb/cleanup.sh.)

5. Restore-into-new adds another orphanable prefix

• Recovery reads from the source serverName/destinationPath (cnpgstrategy_controller.go:1099-1103), but the newly created target app then archives under its own <ns>-<name> serverName / s3://.../<ns>/<name>/ prefix. Deleting the restored app orphans that second prefix too.

Concrete orphan scenarios

Proposed solution

Introduce garbage collection tied to application/tenant lifecycle. Options to weigh (design proposal likely needed, see scope check):

• A controller that watches application / Tenant deletion and, on teardown, either deletes the associated Backup/Plan CRs and issues a driver-specific archive purge, or (safer default) marks them orphaned and surfaces them.
• A teardown purge hook per strategy that deletes the S3 prefix when the app is deleted (opt-in, since some users deliberately keep backups after deleting the source — so this must be a policy, e.g. deletionPolicy: Retain|Delete).
• An orphan-scan / reaper (controller or CronJob) that lists Backup CRs whose applicationRef no longer resolves and object-store prefixes with no owning app, and reports or reclaims them.
• As a stopgap, a bucket-level S3 lifecycle rule so stranded prefixes eventually expire even with no k8s tracking.

Whatever the mechanism, deleting a backup's source should not silently strand its data with no way to ever find or reclaim it.

Scope check

• Single component / package — spans the generic backup framework (Plan/Backup lifecycle), every per-strategy driver, and the tenant/bucket teardown flow. Likely warrants a design proposal; filing here to capture the gap and findings for triage.
• I can describe the expected behavior concretely (GC on app/tenant deletion, with a Retain/Delete policy).

Alternatives considered

• Status quo (document "clean the bucket yourself"): leaves silent unbounded growth in the shared cozy-backups bucket and dangling CRs; unacceptable for a multi-tenant platform.
• Always cascade-delete backups with the app (ownerReference on Backup): rejected upstream by design — see velerostrategy_controller.go:539 — because users must be able to restore after deleting the source. Hence a policy (Retain/Delete) rather than a hard ownerRef.

Additional context

• Related: Configurable backup cleanup / retention is missing (no max-count policy, no unified age-based pruning) #3965 (configurable retention / cleanup policy for live backups; age vs max-count).
• Load-bearing source: internal/backupcontroller/plan_controller.go:84, internal/backupcontroller/backup_controller.go:96-160,227-233, internal/backupcontroller/cnpgstrategy_controller.go:470-482,607-611,1099-1103, internal/backupcontroller/velerostrategy_controller.go:539, packages/system/backupstrategy-controller/templates/strategy-cnpg-default.yaml:9-13, packages/system/backupstrategy-controller/templates/rbac.yaml, packages/apps/postgres/templates/backup.yaml, api/backups/v1alpha1/DESIGN.md:25,30.