Configurable backup cleanup / retention is missing (no max-count policy, no unified age-based pruning) #3965

Problem / motivation

Cozystack's backup subsystem (backups.cozystack.io/v1alpha1: Plan / BackupJob / Backup / RestoreJob / BackupClass) has no configurable, first-class mechanism for cleaning up old backup DATA. There is no central retention/TTL field on the generic CRDs and no controller reconcile loop that enumerates old backups and prunes them. Retention is instead delegated per-strategy to the underlying operator/tool, and only two strategies actually expose a knob for it. The consequence:

• For ClickHouse/Altinity, MongoDB, etcd, FoundationDB and Job strategies there is no age- or count-based cleanup at all — backup artifacts accumulate in the object store indefinitely unless a tenant deletes them by hand.
• Plan schedules backups (cron) but exposes no retention field, even though code comments already reference a "Cozystack Plan-level retention" that does not exist yet (see mariadb_types.go:91, backup_controller.go:107,131).
• Deleting a Backup CR does not delete the S3 artifact for most strategies (the controller explicitly does not own the object-store data — "we do not own S3"). Only the Velero path purges data, via a DeleteBackupRequest.

There is currently no way for a user to say "keep the last N backups" (max-count policy) anywhere in the API. Age-based retention exists only where the underlying operator happens to support it.

What retention exists today (findings)

Additional facts:

• No age-based pruning loop exists in the controller. The only deletion logic is finalizer-driven cleanup on Backup CR deletion — backup_controller.go:60 → cleanupOnDelete (line 92, switch line 96). For CNPG/MariaDB/MongoDB/Altinity/FoundationDB/Job this is a no-op against S3.
• ttlSecondsAfterFinished is used in the repo only on unrelated one-shot k8s Jobs (etcd-operator pre-upgrade, grafana oidc, tenant cleanup, etc.) to GC the finished Job object — it has nothing to do with backup-data retention.
• Legacy red herring: packages/apps/clickhouse/values.yaml:127 cleanupStrategy (--keep-last=3 …, a restic-forget policy) is explicitly marked "Unused by the Altinity strategy" — so even the one place that looks like a max-count policy is dead for the current strategy.
• DESIGN.md already lists retention as a future/optional core responsibility, not implemented: api/backups/v1alpha1/DESIGN.md:377 — "Implement higher-level policies (retention) if needed."

Proposed solution

Introduce a configurable, strategy-agnostic backup cleanup mechanism, most naturally expressed at the Plan (or BackupClass) level so it applies uniformly regardless of the underlying driver. It should support more than one retention policy type, since these are genuinely different semantics that users need independently or in combination:

1. Age-based — keep backups newer than a duration / TTL (what CNPG retentionPolicy, MariaDB MaxRetention, and Velero ttl each already do in their own dialect).
2. Max-count — keep the last N backups regardless of age (not available anywhere today; needs investigation). This is the classic "keep last 7 dailies" knob and is the main gap this issue asks to close.
3. Potentially tiered/GFS policies later (keep-last / keep-daily / keep-weekly / keep-monthly), as the legacy clickhouse cleanupStrategy and Velero-adjacent tooling hint at.

The core would own enumeration + pruning of Backup objects (and delegate actual object-store deletion to the driver, as the Velero path already demonstrates with DeleteBackupRequest), so that strategies whose operators lack native retention (ClickHouse, MongoDB, etcd, FoundationDB) still get cleanup.

Scope check

• This affects a single component / package — it does not. It spans the generic backup framework (Plan/Backup API + controller) and every per-strategy driver. Per the template this likely warrants a design proposal rather than being picked up directly; filing here to capture the gap and the findings, and to start triage on which policy types (age vs max-count vs GFS) are in scope.
• I can describe the expected behavior concretely (the policy types above).

Alternatives considered

• Keep delegating per-strategy (status quo): leaves ClickHouse/MongoDB/etcd/FoundationDB with no cleanup at all, and gives users N different dialects (retentionPolicy vs MaxRetention vs ttl) for the same intent. No max-count anywhere.
• Push a design proposal first (community repo): probably the correct next step given the cross-cutting scope; this issue documents the motivation and current state to feed that.

Additional context

Key source references (paths relative to cozystack/cozystack):

• Generic API, no retention field: api/backups/v1alpha1/plan_types.go (PlanSpec, only Schedule), backup_types.go, backupjob_types.go, backupclass_types.go, restorejob_types.go.
• Per-strategy retention: api/backups/strategy/v1alpha1/cnpg_types.go:117, mariadb_types.go:93, velero_types.go:18.
• Cleanup dispatch: internal/backupcontroller/backup_controller.go:60,92,96 (per-strategy branches, most no-op vs S3); Velero data purge at :204.
• Projection of existing knobs: cnpgstrategy_controller.go:485, mariadbstrategy_controller.go:304.
• Chart defaults: packages/system/backupstrategy-controller/templates/strategy-cnpg-default.yaml:13, strategy-velero-vm{instance,disk}-default.yaml.
• Design note: api/backups/v1alpha1/DESIGN.md:377.
• Legacy/unused clickhouse cleanupStrategy: packages/apps/clickhouse/values.yaml:127, packages/apps/clickhouse/README.md:146.