docs(backups): describe BackupRetentionPolicy in backups core API design - #4192

…d-out backup so it cannot orphan an archive

Every prose surface (the useSystemBucket field, the retentionPolicy field, and
backup-classes.md) tells a tenant that schedule and retentionPolicy are ignored
on this flow, and cozyvalues-gen copies that into the schema, README and the
dashboard. A render-time reject of a non-default retentionPolicy contradicted
all of them: a tenant who followed the documented opt-in with a retentionPolicy
still set got a failed render and a HelmRelease that stopped converging, pointed
at the very doc that says the field does nothing. Drop the reject so
retentionPolicy is inert exactly as documented, handled the same way as schedule
(gated out, no error). The absence of retention on this flow is real but belongs
to a follow-up, so the docs now point at the BackupRetentionPolicy design (#4192)
and the teardown-GC issue (#3966), and recommend an S3 lifecycle policy until
they land.

Cancel a timed-out operator backup instead of only failing the BackupJob. The
deadline fires for a backup the operator has not started; but `waiting` is the
state pbm reports for a backup queued behind another, and the operator starts it
once the slot frees — after the BackupJob is Failed and never reconciled again,
writing an archive into the shared bucket that no Backup object represents. The
deadline branch now deletes the operator CR before failing, which prevents that
late run and, on the useSystemBucket flow, lets its delete-backup finalizer take
anything partially written; the delete is best-effort so it never blocks the
failure.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <[email protected]>

…d-out backup so it cannot orphan an archive

Every prose surface (the useSystemBucket field, the retentionPolicy field, and
backup-classes.md) tells a tenant that schedule and retentionPolicy are ignored
on this flow, and cozyvalues-gen copies that into the schema, README and the
dashboard. A render-time reject of a non-default retentionPolicy contradicted
all of them: a tenant who followed the documented opt-in with a retentionPolicy
still set got a failed render and a HelmRelease that stopped converging, pointed
at the very doc that says the field does nothing. Drop the reject so
retentionPolicy is inert exactly as documented, handled the same way as schedule
(gated out, no error). The absence of retention on this flow is real but belongs
to a follow-up, so the docs now point at the BackupRetentionPolicy design (#4192)
and the teardown-GC issue (#3966), and recommend an S3 lifecycle policy until
they land.

Cancel a timed-out operator backup instead of only failing the BackupJob. The
deadline fires for a backup the operator has not begun streaming; but `waiting`
is the state pbm reports for a backup queued behind another, and the operator
starts it once the slot frees — after the BackupJob is Failed and never
reconciled again, writing an archive into the shared bucket that no Backup
object represents. The deadline branch now deletes the operator CR before
failing, which prevents that late run entirely: at v1.22.0 the operator returns
for a CR with a deletion timestamp before it dispatches to pbm, so nothing is
written. The delete is best-effort so it never blocks the failure.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <[email protected]>

…d-out backup so it cannot orphan an archive

Every prose surface (the useSystemBucket field, the retentionPolicy field, and
backup-classes.md) tells a tenant that schedule and retentionPolicy are ignored
on this flow, and cozyvalues-gen copies that into the schema, README and the
dashboard. A render-time reject of a non-default retentionPolicy contradicted
all of them: a tenant who followed the documented opt-in with a retentionPolicy
still set got a failed render and a HelmRelease that stopped converging, pointed
at the very doc that says the field does nothing. Drop the reject so
retentionPolicy is inert exactly as documented, handled the same way as schedule
(gated out, no error). The absence of retention on this flow is real but belongs
to a follow-up, so the docs now point at the BackupRetentionPolicy design (#4192)
and the teardown-GC issue (#3966), and recommend an S3 lifecycle policy until
they land.

Cancel a timed-out operator backup instead of only failing the BackupJob. The
deadline fires for a backup the operator has not begun streaming; but `waiting`
is the state pbm reports for a backup queued behind another, and the operator
starts it once the slot frees — after the BackupJob is Failed and never
reconciled again, writing an archive into the shared bucket that no Backup
object represents. The deadline branch now deletes the operator CR before
failing, which prevents that late run entirely: at v1.22.0 the operator returns
for a CR with a deletion timestamp before it dispatches to pbm, so nothing is
written. The delete is best-effort so it never blocks the failure.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <[email protected]>

…d-out backup so it cannot orphan an archive

Every prose surface (the useSystemBucket field, the retentionPolicy field, and
backup-classes.md) tells a tenant that schedule and retentionPolicy are ignored
on this flow, and cozyvalues-gen copies that into the schema, README and the
dashboard. A render-time reject of a non-default retentionPolicy contradicted
all of them: a tenant who followed the documented opt-in with a retentionPolicy
still set got a failed render and a HelmRelease that stopped converging, pointed
at the very doc that says the field does nothing. Drop the reject so
retentionPolicy is inert exactly as documented, handled the same way as schedule
(gated out, no error). The absence of retention on this flow is real but belongs
to a follow-up, so the docs now point at the BackupRetentionPolicy design (#4192)
and the teardown-GC issue (#3966), and recommend an S3 lifecycle policy until
they land.

Cancel a timed-out operator backup instead of only failing the BackupJob. The
deadline fires for a backup the operator has not begun streaming; but `waiting`
is the state pbm reports for a backup queued behind another, and the operator
starts it once the slot frees — after the BackupJob is Failed and never
reconciled again, writing an archive into the shared bucket that no Backup
object represents. The deadline branch now deletes the operator CR before
failing, which prevents that late run entirely: at v1.22.0 the operator returns
for a CR with a deletion timestamp before it dispatches to pbm, so nothing is
written. The delete is best-effort so it never blocks the failure.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <[email protected]>

…d-out backup so it cannot orphan an archive

Every prose surface (the useSystemBucket field, the retentionPolicy field, and
backup-classes.md) tells a tenant that schedule and retentionPolicy are ignored
on this flow, and cozyvalues-gen copies that into the schema, README and the
dashboard. A render-time reject of a non-default retentionPolicy contradicted
all of them: a tenant who followed the documented opt-in with a retentionPolicy
still set got a failed render and a HelmRelease that stopped converging, pointed
at the very doc that says the field does nothing. Drop the reject so
retentionPolicy is inert exactly as documented, handled the same way as schedule
(gated out, no error). The absence of retention on this flow is real but belongs
to a follow-up, so the docs now point at the BackupRetentionPolicy design (#4192)
and the teardown-GC issue (#3966), and recommend an S3 lifecycle policy until
they land.

Cancel a timed-out operator backup instead of only failing the BackupJob. The
deadline fires for a backup the operator has not begun streaming; but `waiting`
is the state pbm reports for a backup queued behind another, and the operator
starts it once the slot frees — after the BackupJob is Failed and never
reconciled again, writing an archive into the shared bucket that no Backup
object represents. The deadline branch now deletes the operator CR before
failing, which prevents that late run entirely: at v1.22.0 the operator returns
for a CR with a deletion timestamp before it dispatches to pbm, so nothing is
written. The delete is best-effort so it never blocks the failure.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <[email protected]>

#3967)

## What this PR does

MongoDB was the only `cozy-default` app with **no `useSystemBucket`
flow**: its backup storage was always static on the
`PerconaServerMongoDB` CR (bucket/endpoint/credentials from `backup.*`
values plus a chart-emitted `<release>-s3-creds` Secret), so a tenant
could not back up to the platform system bucket out of the box the way
Postgres/ClickHouse/MariaDB/etcd do. This closes that gap so the default
MongoDB backup-restore works OOTB.

PBM takes `bucket`/`endpointUrl`/`region`/`prefix` as **static CR
fields** (not secret refs), and the platform bucket name/endpoint are
only known at BackupJob time — so, unlike ClickHouse's runtime
`secretKeyRef` env, the chart cannot render them. The implementation
mirrors the CNPG driver's pattern:

- **Chart** (`packages/apps/mongodb`): a new `backup.useSystemBucket`
flag keeps `spec.backup.enabled` (so the pbm agents run) but omits the
static `storages`/`tasks`/`pitr` and the `<release>-s3-creds` Secret.
Helm never owns `spec.backup.storages`, so a Flux re-render can't fight
the driver. A `bootstrap.enabled + useSystemBucket` render guard mirrors
Postgres, and `useSystemBucket: true` with `backup.enabled: false` is
refused both at render time and by an `x-kubernetes-validations` rule on
`backup` in the schema (the same mechanism as the `storageClass`
immutability rule). Note on the schema rule: the aggregated apiserver
does not evaluate CEL on application-backed resources yet (#2657), so
today it is a UI/schema-level marker and the render-time refusal is what
stops a Flux release; it also trips the API review gate (“validation
rule added”), so this PR needs an API-owner approval on top of the code
review.
- **Strategy** (`cozy-default-mongodb`): now carries the S3 coordinates
(`endpointUrl` already scheme'd by the
`backupstrategy-controller.endpoint` helper, `credentialsSecret:
cozy-backups-creds`, `prefix: <namespace>/<application>`).
- **Driver** (`psmdbstrategy_controller.go`): SSA-injects
`spec.backup.storages[s3-storage]` onto the live `PerconaServerMongoDB`
at BackupJob time under a dedicated field manager
(`cozystack-psmdb-backup-driver`) with `ForceOwnership`. Injection is
**gated on the app's `backup.useSystemBucket` flag** (a legacy app
leaves it false and is never touched; a useSystemBucket app is
re-injected once per mint (once per BackupJob, plus once per
storage-race re-mint), before its operator CR is created, so a later
coordinate change catches up on the next job while a streaming dump is
never patched under). The backup CR is minted only once the operator
reports the storage pushed into pbm (`status.observedGeneration` caught
up and `PBMReady` True): pbm writes to what its own config says, not to
the spec, so a CR minted before that push would land at the previous
coordinates while its Backup recorded the new ones; the job waits at
`PerconaServerMongoDBPBMConfigPending` meanwhile, bounded by the backup
deadline. Once a job has its CR, none of the mint-time reads or holds
can fail it: the strategy and app reads the artifact still needs are
bounded by the 24h running ceiling instead, and the cluster is not read
at all **and on the release having rendered that flag**: the app CR is
projected from the HelmRelease's desired values, so the flag reads true
before helm-controller renders the revision, while the tenant's own
storage, scheduled tasks and PITR stream from the previous render still
sit on the cluster. As long as the cluster carries `spec.backup.tasks`
or `spec.backup.pitr`, the driver does not inject and holds the
BackupJob at `Ready=False PerconaServerMongoDBLegacyRender` (failing it
at the deadline), so a wedged legacy revision never has its nightly dump
and oplog redirected into the shared bucket. The backup snapshot falls
back to the strategy coordinates when the operator status echo is empty;
that fallback is gated on the CR's own `percona.com/delete-backup`
finalizer (the flow the CR was minted on), not on the app flag re-read
at completion, so a legacy backup never records the platform bucket and
a flag flipped mid-dump does not change the artifact. The restore path
rebuilds `backupSource` from it (the `cozy-backups-creds` reference is
projected into the restore namespace before dispatch).
- **Retention/cleanup**: on the `useSystemBucket` flow the driver owns
the shared-bucket archive (the chart's psmdb `tasks[].keep` retention is
gated out, and an on-demand backup is not covered by task retention). It
stamps the operator's `percona.com/delete-backup` finalizer on each
backup it mints **against a storage that carries the platform
coordinates** — the strategy's bucket and `credentialsSecret`, read back
from the live cluster after the injection — never on the flag alone: a
strategy without an `s3` block, or a storage that points anywhere else,
holds the job (`MongoDBStrategyHasNoS3` /
`PerconaServerMongoDBStorageForeign`) instead of minting a finalized
dump against a bucket the platform does not own. On Cozystack `Backup`
deletion (by hand today; by platform retention once it lands, since a
`Plan` only creates BackupJobs and never deletes a `Backup`) the driver
deletes the operator `PerconaServerMongoDBBackup` CR so that finalizer
frees the object — the finalizer's presence is the ownership marker
cleanup reads back. That reading is unambiguous because cleanup resolves
the CR by the name recorded in `driverMetadata` (the one the driver
minted), not by scanning: the psmdb operator does stamp
`percona.com/delete-backup` on its own `cron-*` task-minted backups, but
those are never in scope. A legacy backup keeps the pre-existing
hands-off contract, decided from the snapshot before any read of the
operator CR. A terminating namespace or the
`backups.cozystack.io/skip-artifact-cleanup: "true"` annotation strips
the driver's finalizer and releases; a `Get` of the operator CR that
fails with anything other than NotFound/NoMatchError still requeues, and
the operator's own `release-lock` finalizer can hold the CR while the
operator is down, so those two are what can keep a namespace in
`Terminating`. `schedule` and `retentionPolicy` are inert on this flow
(there is no engine-side retention and a `Plan` has no retention field
yet); platform-level retention is tracked in
[#4192](#4192) and teardown
GC of orphaned backups in
[#3966](#3966), so until
those land bound `cozy-backups` with an S3 lifecycle policy.
- **Bounds on the waits**: the injection race with the operator's
cluster cache (an `unable to get storage` error on a freshly injected
storage) is retried by deleting the errored CR and minting a fresh one,
with the retry count kept on the BackupJob
(`backups.cozystack.io/mongodb-storage-race-retries`) because the
deleted CR is reaped before the next poll; after three retries the
operator's error is terminal. A system-bucket dump that has started
streaming is not cancelled on the 30-minute deadline, but fails after 24
hours in `running` (`BackupAbandoned` warning naming the operator CR,
which is left in place as the handle to whatever pbm still writes). The
deadline cancel of a not-yet-started CR pins its delete to the
resourceVersion the live read answered for.
- **RBAC**: `patch` on `perconaservermongodbs` for the injection
(server-side apply), and `create`/`delete`/`patch` on
`perconaservermongodbbackups` — `delete` to prune the shared-bucket
archive on `Backup` deletion, `patch` to strip the driver's finalizer
when releasing a wedged cleanup.

Coverage: Go unit tests over `internal/backupcontroller` — the injection
gate keyed off the app flag (legacy cluster never patched) and off the
applied render (a cluster still carrying tasks or pitr is held,
untouched), the SSA-injection precondition with the whole entry
(per-application prefix included) asserted on the cluster, strategy
templating per application, the finalizer refused against a storage
without platform coordinates (strategy without `s3` over a legacy entry,
foreign coordinates on the merged entry), the retry budget driven with
the operator simulated reaping the deleted CR between polls, the restore
snapshot fallback and credential swap across all four legacy/system flow
combinations, the finalizer-owned cleanup with its escape hatches (skip
annotation before any read, `NoMatchError` treated as gone,
terminating-namespace release using the pre-read CR, a failed strip
surfaced as an Event, a legacy CR never deleted, a legacy snapshot
released without a read, a system-bucket snapshot reaching the prune),
the deadline cancelling a not-yet-running operator backup only on a
positive live read (including no live client at all, and a CR that moved
since the read), and the running ceiling. Plus helm-unittest on both
charts (chart skips the creds Secret and omits
`spec.backup.storages`/`tasks`/`pitr` on `useSystemBucket` and keeps
them on the legacy flow; the strategy renders the S3 coordinates and the
`insecureSkipTLSVerify` knob). A live `useSystemBucket` backup+restore
e2e against the platform bucket is a follow-up (needs the system bucket
wired in CI, same status as the Postgres/ClickHouse useSystemBucket
e2e). One item to confirm in e2e: the psmdb operator accepting
`spec.backup.enabled=true` with an empty `storages` between install and
the first BackupJob.

### Screenshots

Not applicable (no UI changes).

### Downstream repositories

The tenant-facing schema surface is one new field,
`spec.backup.useSystemBucket`, on the MongoDB `values.schema.json` / CRD
(regenerated), plus a validation rule on `spec.backup`.
`terraform-provider-cozystack` hand-models MongoDB and may want to
expose it, but it is a new optional boolean that defaults false (no
drift for existing users), and the chart `README.md` regenerates on
release. Leaving every box unticked per the trigger-map guidance so a
maintainer decides on the provider follow-up.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(mongodb): MongoDB backups now support the platform system bucket via backup.useSystemBucket, so the default cozy-default BackupClass works without per-app S3 credentials — the backup driver injects the system-bucket storage onto the cluster at backup time, once the release has rendered the flag. On this flow nothing prunes the shared bucket automatically yet (a Plan carries no retention field; tracked in #4192 and #3966), so an S3 lifecycle policy on cozy-backups is a prerequisite for enabling it, not a suggestion. useSystemBucket requires backup.enabled=true; the chart refuses the combination with backups disabled at render time, and the schema carries the same rule as a marker until the apiserver enforces CEL (#2657). A system-bucket dump still running after 24 hours fails its BackupJob with the operator backup left in place. A MongoDB restore now adopts the target cluster's S3 credentials only for a storage on the same bucket as the archive (it used to adopt them unconditionally); a legacy restore into a target on a different bucket that relied on one key pair covering both must keep or re-create the source release's -s3-creds Secret by name, and is held at Ready=False RestoreCredentialsMissing until it exists.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added opt-in system bucket mode for MongoDB backups.
* Automatically injects platform bucket storage without
application-managed S3 credentials.
* Supports endpoint, region, path prefix, addressing, and TLS settings.
* Manages shared-bucket backup artifacts when system bucket backups are
deleted.

* **Bug Fixes**
* Improved restore credential selection to avoid cross-bucket
mismatches.
  * Preserved backup metadata when storage details are unavailable.

* **Documentation**
* Documented configuration, scheduling, retention, PITR limitations, and
restore guidance.

* **Validation**
* Prevented incompatible bootstrap and system bucket restore
configurations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->