Skip to content

Commit 1d813c9

Browse files
authored
fix(backups): refuse concurrent restores and skip the CNPG purge only for this restore's Cluster (#4754)
## What this PR does Fixes #3322. An in-place Postgres (CNPG) restore decided whether to skip deleting the live `Cluster` by comparing its `creationTimestamp` with the RestoreJob's `StartedAt`. The timestamp says the Cluster is new, not which restore rendered it, so two cases let a restore skip its purge and report `Succeeded` on data from another restore: 1. Two RestoreJobs into the same application at once. The second sees the Cluster the first produced, created after its own `StartedAt`, and skips its purge. 2. The live Cluster is deleted while the restore waits for the WAL archive. The still-active HelmRelease re-renders it from the previous restore's values, again created after `StartedAt`. ### Refuse a restore into an application another restore holds The RestoreJob controller now refuses, for every driver, a RestoreJob that has not started yet when another unfinished RestoreJob in the namespace writes into the same application and goes first. The refused job fails with reason `ConcurrentRestore`, and its message names the job that goes first. It fails instead of waiting, so the application is never restored a second time later, after the first restore, by a job its author may no longer expect to run. A started restore goes before one that has not started. Between two that have not, the older goes first, by creation time and then by name; creation times have one-second precision, so two jobs created in the same second are ordered by name. A running restore is never failed by a newer one. The target application is resolved by `restoreJobTarget`, the same helper the BackupJob hold uses, so a Velero restore into another namespace does not collide with an in-place one. The gate lists RestoreJobs through the manager's API reader, not the informer cache. A job reconciled before the cache holds its peer's `StartedAt` would otherwise see that peer as not started and, on a same-second name tie, proceed alongside it. The controller runs a single worker, so a peer's `StartedAt` written by an earlier reconcile is already on the API server when the next job reads it; `SetupWithManager` now sets `MaxConcurrentReconciles: 1` explicitly, since the guarantee depends on it. ### Skip the CNPG purge only for this restore's Cluster The restore patch already sets `bootstrap.newServerName` to `restoredServerName(cluster, RestoreJob UID)`, a value unique to each RestoreJob. The postgres chart now annotates the `Cluster` with it as `postgres.cozystack.io/restored-server-name`. The annotation is rendered even when the barman plugin is not; the plugin's `serverName` also carries the value, but the plugin renders only with chart-level S3 coordinates. The driver skips the purge only when the annotation names this RestoreJob. A Cluster produced by a concurrent restore, or re-rendered from an earlier restore's values, carries another value and is purged. The mark lives on the rendered Cluster rather than in RestoreJob status. That keeps it valid in the case the skip exists for: the purge succeeded, but the `TargetPurged` status write failed. A UID recorded in status was rejected in the code for exactly that reason. A Cluster rendered by a chart that predates the annotation has none and keeps the creation-time check, so a restore in flight across an upgrade behaves as before. ### Tests - `TestRestoreAhead` covers order, a started job going first, the name tie-break, finished and terminating jobs, other applications, retargeted restores and a Velero restore into another namespace. - `TestReconcileRestore_ConcurrentRestoreIsRefused` checks that the newer job fails with `ConcurrentRestore` and the running one is left alone. - `TestReconcileRestore_ConcurrentRestoreReadsPastTheCache` gives the cache a stale, not-started peer and the API reader the started one, and requires the job to fail with `ConcurrentRestore`. - `TestCNPGClusterFreshlyRecovered` adds both cases from the issue: a Cluster marked by another restore and created after `StartedAt` must be purged. The timestamp-only cases are kept for the unmarked fallback. - `TestReconcileCNPGRestore_RepeatInPlacePurgesStaleRecoveryCluster` gains two reconcile-level cases. One purges a Cluster that another restore marked, even though it was created after `StartedAt`. The other runs a full purge, takes the mark from the `bootstrap.newServerName` the restore patch wrote, drops `TargetPurged` as a lost status write would, and requires a Cluster carrying that mark to survive, although it predates `StartedAt`. That pins the purge decision to the value the chart actually renders. - `packages/apps/postgres/tests/restore_mark_test.yaml` checks that the annotation is rendered with and without the barman plugin, and is absent on a Cluster that was never restored. Each new test was checked against the code it guards: disabling the gate call, listing through the cache, flipping the started-job precedence, ignoring the annotation, or deriving the expected mark from the app name instead of the cluster name each turns one red. `go test ./internal/backupcontroller/...` and `helm unittest packages/apps/postgres` (152 tests) pass. The restore flow was not run against a live cluster. `docs/operations/backup-classes.md` documents the one-restore-at-a-time rule and the annotation. ### Screenshots N/A — no UI changes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff: Go changes in `internal/backupcontroller` (no CRD or API type change), one annotation added to the postgres chart's Cluster (no `values.yaml` or `values.schema.json` change), and a docs paragraph. No package is added or renamed, no platform values, ApplicationDefinition, RestoreJob CRD, metric, label or namespace changes, so no trigger is reached. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): a RestoreJob now fails with reason ConcurrentRestore when another unfinished RestoreJob is already restoring the same application, instead of running alongside it; create it again once the first restore has finished. An in-place Postgres restore now skips deleting the live Cluster only when that Cluster was rendered for this RestoreJob, identified by the new postgres.cozystack.io/restored-server-name annotation, so it no longer reports Succeeded on data from a concurrent or earlier restore. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Restores targeting the same application are now handled one at a time. A competing restore is refused rather than queued, with started restores taking priority. * PostgreSQL now calculates a default `shared_buffers` value when the memory limit is below 512 MiB, while respecting explicitly configured values. * **Bug Fixes** * Recovery clusters are more reliably distinguished from manual recoveries and clusters created by other restores, reducing the risk of purging the wrong cluster. * **Documentation** * Clarified restore ordering and how recovery-cluster cleanup is determined. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 05cfb63 + 6a75883 commit 1d813c9

8 files changed

Lines changed: 697 additions & 104 deletions

File tree

‎docs/operations/backup-classes.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -375,6 +375,8 @@ spec:
375375
name: orders-db
376376
```
377377

378+
An application takes one restore at a time. A RestoreJob that targets an application another unfinished RestoreJob is already restoring fails with reason `ConcurrentRestore` before it changes anything, and its message names the job that goes first. It is not queued: once that job has finished, create a new RestoreJob if the restore is still wanted. A restore that has started always goes first. Between two that have not, the older one runs; creation times have one-second precision, so of two created in the same second, the one whose name sorts first runs.
379+
378380
## Point-in-time recovery (PostgreSQL)
379381

380382
A `RestoreJob` restores a `Postgres` application from a `Backup`. Omit `spec.options.recoveryTime` to recover to the latest point in the WAL archive; set it (RFC3339) to recover the database to an exact instant — a point-in-time recovery (PITR). The CNPG barman-cloud plugin restores a base backup and replays archived WAL from it: up to `recoveryTime` when one is set, so the restored cluster reflects the database as of that instant and later writes are absent, and to the end of the archive otherwise.
@@ -441,7 +443,7 @@ Recovery follows `recovery_target_timeline: latest`, which CNPG leaves as Postgr
441443

442444
### Idempotency under GitOps
443445

444-
An in-progress restore is safe to reconcile. The driver purges the target `Cluster` + PVCs exactly once per RestoreJob (guarded by the `TargetPurged` condition and a freshly-recovered check), suspends the target's HelmRelease across the purge so Flux cannot race the bootstrap swap, and resumes it once the recovery cluster is rendered. A Flux reconcile (or a controller restart) mid-restore therefore re-attaches to the recovering cluster rather than deleting it and starting over.
446+
An in-progress restore is safe to reconcile. The driver purges the target `Cluster` + PVCs exactly once per RestoreJob (guarded by the `TargetPurged` condition and by the `postgres.cozystack.io/restored-server-name` annotation, which identifies the RestoreJob whose purge produced the live recovery `Cluster`), suspends the target's HelmRelease across the purge so Flux cannot race the bootstrap swap, and resumes it once the recovery cluster is rendered. A Flux reconcile (or a controller restart) mid-restore therefore re-attaches to the recovering cluster rather than deleting it and starting over.
445447

446448
## Kafka: topic metadata only
447449

‎internal/backupcontroller/cnpgstrategy_controller.go‎

Lines changed: 61 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,10 @@ const (
6969
cnpgS3SecretRefKey = "cnpg.io/s3-secret-ref"
7070
cnpgMajorVersionKey = "cnpg.io/major-version"
7171

72+
// Annotation the postgres chart renders on the Cluster from
73+
// bootstrap.newServerName.
74+
cnpgRestoredServerNameAnnotation = "postgres.cozystack.io/restored-server-name"
75+
7276
// Polling cadence for the CNPG backup/restore lifecycle. Mirrors the
7377
// Velero strategy's defaults so behaviour is uniform across drivers.
7478
cnpgPollInterval = 5 * time.Second
@@ -521,40 +525,43 @@ func cnpgPurgeNeeded(purgedCondition, liveClusterFreshlyRecovered bool) bool {
521525

522526
// cnpgClusterFreshlyRecovered reports whether a live cnpg.io Cluster that
523527
// carries spec.bootstrap.recovery was produced by THIS RestoreJob's own purge
524-
// + chart re-render (its creationTimestamp is strictly after the job's
525-
// StartedAt) rather than being left over from an earlier, already-completed
526-
// restore.
528+
// + chart re-render rather than rendered for some other restore.
527529
//
528-
// Only the former is safe to skip re-purging. A leftover recovery Cluster from
529-
// a prior restore predates StartedAt and still holds the prior data, so it must
530-
// be purged for the new restore to re-bootstrap from the backup. The fresh
531-
// Cluster in the status-write-race case is always created after StartedAt (the
532-
// job sets StartedAt on its first reconcile, long before it purges and the
533-
// chart re-renders), so the timestamp comparison cleanly separates the two.
530+
// Only the former is safe to skip re-purging. Any other recovery Cluster holds
531+
// another restore's data and must be purged for this restore to re-bootstrap
532+
// from its backup. That includes a Cluster a concurrent RestoreJob produced,
533+
// and one the chart re-renders from an earlier restore's values after being
534+
// deleted while this restore waits for the WAL archive: both are created after
535+
// this job's StartedAt, so a timestamp alone cannot tell them apart.
534536
//
535-
// The comparison is strict (created > started), so an exact tie resolves to
536-
// "not fresh" and the caller purges. That matches the conservative default
537-
// below: the only classification that must never be wrong is calling a stale
538-
// leftover "fresh" (which reintroduces the silent no-op), and a fresh Cluster
539-
// is always created well after StartedAt (see above), never exactly at it.
537+
// The chart annotates the Cluster with bootstrap.newServerName, which the
538+
// restore patch sets to restoredServerName(cluster, RestoreJob UID). The value
539+
// lives on the object the chart renders, not in RestoreJob status, so it
540+
// survives the failed status write the skip exists for, and it names the
541+
// restore that rendered the Cluster. When the annotation is present, the
542+
// Cluster is fresh only if it carries wantMark.
540543
//
541-
// An identity-based alternative was considered - recording the purged Cluster's
542-
// UID on the TargetPurged condition and comparing UIDs on later reconciles -
543-
// which is immune to clock skew. It was rejected because it leans on the same
544-
// status write that the status-write-race path (the whole reason this skip
545-
// exists) assumes can fail, so it cannot cover that case; the timestamp
546-
// comparison needs no extra persisted state and the fresh-vs-stale gap
547-
// (a full purge + re-render cycle) always dwarfs any plausible control-plane
548-
// clock skew.
544+
// A Cluster without the annotation was rendered either by a chart that
545+
// predates it or from values with no bootstrap.newServerName (a manually
546+
// configured recovery). It is fresh only if appMark, the application's current
547+
// bootstrap.newServerName, is wantMark - this restore has patched the values -
548+
// and it was created strictly after StartedAt, which separates a Cluster from
549+
// an earlier completed restore from this restore's own re-render.
549550
//
550551
// Returns false when freshness cannot be determined (no recovery bootstrap, or
551-
// a missing timestamp): the caller then proceeds to purge, which is the safe
552-
// default for any pre-existing, non-fresh Cluster.
553-
func cnpgClusterFreshlyRecovered(hasRecovery bool, clusterCreatedAt, restoreStartedAt *metav1.Time) bool {
554-
if !hasRecovery || clusterCreatedAt == nil || restoreStartedAt == nil {
552+
// a missing timestamp on the fallback path): the caller then purges, which is
553+
// the safe default.
554+
func cnpgClusterFreshlyRecovered(state recoveryClusterState, wantMark, appMark string, restoreStartedAt *metav1.Time) bool {
555+
if !state.hasRecovery {
556+
return false
557+
}
558+
if state.restoreMark != "" {
559+
return state.restoreMark == wantMark
560+
}
561+
if appMark != wantMark || state.createdAt == nil || restoreStartedAt == nil {
555562
return false
556563
}
557-
return clusterCreatedAt.After(restoreStartedAt.Time)
564+
return state.createdAt.After(restoreStartedAt.Time)
558565
}
559566

560567
// cnpgClusterGVR addresses Clusters on the dynamic client, for reads that must
@@ -1021,17 +1028,16 @@ func (r *RestoreJobReconciler) reconcileCNPGRestore(ctx context.Context, restore
10211028
// check the live Cluster for bootstrap.recovery: if present, the chart
10221029
// has already re-rendered after a previous purge, and we must NOT delete
10231030
// it again.
1024-
hasRecovery, clusterCreatedAt, err := r.recoveryBootstrapClusterState(ctx, target.Namespace, clusterName)
1031+
clusterState, err := r.recoveryBootstrapClusterState(ctx, target.Namespace, clusterName)
10251032
if err != nil {
10261033
return ctrl.Result{}, err
10271034
}
10281035
purgedCondition := apimeta.IsStatusConditionTrue(restoreJob.Status.Conditions, restoreCondTargetPurged)
10291036
// Only skip the destructive purge for a recovery Cluster THIS restore just
1030-
// produced (created at/after StartedAt). A recovery Cluster left over from
1031-
// an earlier completed restore predates StartedAt and still holds the old
1032-
// data, so it must be purged - otherwise a repeat in-place restore silently
1033-
// no-ops.
1034-
freshlyRecovered := cnpgClusterFreshlyRecovered(hasRecovery, clusterCreatedAt, restoreJob.Status.StartedAt)
1037+
// produced. A recovery Cluster rendered for any other restore still holds
1038+
// that restore's data, so it must be purged - otherwise a repeat in-place
1039+
// restore silently no-ops.
1040+
freshlyRecovered := cnpgClusterFreshlyRecovered(clusterState, restoredServerName(clusterName, restoreJob.UID), targetApp.Spec.Bootstrap.NewServerName, restoreJob.Status.StartedAt)
10351041
if cnpgPurgeNeeded(purgedCondition, freshlyRecovered) {
10361042
// A physical restore only opens on the major version that wrote the
10371043
// data files. Across majors the recovery pods loop on "database files
@@ -1206,7 +1212,7 @@ func (r *RestoreJobReconciler) reconcileCNPGRestore(ctx context.Context, restore
12061212
// Getting the order wrong is not just a cosmetic mislabel - a false Failed
12071213
// can be resubmitted, and the next RestoreJob's purge guard would then
12081214
// delete the already-recovered Cluster + PVCs to start over.
1209-
if hasRecovery {
1215+
if clusterState.hasRecovery {
12101216
healthy, herr := r.cnpgClusterHealthy(ctx, target.Namespace, clusterName)
12111217
if herr != nil {
12121218
return ctrl.Result{}, herr
@@ -2070,22 +2076,37 @@ func (r *RestoreJobReconciler) readPodContainerLog(ctx context.Context, namespac
20702076
// drops the change and the cluster ends up with the original initdb spec.
20712077
// Holding here forces the caller to requeue until the old CR is fully GC'd
20722078
// and the chart re-creates a fresh one.
2073-
func (r *RestoreJobReconciler) recoveryBootstrapClusterState(ctx context.Context, namespace, clusterName string) (hasRecovery bool, createdAt *metav1.Time, err error) {
2079+
func (r *RestoreJobReconciler) recoveryBootstrapClusterState(ctx context.Context, namespace, clusterName string) (recoveryClusterState, error) {
20742080
cluster := &cnpgtypes.Cluster{}
20752081
if err := r.Get(ctx, types.NamespacedName{Namespace: namespace, Name: clusterName}, cluster); err != nil {
20762082
if apierrors.IsNotFound(err) {
2077-
return false, nil, nil
2083+
return recoveryClusterState{}, nil
20782084
}
2079-
return false, nil, err
2085+
return recoveryClusterState{}, err
20802086
}
20812087
if !cluster.DeletionTimestamp.IsZero() {
2082-
return false, nil, nil
2088+
return recoveryClusterState{}, nil
20832089
}
20842090
if cluster.Spec.Bootstrap == nil || cluster.Spec.Bootstrap.Recovery == nil {
2085-
return false, nil, nil
2091+
return recoveryClusterState{}, nil
20862092
}
20872093
created := cluster.CreationTimestamp
2088-
return true, &created, nil
2094+
return recoveryClusterState{
2095+
hasRecovery: true,
2096+
createdAt: &created,
2097+
restoreMark: cluster.Annotations[cnpgRestoredServerNameAnnotation],
2098+
}, nil
2099+
}
2100+
2101+
// recoveryClusterState is what the purge decision reads off the live Cluster.
2102+
// The zero value is a Cluster that is absent, terminating, or not bootstrapped
2103+
// from a recovery.
2104+
type recoveryClusterState struct {
2105+
hasRecovery bool
2106+
createdAt *metav1.Time
2107+
// restoreMark is cnpgRestoredServerNameAnnotation, or "" on a Cluster the
2108+
// chart rendered before it set the annotation.
2109+
restoreMark string
20892110
}
20902111

20912112
// ---------------------------------------------------------------------------

0 commit comments

Comments
 (0)