Skip to content

Commit 7212e9b

Browse files
authored
fix(backups/postgres): purge stale recovery cluster on repeat in-place restore (#3318)
## What this PR does Fixes a silent data-integrity failure in the CNPG backup driver: a repeat in-place PostgreSQL `RestoreJob` into a target that was already restored once reported `Succeeded` but did nothing. The PVC was untouched (same uid), no `*-full-recovery` pods appeared, and the database kept its pre-restore contents. Root cause: the restore patch sets `spec.bootstrap.enabled=true` on the Postgres app and never resets it, so the chart keeps rendering `spec.bootstrap.recovery` on the live cnpg.io Cluster. On a second `RestoreJob` the re-purge guard (`cnpgPurgeNeeded`) saw that leftover `bootstrap.recovery` and treated it as "an earlier reconcile of this restore already purged", skipping the destructive Cluster + PVC purge. That guard was meant only to avoid re-purging a freshly-recovered Cluster when the post-purge status write races, but it could not tell that case apart from a Cluster left over by a previous completed restore. The fix distinguishes the two by the live Cluster's `creationTimestamp` relative to the `RestoreJob`'s `StartedAt`: - created after `StartedAt`: freshly recovered by this restore's own purge, so it is skipped (unchanged behaviour for the status-write-race path); - created at or before `StartedAt`: leftover from an earlier completed restore, so it is purged and the restore re-bootstraps from the backup. Design note: an identity-based alternative (record the purged Cluster's UID on the `TargetPurged` condition and compare UIDs on later reconciles) was considered and rejected. It is immune to clock skew but leans on the same status write that the status-write-race path assumes can fail, so it cannot cover the exact case the skip exists for. The timestamp comparison needs no extra persisted state, and the fresh-vs-stale gap (a full purge plus chart re-render cycle) always dwarfs any plausible control-plane clock skew. The comparison is strict, so an exact-second tie resolves to purge (the conservative direction). ### Tests - New `cnpgClusterFreshlyRecovered` unit test: fresh / stale / boundary / missing-timestamp cases. - `TestCNPGPurgeNeeded` extended with the repeat-restore (stale recovery cluster) case that reproduced the bug. - `recoveryBootstrapClusterState` test updated to assert the returned `creationTimestamp`. - New reconcile-level regression test driving `reconcileCNPGRestore` end to end: a stale leftover recovery Cluster is purged, a freshly-recovered Cluster is not re-purged. Bug report: #3311 ### Screenshots Not applicable (no UI changes). ### Downstream repositories The diff touches only `internal/backupcontroller` (operator reconcile logic); no API types, CRDs, chart values, or docs change, so no downstream repository is affected. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(postgres): a repeat in-place restore no longer silently succeeds without restoring; a recovery cluster left over from an earlier completed restore is now purged so the restore re-bootstraps from the backup ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved CNPG restore handling for repeat in-place restores. - Stale recovery clusters are now correctly purged instead of being mistaken for newly recovered clusters. - Newly recovered clusters are protected from unnecessary destructive cleanup. - Restore behavior is more reliable when recovery clusters are left over from earlier attempts. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents f6a03e9 + a8494d8 commit 7212e9b

2 files changed

Lines changed: 338 additions & 46 deletions

File tree

‎internal/backupcontroller/cnpgstrategy_controller.go‎

Lines changed: 76 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -314,16 +314,62 @@ func cnpgBackupDeadlineExceeded(startedAt *metav1.Time) bool {
314314
// destructive step:
315315
// 1. The RestoreJob already records that we've purged
316316
// (restoreCondTargetPurged=True). Normal idempotent path.
317-
// 2. The live Cluster already has spec.bootstrap.recovery populated. This
318-
// only happens when an earlier reconcile purged successfully but the
317+
// 2. The live Cluster is a freshly-recovered one that THIS restore's own
318+
// purge + chart re-render produced (see cnpgClusterFreshlyRecovered).
319+
// This only happens when an earlier reconcile purged successfully but the
319320
// status-condition write failed; the chart has since re-rendered the
320321
// Cluster with our restore-shaped values. Re-purging here would delete
321322
// the Cluster CNPG is actively bootstrapping from S3.
322-
func cnpgPurgeNeeded(purgedCondition, liveClusterHasRecovery bool) bool {
323+
//
324+
// A live Cluster that carries spec.bootstrap.recovery but predates this
325+
// RestoreJob is NOT a skip signal: it is leftover from an earlier, already-
326+
// completed restore and still holds the old data. Skipping the purge on it
327+
// (the previous behaviour, which keyed only on "has recovery bootstrap") made
328+
// a repeat in-place restore a silent no-op - the job reported Succeeded while
329+
// the PVC, disk, and data were never touched.
330+
func cnpgPurgeNeeded(purgedCondition, liveClusterFreshlyRecovered bool) bool {
323331
if purgedCondition {
324332
return false
325333
}
326-
return !liveClusterHasRecovery
334+
return !liveClusterFreshlyRecovered
335+
}
336+
337+
// cnpgClusterFreshlyRecovered reports whether a live cnpg.io Cluster that
338+
// carries spec.bootstrap.recovery was produced by THIS RestoreJob's own purge
339+
// + chart re-render (its creationTimestamp is strictly after the job's
340+
// StartedAt) rather than being left over from an earlier, already-completed
341+
// restore.
342+
//
343+
// Only the former is safe to skip re-purging. A leftover recovery Cluster from
344+
// a prior restore predates StartedAt and still holds the prior data, so it must
345+
// be purged for the new restore to re-bootstrap from the backup. The fresh
346+
// Cluster in the status-write-race case is always created after StartedAt (the
347+
// job sets StartedAt on its first reconcile, long before it purges and the
348+
// chart re-renders), so the timestamp comparison cleanly separates the two.
349+
//
350+
// The comparison is strict (created > started), so an exact tie resolves to
351+
// "not fresh" and the caller purges. That matches the conservative default
352+
// below: the only classification that must never be wrong is calling a stale
353+
// leftover "fresh" (which reintroduces the silent no-op), and a fresh Cluster
354+
// is always created well after StartedAt (see above), never exactly at it.
355+
//
356+
// An identity-based alternative was considered - recording the purged Cluster's
357+
// UID on the TargetPurged condition and comparing UIDs on later reconciles -
358+
// which is immune to clock skew. It was rejected because it leans on the same
359+
// status write that the status-write-race path (the whole reason this skip
360+
// exists) assumes can fail, so it cannot cover that case; the timestamp
361+
// comparison needs no extra persisted state and the fresh-vs-stale gap
362+
// (a full purge + re-render cycle) always dwarfs any plausible control-plane
363+
// clock skew.
364+
//
365+
// Returns false when freshness cannot be determined (no recovery bootstrap, or
366+
// a missing timestamp): the caller then proceeds to purge, which is the safe
367+
// default for any pre-existing, non-fresh Cluster.
368+
func cnpgClusterFreshlyRecovered(hasRecovery bool, clusterCreatedAt, restoreStartedAt *metav1.Time) bool {
369+
if !hasRecovery || clusterCreatedAt == nil || restoreStartedAt == nil {
370+
return false
371+
}
372+
return clusterCreatedAt.After(restoreStartedAt.Time)
327373
}
328374

329375
// applyClusterBarmanObjectStore SSA-patches the live CNPG Cluster's
@@ -614,12 +660,18 @@ func (r *RestoreJobReconciler) reconcileCNPGRestore(ctx context.Context, restore
614660
// check the live Cluster for bootstrap.recovery: if present, the chart
615661
// has already re-rendered after a previous purge, and we must NOT delete
616662
// it again.
617-
hasRecovery, err := r.clusterHasRecoveryBootstrap(ctx, target.Namespace, clusterName)
663+
hasRecovery, clusterCreatedAt, err := r.recoveryBootstrapClusterState(ctx, target.Namespace, clusterName)
618664
if err != nil {
619665
return ctrl.Result{}, err
620666
}
621667
purgedCondition := apimeta.IsStatusConditionTrue(restoreJob.Status.Conditions, restoreCondTargetPurged)
622-
if cnpgPurgeNeeded(purgedCondition, hasRecovery) {
668+
// Only skip the destructive purge for a recovery Cluster THIS restore just
669+
// produced (created at/after StartedAt). A recovery Cluster left over from
670+
// an earlier completed restore predates StartedAt and still holds the old
671+
// data, so it must be purged - otherwise a repeat in-place restore silently
672+
// no-ops.
673+
freshlyRecovered := cnpgClusterFreshlyRecovered(hasRecovery, clusterCreatedAt, restoreJob.Status.StartedAt)
674+
if cnpgPurgeNeeded(purgedCondition, freshlyRecovered) {
623675
// Gate the destructive flow on the source cluster having shipped
624676
// the backup's required WALs to object storage. archive_command runs
625677
// on the source primary; once we delete the Cluster + PVCs, any
@@ -1126,11 +1178,15 @@ func (r *RestoreJobReconciler) cnpgClusterHealthy(ctx context.Context, namespace
11261178
return cluster.Status.Phase == cnpgClusterHealthyPhase, nil
11271179
}
11281180

1129-
// clusterHasRecoveryBootstrap returns true when the live cnpg.io Cluster's
1130-
// spec.bootstrap.recovery is populated - the signal that the chart has
1131-
// re-rendered with our restore-shaped values and the operator is using the
1132-
// recovery bootstrap path. Treats a missing Cluster as "not yet" rather
1133-
// than an error so the caller can keep polling while HelmRelease catches up.
1181+
// recoveryBootstrapClusterState fetches the live cnpg.io Cluster and reports
1182+
// whether its spec.bootstrap.recovery is populated - the signal that the chart
1183+
// has re-rendered with our restore-shaped values and the operator is using the
1184+
// recovery bootstrap path - together with the Cluster's creationTimestamp. The
1185+
// caller pairs the timestamp with the RestoreJob's StartedAt (via
1186+
// cnpgClusterFreshlyRecovered) to tell a Cluster this restore just produced
1187+
// apart from one left over by an earlier completed restore. Treats a missing
1188+
// Cluster as "not yet" rather than an error so the caller can keep polling
1189+
// while HelmRelease catches up.
11341190
//
11351191
// A Cluster carrying DeletionTimestamp is also treated as "not yet": that is
11361192
// the in-flight purge case, where r.Delete has fired but cnpg.io's
@@ -1143,18 +1199,22 @@ func (r *RestoreJobReconciler) cnpgClusterHealthy(ctx context.Context, namespace
11431199
// drops the change and the cluster ends up with the original initdb spec.
11441200
// Holding here forces the caller to requeue until the old CR is fully GC'd
11451201
// and the chart re-creates a fresh one.
1146-
func (r *RestoreJobReconciler) clusterHasRecoveryBootstrap(ctx context.Context, namespace, clusterName string) (bool, error) {
1202+
func (r *RestoreJobReconciler) recoveryBootstrapClusterState(ctx context.Context, namespace, clusterName string) (hasRecovery bool, createdAt *metav1.Time, err error) {
11471203
cluster := &cnpgtypes.Cluster{}
11481204
if err := r.Get(ctx, types.NamespacedName{Namespace: namespace, Name: clusterName}, cluster); err != nil {
11491205
if apierrors.IsNotFound(err) {
1150-
return false, nil
1206+
return false, nil, nil
11511207
}
1152-
return false, err
1208+
return false, nil, err
11531209
}
11541210
if !cluster.DeletionTimestamp.IsZero() {
1155-
return false, nil
1211+
return false, nil, nil
1212+
}
1213+
if cluster.Spec.Bootstrap == nil || cluster.Spec.Bootstrap.Recovery == nil {
1214+
return false, nil, nil
11561215
}
1157-
return cluster.Spec.Bootstrap != nil && cluster.Spec.Bootstrap.Recovery != nil, nil
1216+
created := cluster.CreationTimestamp
1217+
return true, &created, nil
11581218
}
11591219

11601220
// ---------------------------------------------------------------------------

0 commit comments

Comments
 (0)