Skip to content

Commit b13e276

Browse files
androndoclaude
andcommitted
fix(backups): own driver-created ObjectStore + correct stale barman references
Review follow-ups (#3300): - Lifecycle/leak: applyClusterPluginBackup now sets an ownerReference from the driver-created barmancloud.cnpg.io/ObjectStore to its cnpg.io/Cluster, so Kubernetes garbage-collects it when the Cluster is deleted (tenant app teardown or the RestoreJob purge). In the platform useSystemBucket flow the chart does not render this ObjectStore, so previously it was orphaned in the tenant namespace. Drop the now-accurate-only 'delete' verb on objectstores from the driver RBAC (GC handles removal; the driver never deletes it) and correct the RBAC comment that falsely claimed the purge deletes it. - Diagnostics/comments: the RestoreJob WAL-archive-deadline failure message and several doc comments still told operators to check spec.backup.barmanObjectStore / externalClusters[].barmanObjectStore, fields no longer emitted after the migration. Point them at the barman-cloud plugin (spec.plugins + ObjectStore, externalClusters[].plugin) instead so they don't mislead during an incident. - Test: TestApplyClusterPluginBackup_PatchesExistingCluster now asserts the ObjectStore carries an ownerReference to the Cluster (kind/name/uid). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Signed-off-by: Andrey Kolkov <[email protected]>
1 parent c158f95 commit b13e276

3 files changed

Lines changed: 42 additions & 14 deletions

File tree

‎internal/backupcontroller/cnpgstrategy_controller.go‎

Lines changed: 27 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -84,9 +84,9 @@ const (
8484
// Wall-clock cap on how long the WALArchiveReady gate can stay False
8585
// before the RestoreJob is marked Failed. The gate fires before the
8686
// destructive purge to confirm the backup's endWal is in object
87-
// storage; if the source's lastArchivedWAL never advances, archive_-
88-
// command on the source is broken (no barmanObjectStore on the
89-
// Cluster, S3 outage, bad credentials, etc.) - waiting longer won't
87+
// storage; if the source's lastArchivedWAL never advances, WAL
88+
// archiving on the source is broken (no barman-cloud plugin wired on
89+
// the Cluster, S3 outage, bad credentials, etc.) - waiting longer won't
9090
// fix it. This is independent of cnpgDefaultRestoreDeadline because
9191
// archive lag and recovery time scale with completely different
9292
// inputs: the gate clears in seconds for any healthy cluster
@@ -359,6 +359,19 @@ func (r *BackupJobReconciler) applyClusterPluginBackup(ctx context.Context, name
359359
// same s3://.../<serverName>/ path the native barmanObjectStore used.
360360
objStoreName := clusterName
361361
objStore := newObjectStorePatch(namespace, objStoreName)
362+
// Own the ObjectStore by the live Cluster so Kubernetes garbage-collects it
363+
// when the Cluster is deleted (tenant app teardown, or the RestoreJob purge).
364+
// The platform flow does not chart-render this ObjectStore, so without an
365+
// owner reference it would otherwise be orphaned in the tenant namespace.
366+
// Owner and owned are in the same namespace, as GC requires.
367+
controller := true
368+
objStore.OwnerReferences = []metav1.OwnerReference{{
369+
APIVersion: cnpgtypes.GroupVersion.String(),
370+
Kind: "Cluster",
371+
Name: existing.Name,
372+
UID: existing.UID,
373+
Controller: &controller,
374+
}}
362375
objStore.Spec = cnpgtypes.ObjectStoreSpec{
363376
Configuration: *buildBarmanObjectStore(t.BarmanObjectStore, ""),
364377
RetentionPolicy: t.BarmanObjectStore.RetentionPolicy,
@@ -657,7 +670,7 @@ func (r *RestoreJobReconciler) reconcileCNPGRestore(ctx context.Context, restore
657670
// bootstrap.initdb -> bootstrap.recovery swap with "Only one
658671
// bootstrap". helm-controller then drops into an UpgradeFailed /
659672
// RollbackFailed loop whose rollbacks strip the SSA-applied
660-
// spec.backup.barmanObjectStore - which stops archive_command
673+
// spec.plugins (and its ObjectStore) - which stops WAL archiving
661674
// and pins lastArchivedWAL at "" forever. By gating before the
662675
// patch, the HelmRelease stays untouched while we wait, the
663676
// source primary keeps archiving, and the chart's failure-loop
@@ -689,7 +702,7 @@ func (r *RestoreJobReconciler) reconcileCNPGRestore(ctx context.Context, restore
689702
time.Since(cond.LastTransitionTime.Time) > walDeadline {
690703
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf(
691704
"WAL archive did not catch up to the backup endWal within %s: %s "+
692-
"(check that the source cnpg.io/Cluster has spec.backup.barmanObjectStore set and archive_command is shipping WALs to object storage; override via spec.options.walArchiveTimeoutSeconds)",
705+
"(check that the source cnpg.io/Cluster has the barman-cloud plugin wired - spec.plugins referencing an ObjectStore - and is shipping WALs to object storage; override via spec.options.walArchiveTimeoutSeconds)",
693706
walDeadline, walMessage))
694707
}
695708
return ctrl.Result{RequeueAfter: cnpgPollInterval}, nil
@@ -835,11 +848,13 @@ func (r *RestoreJobReconciler) resolveCNPGRestoreTarget(restoreJob *backupsv1alp
835848
// patchPostgresAppForRestore writes the restore-related fields into the
836849
// target Postgres app instance spec. The chart already exposes these knobs;
837850
// once the HelmRelease re-renders, the cnpg.io Cluster picks up
838-
// bootstrap.recovery and externalClusters[].barmanObjectStore.
851+
// bootstrap.recovery and externalClusters[].plugin referencing the
852+
// chart-rendered recovery ObjectStore.
839853
//
840854
// Credentials are forwarded as a Secret reference (spec.backup.s3CredentialsSecret),
841855
// not as cleartext keys. The controller never reads the Secret itself; the
842-
// chart wires the named Secret straight into barmanObjectStore.s3Credentials.
856+
// chart wires the named Secret into the recovery ObjectStore's
857+
// spec.configuration.s3Credentials.
843858
// This keeps S3 access keys out of the Postgres CR .spec, etcd, audit logs,
844859
// and any tenant-readable copies.
845860
//
@@ -902,10 +917,11 @@ func buildPostgresAppRestorePatch(
902917
SecretAccessKeyKey: credsRef.SecretAccessKeyKey,
903918
}
904919
}
905-
// endpointCA flows into both the chart's spec.backup.barmanObjectStore
906-
// and externalClusters[].barmanObjectStore. The recovery path
907-
// specifically needs it - without a trusted CA the cnpg-instance-manager
908-
// panics in InitInfo.loadBackup when it can't verify the seaweedfs cert.
920+
// endpointCA flows into both the chart's backup ObjectStore
921+
// (spec.configuration.endpointCA) and the recovery ObjectStore referenced
922+
// from externalClusters[].plugin. The recovery path specifically needs it -
923+
// without a trusted CA the cnpg-instance-manager panics in
924+
// InitInfo.loadBackup when it can't verify the seaweedfs cert.
909925
patched.Spec.Backup.EndpointCA = postgresapp.EndpointCA{}
910926
if caRef != nil && caRef.SecretRef.Name != "" {
911927
patched.Spec.Backup.EndpointCA = postgresapp.EndpointCA{

‎internal/backupcontroller/cnpgstrategy_controller_test.go‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1277,7 +1277,7 @@ func testCNPGScheme(t *testing.T) *runtime.Scheme {
12771277
// parameter, never inside the ObjectStore configuration (the plugin forbids it).
12781278
func TestApplyClusterPluginBackup_PatchesExistingCluster(t *testing.T) {
12791279
cluster := &cnpgtypes.Cluster{
1280-
ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "postgres-app"},
1280+
ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "postgres-app", UID: "cluster-uid-123"},
12811281
}
12821282
c := newCNPGStrategyTestClient(t, cluster)
12831283
r := &BackupJobReconciler{Client: c}
@@ -1303,6 +1303,16 @@ func TestApplyClusterPluginBackup_PatchesExistingCluster(t *testing.T) {
13031303
if store.Spec.RetentionPolicy != "30d" {
13041304
t.Errorf("ObjectStore retentionPolicy: got %q", store.Spec.RetentionPolicy)
13051305
}
1306+
// The ObjectStore must be owner-referenced to the Cluster so Kubernetes GC
1307+
// removes it when the Cluster is deleted (no orphan in the platform flow,
1308+
// where the chart does not render this ObjectStore).
1309+
if len(store.OwnerReferences) != 1 {
1310+
t.Fatalf("expected exactly one ownerReference on the ObjectStore, got %+v", store.OwnerReferences)
1311+
}
1312+
own := store.OwnerReferences[0]
1313+
if own.Kind != "Cluster" || own.Name != "postgres-app" || own.UID != "cluster-uid-123" {
1314+
t.Errorf("ObjectStore ownerReference: got kind=%q name=%q uid=%q, want Cluster/postgres-app/cluster-uid-123", own.Kind, own.Name, own.UID)
1315+
}
13061316
if store.Spec.Configuration.ServerName != "" {
13071317
t.Errorf("ObjectStore configuration must not carry serverName (plugin forbids it); got %q", store.Spec.Configuration.ServerName)
13081318
}

‎packages/system/backupstrategy-controller/templates/rbac.yaml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -97,10 +97,12 @@ rules:
9797
# barman-cloud plugin ObjectStore CRs: the CNPG driver SSA-applies one per
9898
# platform-managed Cluster to carry the S3/barman configuration that native
9999
# spec.backup.barmanObjectStore used to hold (deprecated in CNPG 1.27, removed
100-
# in 1.29). Delete lets the RestoreJob purge clean them up alongside the Cluster.
100+
# in 1.29). The driver owner-references each ObjectStore to its Cluster, so
101+
# Kubernetes garbage-collects it when the Cluster is deleted (app teardown or
102+
# the RestoreJob purge) — the driver never deletes it itself, hence no delete verb.
101103
- apiGroups: ["barmancloud.cnpg.io"]
102104
resources: ["objectstores"]
103-
verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
105+
verbs: ["get", "list", "watch", "create", "patch", "update"]
104106
# Secrets: the BackupJob/RestoreJob reconcilers project the platform-managed
105107
# S3 credentials Secret from the backupStorage.namespace (tenant-root by
106108
# default) into each tenant namespace as cozy-backups-creds. The verbs are

0 commit comments

Comments
 (0)