Skip to content

Commit 6fba946

Browse files
committed
fix(backupstrategy-controller): reclaim Bucket backups, bound the mirror Job, wire its e2e
Address the round-2 review on the S3 Bucket backup strategy. - e2e wiring: the backup round-trip lived in hack/e2e-chainsaw/bucket-backup/, which no Test Impact Analysis rule selected, so a scoped edit to the harness ran the bucket app-create suite instead and the round-trip never ran under TIA. Route examples/backups/<app>/ to an "<app>-backup" suite when one exists (select-e2e.sh), map bucket-backup to the bucket-application source in both mapping tables so it round-trips (select-install.sh + src_to_suites), and pin it with a bats case. - reclaim on delete: deleting a Backup left its mirrored objects orphaned in cozy-backups. Add an `s3-mirror --mode=delete` that purges a repo prefix, a cleanupBucketBackup finalizer branch that runs it as a one-shot Job and waits (failing open on a terminating namespace / missing strategy / unprojectable credentials, with the skip-artifact-cleanup escape hatch), mirroring Redis. - mirror Job deadline: the mirror Pod set no activeDeadlineSeconds, so a wedged copy ran forever. Set one and promote it to the Job in buildJobStrategyBatchJob so the backoffLimit retries share one ceiling (no-op for drivers without it). - empty-source restore: a restore-as-copy whose source lists zero objects now fails instead of reporting success having restored nothing; the refusal no longer names the CLI-only --allow-empty-source flag. - TLS: --ca-file now appends to the system roots instead of replacing them, so the one transport serving both endpoints does not strip public trust from the side that relies on it. - tests: cover buildBucketMirrorPod (args, delete flag, deadline, env) and the s3-mirror purge; docs: the in-place restore note no longer claims byte-for-byte. Assisted-By: LLM Signed-off-by: Andrey Kolkov <[email protected]>
1 parent 5f163c4 commit 6fba946

10 files changed

Lines changed: 407 additions & 41 deletions

File tree

‎examples/backups/bucket/90-scenario.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ A tenant stores objects in an `apps.cozystack.io/Bucket` and enables backups by
44

55
## In-place restore
66

7-
After an accidental deletion the tenant creates a `RestoreJob` with `backupRef` set and no `targetApplicationRef` (`35-restorejob-in-place.yaml`). The driver provisions a read-write `BucketAccess` on the source bucket and mirrors the chosen backup snapshot back over the live contents, deleting objects the snapshot does not contain, so the bucket ends up byte-for-byte as it was at backup time. This is destructive to anything written after the backup — take a fresh backup first if the current state matters.
7+
After an accidental deletion the tenant creates a `RestoreJob` with `backupRef` set and no `targetApplicationRef` (`35-restorejob-in-place.yaml`). The driver provisions a read-write `BucketAccess` on the source bucket and mirrors the chosen backup snapshot back over the live contents, deleting objects the snapshot does not contain, so the bucket's objects and their content type are restored to the snapshot (object versions and tags are not carried). This is destructive to anything written after the backup — take a fresh backup first if the current state matters.
88

99
## To-copy restore
1010

‎hack/select-e2e.sh‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -246,6 +246,9 @@ src_to_suites() {
246246
# roundtrip, so an edit to packages/apps/kafka (or kafka-operator, which
247247
# reaches this source) selects both.
248248
kafka-application) echo "kafka kafka-metadata" ;;
249+
# The Bucket app source owns both the app-create suite and the backup
250+
# round-trip suite, so a bucket-app change exercises both.
251+
bucket-application) echo "bucket bucket-backup" ;;
249252
*-application) echo "${1%-application}" ;;
250253
*) echo "$1" ;;
251254
esac
@@ -409,8 +412,17 @@ while IFS= read -r file || [ -n "$file" ]; do
409412
# selected_apps would empty the final intersection and trip the
410413
# full-suite safety net).
411414
app=$(echo "$file" | sed -nE 's,^examples/backups/([^/]+)/.*,\1,p')
412-
if echo "$all_apps" | grep -Fxq "$app"; then
413-
selected_apps="$selected_apps $app"
415+
# The round-trip Test conventionally lives in its app's own suite dir, but
416+
# an app that already ships a non-backup suite (the Bucket app-create test
417+
# holds the "bucket" suite) keeps its round-trip in "<app>-backup"; prefer
418+
# that when it exists so the harness edit selects the suite that exercises
419+
# the backup path rather than the app-create one.
420+
suite="$app"
421+
if echo "$all_apps" | grep -Fxq "${app}-backup"; then
422+
suite="${app}-backup"
423+
fi
424+
if echo "$all_apps" | grep -Fxq "$suite"; then
425+
selected_apps="$selected_apps $suite"
414426
trigger_any=1
415427
fi
416428
continue ;;

‎hack/select-e2e_test.bats‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,15 @@ assert_full_suite() {
274274
[ "$output" = "postgres" ]
275275
}
276276

277+
@test "backup example harness edit prefers the app's -backup suite when present" {
278+
tmp=$(mktemp -d)
279+
trap 'rm -rf "$tmp"' EXIT
280+
cp -r packages/core/platform/sources "$tmp/sources"
281+
echo "examples/backups/bucket/run-all.sh" > "$tmp/diff"
282+
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
283+
[ "$output" = "bucket-backup" ]
284+
}
285+
277286
@test "backup example without a matching suite selects nothing" {
278287
tmp=$(mktemp -d)
279288
trap 'rm -rf "$tmp"' EXIT

‎hack/select-install.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,10 @@ suite_to_source() {
7878
vminstance) echo cozystack.vm-instance-application ; return ;;
7979
securitygroup) echo cozystack.securitygroup-controller ; return ;;
8080
kafka-metadata) echo cozystack.kafka-application ; return ;;
81+
# The Bucket backup round-trip installs the Bucket app (it provisions its own
82+
# buckets through the example scripts), so it rides the same source as the
83+
# bucket app-create suite; select-e2e.sh maps that source back to both.
84+
bucket-backup) echo cozystack.bucket-application ; return ;;
8185
esac
8286
for cand in "cozystack.$1-application" "cozystack.$1"; do
8387
if echo "$NODES" | grep -Fxq "$cand"; then echo "$cand"; return; fi

‎internal/backupcontroller/backup_controller.go‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -185,6 +185,14 @@ func (r *BackupReconciler) cleanupOnDelete(ctx context.Context, backup *backupsv
185185
// deletes the object and WAITS for that delete before the Backup is
186186
// removed, so no object is orphaned (see cleanupKafkaBackup).
187187
return r.cleanupKafkaBackup(ctx, backup)
188+
case strategyv1alpha1.BucketStrategyKind:
189+
// Like Rabbitmq/Redis, the Bucket driver OWNS its artifact - the mirrored
190+
// objects under the repo prefix, which no engine retention prunes - so
191+
// this branch deletes them and WAITS for that delete before the Backup is
192+
// removed. Without it a retention-pruned Plan leaves orphaned copies in
193+
// cozy-backups with nothing left that knows their prefix (see
194+
// cleanupBucketBackup).
195+
return r.cleanupBucketBackup(ctx, backup)
188196
case strategyv1alpha1.VeleroStrategyKind:
189197
return ctrl.Result{}, r.cleanupVeleroBackup(ctx, backup)
190198
default:

‎internal/backupcontroller/bucketstrategy_controller.go‎

Lines changed: 187 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ import (
1717
"k8s.io/apimachinery/pkg/types"
1818
ctrl "sigs.k8s.io/controller-runtime"
1919
"sigs.k8s.io/controller-runtime/pkg/client"
20+
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
2021

2122
strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1"
2223
backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1"
@@ -44,6 +45,14 @@ const (
4445
// bucketPollInterval matches the Job / Altinity cadence.
4546
bucketPollInterval = 5 * time.Second
4647

48+
// bucketMirrorDeadlineSeconds bounds a single mirror Job run. Without it a
49+
// wedged copy (a hung S3 connection, a never-completing list) keeps the
50+
// mirror Pod Running forever and the BackupJob polling it never terminates.
51+
// Generous so a large legitimate bucket still finishes; the point is a
52+
// finite ceiling, not a tight SLA. ensureJobStrategy*Job promotes it to the
53+
// Job so retries share the ceiling rather than each getting a fresh one.
54+
bucketMirrorDeadlineSeconds int64 = 2 * 60 * 60
55+
4756
// bucketFieldManager owns the COSI BucketAccess objects the driver
4857
// applies via server-side apply.
4958
bucketFieldManager = "cozystack-bucket-backup-driver"
@@ -845,11 +854,186 @@ func buildBucketMirrorPod(mode string, tmpl strategyv1alpha1.BucketTemplate, app
845854
}
846855
}
847856

857+
deadline := bucketMirrorDeadlineSeconds
858+
return &corev1.PodTemplateSpec{
859+
Spec: corev1.PodSpec{
860+
RestartPolicy: corev1.RestartPolicyNever,
861+
ActiveDeadlineSeconds: &deadline,
862+
Containers: []corev1.Container{container},
863+
Volumes: volumes,
864+
},
865+
}
866+
}
867+
868+
// buildBucketCleanupPod builds the one-shot Pod that reclaims a backup's repo
869+
// objects (`s3-mirror --mode=delete`) when its Backup is deleted. Only the repo
870+
// side matters to a purge, so no BucketAccess or app credentials are wired.
871+
func buildBucketCleanupPod(snapshot *bucketBackupSnapshot, image string) *corev1.PodTemplateSpec {
872+
args := []string{
873+
"s3-mirror",
874+
"--mode=delete",
875+
"--repo-endpoint=" + snapshot.RepoEndpoint,
876+
"--repo-bucket=" + snapshot.RepoBucket,
877+
"--repo-prefix=" + snapshot.RepoPrefix,
878+
}
879+
if snapshot.RepoRegion != "" {
880+
args = append(args, "--repo-region="+snapshot.RepoRegion)
881+
}
882+
883+
secretKeySecret := snapshot.RepoCredentialsSecretKeySecret
884+
if secretKeySecret == "" {
885+
secretKeySecret = snapshot.RepoCredentialsSecret
886+
}
887+
env := []corev1.EnvVar{
888+
{
889+
Name: "REPO_ACCESS_KEY",
890+
ValueFrom: &corev1.EnvVarSource{SecretKeyRef: &corev1.SecretKeySelector{
891+
LocalObjectReference: corev1.LocalObjectReference{Name: snapshot.RepoCredentialsSecret},
892+
Key: snapshot.RepoCredentialsAccessKeyKey,
893+
}},
894+
},
895+
{
896+
Name: "REPO_SECRET_KEY",
897+
ValueFrom: &corev1.EnvVarSource{SecretKeyRef: &corev1.SecretKeySelector{
898+
LocalObjectReference: corev1.LocalObjectReference{Name: secretKeySecret},
899+
Key: snapshot.RepoCredentialsSecretKeyKey,
900+
}},
901+
},
902+
}
903+
904+
container := corev1.Container{Name: "cleanup", Image: image, Args: args, Env: env}
905+
906+
var volumes []corev1.Volume
907+
if snapshot.RepoCACertSecret != "" {
908+
container.Args = append(container.Args, "--ca-file="+bucketCAMountPath+"/"+bucketCAFileName)
909+
container.VolumeMounts = append(container.VolumeMounts, corev1.VolumeMount{
910+
Name: "s3-ca",
911+
MountPath: bucketCAMountPath,
912+
ReadOnly: true,
913+
})
914+
volumes = append(volumes, corev1.Volume{
915+
Name: "s3-ca",
916+
VolumeSource: corev1.VolumeSource{Secret: &corev1.SecretVolumeSource{
917+
SecretName: snapshot.RepoCACertSecret,
918+
Items: []corev1.KeyToPath{{Key: snapshot.RepoCACertKey, Path: bucketCAFileName}},
919+
}},
920+
})
921+
} else if snapshot.InsecureSkipVerify {
922+
container.Args = append(container.Args, "--insecure")
923+
}
924+
925+
deadline := bucketMirrorDeadlineSeconds
848926
return &corev1.PodTemplateSpec{
849927
Spec: corev1.PodSpec{
850-
RestartPolicy: corev1.RestartPolicyNever,
851-
Containers: []corev1.Container{container},
852-
Volumes: volumes,
928+
RestartPolicy: corev1.RestartPolicyNever,
929+
ActiveDeadlineSeconds: &deadline,
930+
Containers: []corev1.Container{container},
931+
Volumes: volumes,
853932
},
854933
}
855934
}
935+
936+
// cleanupBucketBackup deletes a Backup's mirrored repo objects when the Backup
937+
// is removed. Like the Redis / Rabbitmq drivers the Bucket driver OWNS its
938+
// artifact (the objects under the repo prefix; no engine retention prunes them),
939+
// so it runs a one-shot delete Job and WAITS for it before the finalizer is
940+
// removed, leaving nothing orphaned in cozy-backups. It fails open - releasing
941+
// the Backup and leaving the objects - when the delete cannot proceed (namespace
942+
// terminating, strategy gone, credentials unprojectable) rather than wedging the
943+
// Backup Terminating forever; `backups.cozystack.io/skip-artifact-cleanup` is the
944+
// operator escape hatch.
945+
func (r *BackupReconciler) cleanupBucketBackup(ctx context.Context, backup *backupsv1alpha1.Backup) (ctrl.Result, error) {
946+
logger := getLogger(ctx)
947+
jobName := backup.Name + "-cleanup"
948+
949+
if backup.Annotations[redisSkipArtifactCleanupAnnotation] == "true" {
950+
existing := &batchv1.Job{}
951+
if err := r.Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: jobName}, existing); err == nil {
952+
_ = r.deleteBucketCleanupJob(ctx, existing)
953+
}
954+
logger.Debug("skipping Bucket artifact cleanup per annotation; objects left in the repo bucket", "backup", backup.Name)
955+
return ctrl.Result{}, nil
956+
}
957+
958+
snapshot, err := unmarshalBucketSnapshot(backup.Status.UnderlyingResources)
959+
if err != nil {
960+
return ctrl.Result{}, fmt.Errorf("decode Backup snapshot: %w", err)
961+
}
962+
if snapshot == nil || snapshot.RepoBucket == "" || snapshot.RepoPrefix == "" {
963+
// A Backup written before the snapshot recorded repo coordinates names
964+
// nothing to delete.
965+
return ctrl.Result{}, nil
966+
}
967+
968+
strategy := &strategyv1alpha1.Bucket{}
969+
if err := r.Get(ctx, client.ObjectKey{Name: backup.Spec.StrategyRef.Name}, strategy); err != nil {
970+
if apierrors.IsNotFound(err) {
971+
return r.releaseBucketCleanup(ctx, backup, snapshot.RepoPrefix, "strategy CR is gone"), nil
972+
}
973+
return ctrl.Result{}, err
974+
}
975+
976+
job := &batchv1.Job{}
977+
err = r.Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: jobName}, job)
978+
switch {
979+
case apierrors.IsNotFound(err):
980+
terminating, nsErr := r.namespaceTerminating(ctx, backup.Namespace)
981+
if nsErr != nil {
982+
return ctrl.Result{}, nsErr
983+
}
984+
if terminating {
985+
return r.releaseBucketCleanup(ctx, backup, snapshot.RepoPrefix, "namespace is terminating"), nil
986+
}
987+
if perr := ProjectBackupCredentials(ctx, r.Client, r.CredentialsConfig, backup.Namespace); perr != nil {
988+
return r.releaseBucketCleanup(ctx, backup, snapshot.RepoPrefix, fmt.Sprintf("cannot project credentials: %v", perr)), nil
989+
}
990+
pod := buildBucketCleanupPod(snapshot, strategy.Spec.Template.Image)
991+
desired := buildJobStrategyBatchJob(backup.Namespace, jobName, map[string]string{bucketLabelMode: "cleanup"}, pod)
992+
if cerr := controllerutil.SetControllerReference(backup, desired, r.Scheme); cerr != nil {
993+
return ctrl.Result{}, fmt.Errorf("set controller reference on cleanup Job: %w", cerr)
994+
}
995+
if cerr := r.Create(ctx, desired); cerr != nil && !apierrors.IsAlreadyExists(cerr) {
996+
// Forbidden (namespace went Terminating after the check) or Invalid
997+
// (a long Backup name overflows the 63-char Job name) cannot be fixed
998+
// by retrying; release. Other errors are transient - requeue.
999+
if apierrors.IsForbidden(cerr) || apierrors.IsInvalid(cerr) {
1000+
return r.releaseBucketCleanup(ctx, backup, snapshot.RepoPrefix, fmt.Sprintf("cannot create cleanup Job: %v", cerr)), nil
1001+
}
1002+
return ctrl.Result{}, cerr
1003+
}
1004+
return ctrl.Result{RequeueAfter: bucketPollInterval}, nil
1005+
case err != nil:
1006+
return ctrl.Result{}, err
1007+
}
1008+
1009+
if !job.DeletionTimestamp.IsZero() {
1010+
// A prior failed attempt is being collected; wait for it to go, then the
1011+
// NotFound branch recreates a fresh one.
1012+
return ctrl.Result{RequeueAfter: bucketPollInterval}, nil
1013+
}
1014+
switch jobConditionState(job) {
1015+
case batchv1.JobComplete:
1016+
_ = r.deleteBucketCleanupJob(ctx, job)
1017+
logger.Debug("Bucket backup objects deleted", "backup", backup.Name, "prefix", snapshot.RepoPrefix)
1018+
return ctrl.Result{}, nil
1019+
case batchv1.JobFailed:
1020+
_ = r.deleteBucketCleanupJob(ctx, job)
1021+
logger.Debug("Bucket cleanup Job failed; retrying", "backup", backup.Name)
1022+
return ctrl.Result{RequeueAfter: bucketPollInterval}, nil
1023+
default:
1024+
return ctrl.Result{RequeueAfter: bucketPollInterval}, nil
1025+
}
1026+
}
1027+
1028+
func (r *BackupReconciler) deleteBucketCleanupJob(ctx context.Context, job *batchv1.Job) error {
1029+
policy := metav1.DeletePropagationBackground
1030+
return client.IgnoreNotFound(r.Delete(ctx, job, &client.DeleteOptions{PropagationPolicy: &policy}))
1031+
}
1032+
1033+
func (r *BackupReconciler) releaseBucketCleanup(ctx context.Context, backup *backupsv1alpha1.Backup, prefix, reason string) ctrl.Result {
1034+
getLogger(ctx).Info("releasing Backup without deleting its repo objects", "backup", backup.Name, "prefix", prefix, "reason", reason)
1035+
if r.Recorder != nil {
1036+
r.Recorder.Eventf(backup, corev1.EventTypeWarning, "ArtifactNotDeleted", "left objects under %s in the repo bucket: %s", prefix, reason)
1037+
}
1038+
return ctrl.Result{}
1039+
}

‎internal/backupcontroller/bucketstrategy_controller_test.go‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -259,3 +259,55 @@ func TestReconcileBucketAccessReusesOwnedMatch(t *testing.T) {
259259
t.Fatalf("returned spec = %+v, want the existing %+v", got.Spec, owned.Spec)
260260
}
261261
}
262+
263+
func hasArg(args []string, want string) bool {
264+
for _, a := range args {
265+
if a == want {
266+
return true
267+
}
268+
}
269+
return false
270+
}
271+
272+
func TestBuildBucketMirrorPod(t *testing.T) {
273+
tmpl := strategyv1alpha1.BucketTemplate{
274+
Image: "controller:latest",
275+
Destination: strategyv1alpha1.BucketDestination{
276+
Bucket: "cozy-backups",
277+
Endpoint: "http://seaweedfs:8333",
278+
Region: "us-east-1",
279+
AccessKeyIDSecretKeyRef: strategyv1alpha1.BucketSecretKeySelector{Name: "cozy-backups-creds", Key: "AWS_ACCESS_KEY_ID"},
280+
SecretAccessKeySecretKeyRef: strategyv1alpha1.BucketSecretKeySelector{Name: "cozy-backups-creds", Key: "AWS_SECRET_ACCESS_KEY"},
281+
},
282+
}
283+
284+
backupPod := buildBucketMirrorPod(bucketModeBackup, tmpl, "bucket-web-cozy-backup", "tenant-x/web/bk1/", false)
285+
c := backupPod.Spec.Containers[0]
286+
for _, want := range []string{"s3-mirror", "--mode=backup", "--repo-bucket=cozy-backups", "--repo-prefix=tenant-x/web/bk1/", "--repo-region=us-east-1"} {
287+
if !hasArg(c.Args, want) {
288+
t.Errorf("backup args missing %q: %v", want, c.Args)
289+
}
290+
}
291+
if hasArg(c.Args, "--delete-extraneous") {
292+
t.Errorf("backup must not set --delete-extraneous: %v", c.Args)
293+
}
294+
if backupPod.Spec.ActiveDeadlineSeconds == nil {
295+
t.Error("mirror pod has no ActiveDeadlineSeconds")
296+
}
297+
gotEnv := map[string]bool{}
298+
for _, e := range c.Env {
299+
gotEnv[e.Name] = true
300+
}
301+
for _, want := range []string{"APP_BUCKETINFO", "REPO_ACCESS_KEY", "REPO_SECRET_KEY"} {
302+
if !gotEnv[want] {
303+
t.Errorf("env missing %q", want)
304+
}
305+
}
306+
307+
// The delete flag (the one that erases objects) rides only on an in-place
308+
// restore, so it must appear exactly when the caller asks for it.
309+
restorePod := buildBucketMirrorPod(bucketModeRestore, tmpl, "bucket-web-cozy-restore", "tenant-x/web/bk1/", true)
310+
if !hasArg(restorePod.Spec.Containers[0].Args, "--delete-extraneous") {
311+
t.Errorf("in-place restore must set --delete-extraneous: %v", restorePod.Spec.Containers[0].Args)
312+
}
313+
}

‎internal/backupcontroller/jobstrategy_controller.go‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -314,16 +314,23 @@ func buildJobStrategyBatchJob(namespace, name string, labels map[string]string,
314314
// retries while a sidecar wants more - so making it operator-configurable
315315
// is tracked in https://github.com/cozystack/cozystack/issues/2742.
316316
backoffLimit := int32(2)
317+
spec := batchv1.JobSpec{
318+
BackoffLimit: &backoffLimit,
319+
Template: pod,
320+
}
321+
// A Pod-level activeDeadlineSeconds bounds a single attempt; promote it to
322+
// the Job so the backoffLimit retries share one ceiling rather than each
323+
// getting a fresh deadline. No-op for drivers that set no pod deadline.
324+
if d := pod.Spec.ActiveDeadlineSeconds; d != nil {
325+
spec.ActiveDeadlineSeconds = d
326+
}
317327
return &batchv1.Job{
318328
ObjectMeta: metav1.ObjectMeta{
319329
Namespace: namespace,
320330
Name: name,
321331
Labels: labels,
322332
},
323-
Spec: batchv1.JobSpec{
324-
BackoffLimit: &backoffLimit,
325-
Template: pod,
326-
},
333+
Spec: spec,
327334
}
328335
}
329336

0 commit comments

Comments
 (0)