Skip to content

Commit f4bde9e

Browse files
authored
fix(backups): delete the original PVC of an in-place VM restore without keepOriginalPVC (#4672)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does An in-place VMInstance restore with `keepOriginalPVC: false` reports `Succeeded` without restoring the disks, and leaves every disk it touched unable to come back. `prepareForRestore` deletes each DataVolume with Orphan propagation and leaves the PVC in place, on the assumption, stated in the `deleteDataVolume` comment, that Velero overwrites it. Velero does not overwrite a PVC. Its CSI restore action returns early for a claim that already exists, so no DataDownload is ever created, and with `existingResourcePolicy: update` the merge patch it attempts instead is rejected because a bound claim's spec is immutable. All that reaches the claim is the labels-only fallback patch, which also drops the `cdi.kubevirt.io/allowClaimAdoption` annotation the resource modifier added. The claim left behind then blocks the disk for good. Orphan propagation has removed its owner reference to the DataVolume, and it lacks the `cdi.kubevirt.io/storage.populatedFor` annotation that kubevirt-velero-plugin writes on the backed-up copy, so when the HelmRelease resumes, the CDI webhook refuses the DataVolume it creates with `Destination PVC <ns>/<name> already exists`, on every reconcile. Deleting the disk through the API leaves the PVC behind, and a new disk of the same name fails the same way. Observed on a v1.6.3 cluster, for a VMInstance restore with `keepOriginalPVC: false` whose disk was attached to the VM. The Velero log for the restore, trimmed: ```text level=info msg="Retrieved per-restore resource modifiers from configmap cozy-velero/restore-modifiers-..." level=warning msg="PVC already exists and ExistingVolumeDataPolicy is not in-place restore. Skip restore this PVC." Action=PVCCSIRestoreItemAction PVC=tenant-datalab/vm-disk-e2edisk level=info msg="Applying resource modifier patch on tenant-datalab/vm-disk-e2edisk" level=warning msg="patch attempt failed for /v1, Kind=PersistentVolumeClaim tenant-datalab/vm-disk-e2edisk: PersistentVolumeClaim \"vm-disk-e2edisk\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims" level=info msg="backup/restore labels successfully updated for PersistentVolumeClaim tenant-datalab/vm-disk-e2edisk" ``` `velero restore describe --details` lists the PVC as `(failed)` and the Restore is `Completed` with two warnings. Afterwards the claim had its original creation timestamp and UID, no owner reference, no `allowClaimAdoption`, no `populatedFor`, and the disk's HelmRelease was `Ready=False` with the webhook error above. That cluster runs a Velero build from `main`, whose wording is quoted above; the released v1.17.1 and v1.18.3 return early at the same point with `PVC already exists. Skip restore this PVC.` (`pkg/restore/actions/csi/pvc_action.go`). The fix deletes the PVC when `keepOriginalPVC` is false, only once its DataVolume is confirmed gone so that CDI cannot recreate it, and holds the Velero Restore until the claim has actually disappeared. A claim still mounted stays `Terminating` behind `kubernetes.io/pvc-protection`, and creating the Restore then would hit the same existing-claim path. Velero then creates the PVC from the backup, the data mover fills it, and the recreated DataVolume adopts it through `populatedFor`. A failed deletion now fails the RestoreJob instead of letting it report `Succeeded` over the old data. Before 2234824 the DataVolume was deleted with background propagation, so garbage collection removed the claim, but nothing waited for it. Deleting the claim is only safe when Velero can put the disk back, so before any preparation the reconciler now gets the Velero Backup and fails the RestoreJob unless it exists and is `Completed`. The Backup artifact outlives the Velero Backup, which Velero garbage-collects at its TTL, and without the check a restore of an expired backup would halt the VM and delete its disks before failing. The check runs for every Velero restore, whatever `keepOriginalPVC` says. A DataVolume that cannot be deleted now fails the RestoreJob too, and the preparation waits for every DataVolume to be gone before it goes further: a surviving DataVolume makes CDI recreate the claim, which Velero then skips. Reading the DataVolume back needs `get` on `datavolumes`, which the backupstrategy-controller ClusterRole now grants next to `delete`. `TestPrepareForRestore_KeepOriginalPVCFalse_SkipsRename` asserted the old behaviour, that the original PVC survives, and is replaced by `TestPrepareForRestore_KeepOriginalPVCFalse_DeletesPVC`. `TestPrepareForRestore_KeepOriginalPVCFalse_WaitsForTerminatingPVC` covers the wait. Both fail on `upstream/main` without the controller change. `TestReconcileVeleroRestore_UnusableVeleroBackupTouchesNothing` covers an expired Velero Backup and one being deleted, `TestPrepareForRestore_KeepOriginalPVCFalse_KeepsPVCWhileDataVolumeRemains` a failed and a pending DataVolume deletion, and `TestPrepareForRestore_KeepOriginalPVCTrue_FailedRenameKeepsPVC` a failed rename under `keepOriginalPVC: true`, which must leave the claim in place. A helm-unittest suite, `tests/restore_rbac_test.yaml`, checks the new verb. Each test fails with its fix reverted, with one exception: `TestPrepareForRestore_KeepOriginalPVCTrue_FailedRenameKeepsPVC` still passes with the Step 5 condition cut to `vmRes != nil`, because a failed rename holds the restore back before Step 5 is reached. That guard is pinned by `TestPrepareForRestore_KeepOriginalPVC_CacheLagsBehindWrites`, which fails with the same cut. `helm unittest packages/system/backupstrategy-controller` passes. `go test ./internal/backupcontroller/...` passes. Related: #4601 reports the same Velero mechanism for a standalone VMDisk, where `prepareForRestore` does nothing at all because `collectUnderlyingResources` records no disk. #3427 makes a VMDisk report its own disk; with `keepOriginalPVC: false` that path then runs into this defect too, and this change covers it once both are in. This PR does not touch the VMDisk path itself. ### Screenshots N/A — no UI changes. ### Downstream repositories Walked the trigger map against the diff, which is the restore preparation in `internal/backupcontroller/velerostrategy_controller.go` and its unit tests, plus one verb in the backupstrategy-controller ClusterRole and its helm-unittest: no package added or renamed, no `values.yaml` or `values.schema.json` change, no CRD, ApplicationDefinition, label, annotation, namespace or metric change. `keepOriginalPVC` lives in the opaque `RestoreJob.spec.options`, and a search of the organisation finds it only in this repository, so neither the website nor the Terraform provider documents or models it. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): an in-place VMInstance RestoreJob with keepOriginalPVC=false now deletes each original disk PVC, and waits for it to be gone, before Velero restores it. Previously the PVC was left in place, Velero skipped restoring its data, the RestoreJob still reported Succeeded, and the disk's HelmRelease could never recreate its DataVolume ("Destination PVC already exists"). A Velero RestoreJob now also fails before touching the application when its Velero Backup is gone or not Completed, for example after the Backup expired at its TTL. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * When a restore is configured not to keep the original disk claim, the claim is deleted before Velero restores it, allowing Velero to recreate it. * Restores wait while the original disk claim or its DataVolume is being deleted. Deletion or status-check errors stop restore preparation. * Restores configured to keep the original disk claim continue to preserve it. * Restores stop before changing resources if the referenced Velero backup is missing or has not completed. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents c290b75 + 6d064a8 commit f4bde9e

4 files changed

Lines changed: 423 additions & 59 deletions

File tree

‎internal/backupcontroller/velerostrategy_controller.go‎

Lines changed: 128 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,8 @@ type CommonRestoreOptions struct {
109109
// KeepOriginalIpAndMac) are only effective when the application kind is VMInstance.
110110
type RestoreOptions struct {
111111
CommonRestoreOptions `json:",inline"`
112-
// KeepOriginalPVC renames the original PVC to <name>-orig-<hash> before restore.
112+
// KeepOriginalPVC renames the original PVC to <name>-orig-<hash> before restore;
113+
// when false, the original PVC is deleted so that Velero recreates it.
113114
// Only effective for in-place VMInstance restore (no targetNamespace). Defaults to true when omitted.
114115
KeepOriginalPVC *bool `json:"keepOriginalPVC,omitempty"`
115116
// KeepOriginalIpAndMac preserves the original IP and MAC address via OVN
@@ -697,8 +698,26 @@ func (r *RestoreJobReconciler) reconcileVeleroRestore(ctx context.Context, resto
697698
}
698699
}
699700

701+
// The Backup artifact outlives the Velero Backup it points to, which
702+
// Velero garbage-collects once its TTL expires. Checked before
703+
// prepareForRestore halts the application and deletes its disks, so a
704+
// restore that cannot run leaves them in place.
705+
veleroBackup := &velerov1.Backup{}
706+
if err := r.Get(ctx, client.ObjectKey{Namespace: veleroNamespace, Name: veleroBackupName}, veleroBackup); err != nil {
707+
if errors.IsNotFound(err) {
708+
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf(
709+
"Velero Backup %s/%s not found; it may have expired", veleroNamespace, veleroBackupName))
710+
}
711+
return ctrl.Result{}, fmt.Errorf("failed to get Velero Backup %s: %w", veleroBackupName, err)
712+
}
713+
if veleroBackup.Status.Phase != velerov1.BackupPhaseCompleted {
714+
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf(
715+
"Velero Backup %s/%s is %q, not %q", veleroNamespace, veleroBackupName,
716+
veleroBackup.Status.Phase, velerov1.BackupPhaseCompleted))
717+
}
718+
700719
// Resolve underlying resources once; prefer Backup status, fall back to Velero annotation.
701-
ur := r.resolveUnderlyingResourcesForRestore(ctx, backup, veleroBackupName)
720+
ur := resolveUnderlyingResourcesForRestore(backup, veleroBackup)
702721

703722
// Pre-restore: graceful shutdown, suspend HRs, rename PVCs (skipped for copy)
704723
ready, result, err := r.prepareForRestore(ctx, restoreJob, backup, ur, target, restoreOpts)
@@ -960,14 +979,10 @@ func (r *RestoreJobReconciler) createResourceModifiersConfigMap(ctx context.Cont
960979
// resolveUnderlyingResourcesForRestore returns underlying resources for symmetric
961980
// restore label selectors. Velero Backup annotation is used when Backup.status was empty
962981
// (e.g. CRD without underlyingResources in schema).
963-
func (r *RestoreJobReconciler) resolveUnderlyingResourcesForRestore(ctx context.Context, backup *backupsv1alpha1.Backup, veleroBackupName string) *runtime.RawExtension {
982+
func resolveUnderlyingResourcesForRestore(backup *backupsv1alpha1.Backup, vb *velerov1.Backup) *runtime.RawExtension {
964983
if backup.Status.UnderlyingResources != nil && len(backup.Status.UnderlyingResources.Raw) > 0 {
965984
return backup.Status.UnderlyingResources
966985
}
967-
vb := &velerov1.Backup{}
968-
if err := r.Get(ctx, client.ObjectKey{Namespace: veleroNamespace, Name: veleroBackupName}, vb); err != nil {
969-
return backup.Status.UnderlyingResources
970-
}
971986
if urJSON, ok := vb.Annotations[underlyingResourcesAnnotation]; ok && urJSON != "" {
972987
return &runtime.RawExtension{Raw: []byte(urJSON)}
973988
}
@@ -1036,13 +1051,14 @@ func shortHash(input string) string {
10361051
// 5. Renames existing PVCs to <name>-orig-<hash> so Velero can create fresh
10371052
// ones via Data Movement.
10381053
//
1039-
// Failures in individual steps are non-fatal: missing resources are expected
1040-
// (e.g. restore requested when app was already deleted). Each action emits
1041-
// a Kubernetes Event on the RestoreJob for observability. The rename is the
1042-
// exception: the DataVolume deletion after it would hand a disk that was not
1043-
// kept to the restore, as keepOriginalPVC=false does. A failed rename holds
1044-
// the restore back and is retried, and an -orig name another PVC holds fails
1045-
// the restore.
1054+
// Missing resources are expected (e.g. restore requested when app was already
1055+
// deleted), and a failure to suspend or halt is non-fatal. A failed rename
1056+
// holds the restore back and is retried, since the DataVolume deletion after
1057+
// it would hand a disk that was not kept to the restore, as
1058+
// keepOriginalPVC=false does, and an -orig name another PVC holds fails the
1059+
// restore. A DataVolume or a replaced PVC that cannot be deleted fails the
1060+
// restore, since Velero would skip the claim left in place. Each action emits
1061+
// a Kubernetes Event on the RestoreJob for observability.
10461062
//
10471063
// postRestoreRename renames VMInstance HelmRelease after Velero Restore completes.
10481064
// Velero resource modifiers cannot change metadata.name, so this step creates
@@ -1204,16 +1220,54 @@ func (r *RestoreJobReconciler) prepareForRestore(ctx context.Context, restoreJob
12041220
}
12051221

12061222
// --- Step 4: Delete DataVolumes so CDI doesn't recreate PVCs ---
1223+
// A DataVolume still present makes CDI recreate its claim under the original
1224+
// name, which Velero then skips as an existing PVC: the restore would end
1225+
// Succeeded on a freshly imported disk, so nothing goes further until every
1226+
// DataVolume is gone.
12071227
if vmRes != nil {
1228+
var terminating []string
12081229
for _, dv := range vmRes.DataVolumes {
1209-
if err := r.deleteDataVolume(ctx, ns, dv.DataVolumeName); err != nil {
1210-
r.Recorder.Event(restoreJob, corev1.EventTypeWarning, "PrepareForRestore",
1211-
fmt.Sprintf("Failed to delete DataVolume %s: %v", dv.DataVolumeName, err))
1212-
} else {
1213-
r.Recorder.Event(restoreJob, corev1.EventTypeNormal, "PrepareForRestore",
1214-
fmt.Sprintf("Deleted DataVolume %s", dv.DataVolumeName))
1230+
gone, err := r.deleteDataVolume(ctx, ns, dv.DataVolumeName)
1231+
if err != nil {
1232+
return false, ctrl.Result{}, fmt.Errorf("failed to delete DataVolume %s: %w", dv.DataVolumeName, err)
1233+
}
1234+
if !gone {
1235+
terminating = append(terminating, dv.DataVolumeName)
1236+
continue
1237+
}
1238+
r.Recorder.Event(restoreJob, corev1.EventTypeNormal, "PrepareForRestore",
1239+
fmt.Sprintf("Deleted DataVolume %s", dv.DataVolumeName))
1240+
}
1241+
if len(terminating) > 0 {
1242+
r.Recorder.Event(restoreJob, corev1.EventTypeNormal, "PrepareForRestore",
1243+
fmt.Sprintf("Waiting for DataVolumes %s to be deleted", strings.Join(terminating, ", ")))
1244+
return false, ctrl.Result{RequeueAfter: defaultRestoreRequeueAfter}, nil
1245+
}
1246+
}
1247+
1248+
// --- Step 5: Delete the PVCs a restore without keepOriginalPVC replaces ---
1249+
// Velero restores a PVC only by creating it: its CSI restore action skips the
1250+
// data mover for a PVC that already exists, and existingResourcePolicy=update
1251+
// cannot patch the immutable spec of a bound claim. A PVC left in place keeps
1252+
// its old data, and, orphaned from its DataVolume in step 4 and without the
1253+
// cdi.kubevirt.io/storage.populatedFor the backed-up copy carries, it makes
1254+
// CDI refuse the DataVolume the resumed HelmRelease creates.
1255+
if !opts.GetKeepOriginalPVC() && vmRes != nil {
1256+
var terminating []string
1257+
for _, dv := range vmRes.DataVolumes {
1258+
gone, err := r.deletePVC(ctx, ns, dv.DataVolumeName)
1259+
if err != nil {
1260+
return false, ctrl.Result{}, fmt.Errorf("failed to delete PVC %s: %w", dv.DataVolumeName, err)
1261+
}
1262+
if !gone {
1263+
terminating = append(terminating, dv.DataVolumeName)
12151264
}
12161265
}
1266+
if len(terminating) > 0 {
1267+
r.Recorder.Event(restoreJob, corev1.EventTypeNormal, "PrepareForRestore",
1268+
fmt.Sprintf("Waiting for PVCs %s to be deleted", strings.Join(terminating, ", ")))
1269+
return false, ctrl.Result{RequeueAfter: defaultRestoreRequeueAfter}, nil
1270+
}
12171271
}
12181272

12191273
r.Recorder.Event(restoreJob, corev1.EventTypeNormal, "PrepareForRestore", "Pre-restore preparation complete")
@@ -1273,19 +1327,35 @@ func (r *RestoreJobReconciler) haltVirtualMachine(ctx context.Context, ns, vmNam
12731327
return false, nil
12741328
}
12751329

1276-
// deleteDataVolume deletes a DataVolume so CDI doesn't recreate the PVC after rename.
1277-
// Uses Orphan propagation to avoid cascade-deleting the PVC that the DV owns via
1278-
// ownerReference. Without this, keepOriginalPVC=false would silently destroy the
1279-
// original PVC through garbage collection instead of leaving it for Velero to overwrite.
1280-
func (r *RestoreJobReconciler) deleteDataVolume(ctx context.Context, ns, name string) error {
1281-
orphan := metav1.DeletePropagationOrphan
1282-
err := r.Resource(dataVolumeGVR).Namespace(ns).Delete(ctx, name, metav1.DeleteOptions{
1283-
PropagationPolicy: &orphan,
1284-
})
1285-
if err != nil && !errors.IsNotFound(err) {
1286-
return err
1330+
// deleteDataVolume deletes a DataVolume so CDI doesn't recreate the PVC after
1331+
// rename, and reports whether it is gone. Orphan propagation holds the DV behind
1332+
// the orphan finalizer until garbage collection has released its PVC, and leaves
1333+
// that PVC to prepareForRestore, which waits for its deletion, rather than to
1334+
// background garbage collection, which nothing waits for before the Velero
1335+
// Restore starts.
1336+
func (r *RestoreJobReconciler) deleteDataVolume(ctx context.Context, ns, name string) (bool, error) {
1337+
dvClient := r.Resource(dataVolumeGVR).Namespace(ns)
1338+
dv, err := dvClient.Get(ctx, name, metav1.GetOptions{})
1339+
if err != nil {
1340+
if errors.IsNotFound(err) {
1341+
return true, nil
1342+
}
1343+
return false, err
12871344
}
1288-
return nil
1345+
if dv.GetDeletionTimestamp() == nil {
1346+
orphan := metav1.DeletePropagationOrphan
1347+
err := dvClient.Delete(ctx, name, metav1.DeleteOptions{PropagationPolicy: &orphan})
1348+
if err != nil && !errors.IsNotFound(err) {
1349+
return false, err
1350+
}
1351+
}
1352+
if _, err := dvClient.Get(ctx, name, metav1.GetOptions{}); err != nil {
1353+
if errors.IsNotFound(err) {
1354+
return true, nil
1355+
}
1356+
return false, err
1357+
}
1358+
return false, nil
12891359
}
12901360

12911361
// getOrigPVC returns the PVC named name, or nil when there is none. It reads
@@ -1310,6 +1380,32 @@ func foreignOrigPVC(pvc *corev1.PersistentVolumeClaim, restoreJob *backupsv1alph
13101380
"remove that PVC or create the RestoreJob under another name", pvc.Namespace, pvc.Name)
13111381
}
13121382

1383+
// deletePVC deletes a PVC and reports whether it is gone. A claim still mounted
1384+
// stays Terminating behind kubernetes.io/pvc-protection, and Velero would find
1385+
// it still there.
1386+
func (r *RestoreJobReconciler) deletePVC(ctx context.Context, ns, name string) (bool, error) {
1387+
key := client.ObjectKey{Namespace: ns, Name: name}
1388+
pvc := &corev1.PersistentVolumeClaim{}
1389+
if err := r.Get(ctx, key, pvc); err != nil {
1390+
if errors.IsNotFound(err) {
1391+
return true, nil
1392+
}
1393+
return false, err
1394+
}
1395+
if pvc.DeletionTimestamp == nil {
1396+
if err := r.Delete(ctx, pvc); err != nil && !errors.IsNotFound(err) {
1397+
return false, err
1398+
}
1399+
}
1400+
if err := r.Get(ctx, key, pvc); err != nil {
1401+
if errors.IsNotFound(err) {
1402+
return true, nil
1403+
}
1404+
return false, err
1405+
}
1406+
return false, nil
1407+
}
1408+
13131409
// renamePVC preserves an existing PVC by rebinding it under a new name.
13141410
// The original PVC is deleted and a new one pointing to the same PV is created.
13151411
// Missing resources are silently skipped (non-fatal). Each step can be

0 commit comments

Comments
 (0)