Skip to content

Commit d8d9608

Browse files
fix(backups): refuse a foreign -orig PVC and record the kept disk's reclaim policy
renamePVC returned early whenever a PVC named <dv>-orig-<hash> already existed, without checking it came from this rename. The hash covers only the RestoreJob name, so a RestoreJob created under the name of a deleted one finds the -orig PVC its predecessor left behind. The rename was skipped, the DataVolume deleted anyway, and the restore ran as if keepOriginalPVC were false while reporting success: the live disk was not kept, and the -orig name held an older one. Any other rename failure ended the same way, since its error only became an event. A -orig PVC now records the UID of the RestoreJob that created it. A name held by any other PVC fails the restore before the VM is halted, an interrupted rename of our own resumes instead of stopping halfway, and a rename failure fails the restore before the DataVolume goes. The PV is switched to Retain so it outlives the original PVC, and the policy it had was discarded. Nothing in the controller deletes the -orig PVC or restores that policy, so a PV that was Delete stayed Retain and its volume leaked once an operator removed the claim. The policy is now recorded on the PV and on the -orig PVC, and the restore scenario documents how to set it back. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent bc400df commit d8d9608

3 files changed

Lines changed: 363 additions & 47 deletions

File tree

‎examples/backups/vmi/scenario-user-restore.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,22 @@ The backup controller performs the following steps automatically:
3939
After the Velero Restore completes, the HelmReleases resume and the VM boots
4040
with the restored disk data while retaining its original IP and MAC addresses.
4141

42+
### Reclaiming the kept disks
43+
44+
With `keepOriginalPVC: true`, the default, step 3 switches each disk's PV to the `Retain` reclaim policy so that it outlives the PVC it is renamed from. Nothing removes the `-orig` PVCs or sets the policy back afterwards: they stay until an operator deletes them.
45+
46+
The policy the PV had before the restore is recorded in the `backups.cozystack.io/original-reclaim-policy` annotation, on the PV and on the `-orig` PVC. Set it back on the PV before deleting the `-orig` PVC, or a PV that was `Delete` stays `Retain` once released and its volume is never reclaimed. A PVC renamed by an older controller carries no annotation; the snippet then leaves the PV on `Retain` for you to decide.
47+
48+
```bash
49+
PVC=vm-disk-ubuntu-source-orig-1a2b
50+
PV=$(kubectl -n "$NAMESPACE" get pvc "$PVC" -o jsonpath='{.spec.volumeName}')
51+
POLICY=$(kubectl get pv "$PV" -o jsonpath='{.metadata.annotations.backups\.cozystack\.io/original-reclaim-policy}')
52+
kubectl patch pv "$PV" --type=merge -p "{\"spec\":{\"persistentVolumeReclaimPolicy\":\"${POLICY:-Retain}\"}}"
53+
kubectl -n "$NAMESPACE" delete pvc "$PVC"
54+
```
55+
56+
The `<hash>` is derived from the RestoreJob name, so an `-orig` PVC left behind by a deleted RestoreJob takes the name a new RestoreJob of the same name would use. That RestoreJob fails before it halts the VM, naming the PVC, rather than restoring without keeping the current disk. Reclaim the leftover PVC as above, or give the new RestoreJob another name.
57+
4258
## Method 2: Cross-Namespace Restore (Copy)
4359

4460
```bash

‎internal/backupcontroller/velerostrategy_controller.go‎

Lines changed: 114 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,17 @@ const (
5858
// Annotation key for persisting underlying resources on the Velero Backup object
5959
underlyingResourcesAnnotation = "backups.cozystack.io/underlying-resources"
6060

61+
// keepOriginalPVC moves a disk's PV to Retain so it survives the deletion
62+
// of its PVC. The policy it had is kept here, on the PV and on the
63+
// <name>-orig-<hash> PVC, because nothing in the controller restores it:
64+
// whoever deletes the -orig PVC must set it back first, or a PV that was
65+
// Delete stays Retain and its volume is never reclaimed.
66+
originalReclaimPolicyAnnotation = "backups.cozystack.io/original-reclaim-policy"
67+
// The -orig suffix hashes the RestoreJob name, and a name can be reused
68+
// once the previous RestoreJob is deleted, leaving its -orig PVC behind.
69+
// The UID tells a rename this RestoreJob started apart from that one.
70+
origPVCRestoreJobUIDAnnotation = "backups.cozystack.io/restore-job-uid"
71+
6172
// VM-specific constants
6273
vmInstanceKind = "VMInstance"
6374
vmDiskAppKind = "VMDisk"
@@ -1026,7 +1037,9 @@ func shortHash(input string) string {
10261037
//
10271038
// Failures in individual steps are non-fatal: missing resources are expected
10281039
// (e.g. restore requested when app was already deleted). Each action emits
1029-
// a Kubernetes Event on the RestoreJob for observability.
1040+
// a Kubernetes Event on the RestoreJob for observability. The rename is the
1041+
// exception: the DataVolume deletion after it would hand a disk that was not
1042+
// kept to the restore, as keepOriginalPVC=false does, so it fails the restore.
10301043
//
10311044
// postRestoreRename renames VMInstance HelmRelease after Velero Restore completes.
10321045
// Velero resource modifiers cannot change metadata.name, so this step creates
@@ -1117,8 +1130,19 @@ func (r *RestoreJobReconciler) prepareForRestore(ctx context.Context, restoreJob
11171130
appKind := backup.Spec.ApplicationRef.Kind
11181131
origSuffix := "-orig-" + shortHash(restoreJob.Name)
11191132

1120-
// --- Step 1: Suspend HelmReleases ---
11211133
vmRes := getVMInstanceResources(ur)
1134+
1135+
// Refuse a taken -orig name before anything is halted or deleted, rather
1136+
// than fail halfway through a multi-disk rename.
1137+
if opts.GetKeepOriginalPVC() && vmRes != nil {
1138+
for _, dv := range vmRes.DataVolumes {
1139+
if _, err := r.checkOrigPVCName(ctx, restoreJob, ns, dv.DataVolumeName+origSuffix); err != nil {
1140+
return false, ctrl.Result{}, err
1141+
}
1142+
}
1143+
}
1144+
1145+
// --- Step 1: Suspend HelmReleases ---
11221146
hrNames := []string{}
11231147
if appKind == vmInstanceKind {
11241148
hrNames = append(hrNames, vmNamePrefix+appName)
@@ -1165,6 +1189,7 @@ func (r *RestoreJobReconciler) prepareForRestore(ctx context.Context, restoreJob
11651189
if err := r.renamePVC(ctx, restoreJob, ns, dv.DataVolumeName, dv.DataVolumeName+origSuffix); err != nil {
11661190
r.Recorder.Event(restoreJob, corev1.EventTypeWarning, "PrepareForRestore",
11671191
fmt.Sprintf("Failed to keep old PVC %s: %v", dv.DataVolumeName, err))
1192+
return false, ctrl.Result{}, fmt.Errorf("failed to keep PVC %s: %w", dv.DataVolumeName, err)
11681193
}
11691194
}
11701195
}
@@ -1254,77 +1279,119 @@ func (r *RestoreJobReconciler) deleteDataVolume(ctx context.Context, ns, name st
12541279
return nil
12551280
}
12561281

1282+
// checkOrigPVCName fails when newName is already taken by a PVC this
1283+
// RestoreJob did not create. Proceeding would hand the live disk to the
1284+
// restore under its own name, as keepOriginalPVC=false does, while the -orig
1285+
// name holds some other disk. It returns the PVC this RestoreJob did create,
1286+
// or nil.
1287+
func (r *RestoreJobReconciler) checkOrigPVCName(ctx context.Context, restoreJob *backupsv1alpha1.RestoreJob, ns, newName string) (*corev1.PersistentVolumeClaim, error) {
1288+
existing := &corev1.PersistentVolumeClaim{}
1289+
if err := r.Get(ctx, client.ObjectKey{Namespace: ns, Name: newName}, existing); err != nil {
1290+
return nil, client.IgnoreNotFound(err)
1291+
}
1292+
if existing.Annotations[origPVCRestoreJobUIDAnnotation] != string(restoreJob.UID) {
1293+
return nil, fmt.Errorf("PVC %s/%s already exists and was not created by this RestoreJob, so the current disk cannot be kept under that name: "+
1294+
"remove that PVC or create the RestoreJob under another name", ns, newName)
1295+
}
1296+
return existing, nil
1297+
}
1298+
12571299
// renamePVC preserves an existing PVC by rebinding it under a new name.
12581300
// The original PVC is deleted and a new one pointing to the same PV is created.
1259-
// Missing resources are silently skipped (non-fatal).
1301+
// Missing resources are silently skipped (non-fatal). Each step can be
1302+
// resumed, so a rename interrupted by a controller restart completes on the
1303+
// next reconcile.
12601304
func (r *RestoreJobReconciler) renamePVC(ctx context.Context, restoreJob *backupsv1alpha1.RestoreJob, ns, oldName, newName string) error {
12611305
logger := getLogger(ctx)
12621306

1263-
// Check if already renamed
1264-
existingNew := &corev1.PersistentVolumeClaim{}
1265-
if err := r.Get(ctx, client.ObjectKey{Namespace: ns, Name: newName}, existingNew); err == nil {
1266-
return nil
1307+
newPVC, err := r.checkOrigPVCName(ctx, restoreJob, ns, newName)
1308+
if err != nil {
1309+
return err
12671310
}
1311+
resuming := newPVC != nil
12681312

1269-
// Get the original PVC
12701313
oldPVC := &corev1.PersistentVolumeClaim{}
12711314
if err := r.Get(ctx, client.ObjectKey{Namespace: ns, Name: oldName}, oldPVC); err != nil {
1272-
if errors.IsNotFound(err) {
1273-
return nil // nothing to rename
1315+
if !errors.IsNotFound(err) {
1316+
return err
12741317
}
1275-
return err
1318+
oldPVC = nil
12761319
}
12771320

1278-
pvName := oldPVC.Spec.VolumeName
1279-
if pvName == "" {
1321+
pv := &corev1.PersistentVolume{}
1322+
var pvName string
1323+
switch {
1324+
case resuming:
1325+
pvName = newPVC.Spec.VolumeName
1326+
case oldPVC == nil:
1327+
return nil // nothing to rename
1328+
case oldPVC.Spec.VolumeName == "":
12801329
logger.Debug("PVC not bound, deleting", "name", oldName)
12811330
return r.Delete(ctx, oldPVC)
1282-
}
1331+
default:
1332+
pvName = oldPVC.Spec.VolumeName
12831333

1284-
// Patch PV reclaim policy to Retain so it survives PVC deletion
1285-
pv := &corev1.PersistentVolume{}
1286-
if err := r.Get(ctx, client.ObjectKey{Name: pvName}, pv); err != nil {
1287-
return fmt.Errorf("failed to get PV %s: %w", pvName, err)
1288-
}
1289-
if pv.Spec.PersistentVolumeReclaimPolicy != corev1.PersistentVolumeReclaimRetain {
1290-
pv.Spec.PersistentVolumeReclaimPolicy = corev1.PersistentVolumeReclaimRetain
1291-
if err := r.Update(ctx, pv); err != nil {
1292-
return fmt.Errorf("failed to set Retain policy on PV %s: %w", pvName, err)
1334+
// Patch PV reclaim policy to Retain so it survives PVC deletion.
1335+
// A policy already recorded is the one from before any rename.
1336+
if err := r.Get(ctx, client.ObjectKey{Name: pvName}, pv); err != nil {
1337+
return fmt.Errorf("failed to get PV %s: %w", pvName, err)
1338+
}
1339+
originalPolicy, recorded := pv.Annotations[originalReclaimPolicyAnnotation]
1340+
if !recorded {
1341+
originalPolicy = string(pv.Spec.PersistentVolumeReclaimPolicy)
1342+
metav1.SetMetaDataAnnotation(&pv.ObjectMeta, originalReclaimPolicyAnnotation, originalPolicy)
1343+
}
1344+
if !recorded || pv.Spec.PersistentVolumeReclaimPolicy != corev1.PersistentVolumeReclaimRetain {
1345+
pv.Spec.PersistentVolumeReclaimPolicy = corev1.PersistentVolumeReclaimRetain
1346+
if err := r.Update(ctx, pv); err != nil {
1347+
return fmt.Errorf("failed to set Retain policy on PV %s: %w", pvName, err)
1348+
}
12931349
}
1294-
}
12951350

1296-
// Create the new -orig PVC first (unbound, just the object)
1297-
newPVC := &corev1.PersistentVolumeClaim{
1298-
ObjectMeta: metav1.ObjectMeta{
1299-
Name: newName,
1300-
Namespace: ns,
1301-
},
1302-
Spec: corev1.PersistentVolumeClaimSpec{
1303-
AccessModes: oldPVC.Spec.AccessModes,
1304-
Resources: oldPVC.Spec.Resources,
1305-
StorageClassName: oldPVC.Spec.StorageClassName,
1306-
VolumeMode: oldPVC.Spec.VolumeMode,
1307-
VolumeName: pvName,
1308-
},
1309-
}
1310-
if err := r.Create(ctx, newPVC); err != nil && !errors.IsAlreadyExists(err) {
1311-
return fmt.Errorf("failed to create -orig PVC %s: %w", newName, err)
1312-
}
1313-
// Re-read to get UID for the claimRef
1314-
if err := r.Get(ctx, client.ObjectKey{Namespace: ns, Name: newName}, newPVC); err != nil {
1315-
return fmt.Errorf("failed to get -orig PVC %s: %w", newName, err)
1351+
// Create the new -orig PVC first (unbound, just the object)
1352+
newPVC = &corev1.PersistentVolumeClaim{
1353+
ObjectMeta: metav1.ObjectMeta{
1354+
Name: newName,
1355+
Namespace: ns,
1356+
Annotations: map[string]string{
1357+
originalReclaimPolicyAnnotation: originalPolicy,
1358+
origPVCRestoreJobUIDAnnotation: string(restoreJob.UID),
1359+
},
1360+
},
1361+
Spec: corev1.PersistentVolumeClaimSpec{
1362+
AccessModes: oldPVC.Spec.AccessModes,
1363+
Resources: oldPVC.Spec.Resources,
1364+
StorageClassName: oldPVC.Spec.StorageClassName,
1365+
VolumeMode: oldPVC.Spec.VolumeMode,
1366+
VolumeName: pvName,
1367+
},
1368+
}
1369+
if err := r.Create(ctx, newPVC); err != nil {
1370+
return fmt.Errorf("failed to create -orig PVC %s: %w", newName, err)
1371+
}
1372+
// Re-read to get UID for the claimRef
1373+
if err := r.Get(ctx, client.ObjectKey{Namespace: ns, Name: newName}, newPVC); err != nil {
1374+
return fmt.Errorf("failed to get -orig PVC %s: %w", newName, err)
1375+
}
13161376
}
13171377

1318-
// Delete the original PVC
1319-
if err := r.Delete(ctx, oldPVC); err != nil && !errors.IsNotFound(err) {
1320-
return fmt.Errorf("failed to delete PVC %s: %w", oldName, err)
1378+
// Delete the original PVC. Only while it still claims the kept PV: after
1379+
// an interrupted rename, CDI may have recreated an empty one under the old
1380+
// name, and that one is left to the restore.
1381+
if oldPVC != nil && oldPVC.Spec.VolumeName == pvName {
1382+
if err := r.Delete(ctx, oldPVC); err != nil && !errors.IsNotFound(err) {
1383+
return fmt.Errorf("failed to delete PVC %s: %w", oldName, err)
1384+
}
13211385
}
13221386

13231387
// Point the PV's claimRef directly to the new -orig PVC.
13241388
// This is atomic — no window where the PV is Available for other PVCs to grab.
13251389
if err := r.Get(ctx, client.ObjectKey{Name: pvName}, pv); err != nil {
13261390
return fmt.Errorf("failed to re-fetch PV %s: %w", pvName, err)
13271391
}
1392+
if ref := pv.Spec.ClaimRef; ref != nil && ref.Namespace == ns && ref.Name == newName && ref.UID == newPVC.UID {
1393+
return nil
1394+
}
13281395
pv.Spec.ClaimRef = &corev1.ObjectReference{
13291396
APIVersion: "v1",
13301397
Kind: "PersistentVolumeClaim",

0 commit comments

Comments
 (0)