Skip to content

Commit 9ad9ac0

Browse files
[Backport release-1.6] fix(backups): refuse a foreign -orig PVC and record the kept disk's reclaim policy (#5079)
## What this PR does Backport of #4671 and #4672 to release-1.6. - #4672 builds on #4671, so both go in one PR, picked in order. - The test file needs the `errors` and `strings` imports on release-1.6, one conflicting test hunk in #4672 is resolved to the original's side. ### Screenshots Not a UI change. ### Downstream repositories The website v1.6 backup-and-recovery page tells users to delete `*-orig-*` PVCs to free storage, which a Retain PV does not do, and does not mention the new reclaim-policy annotation, I will open a website issue for it. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(backups): an in-place VM restore with keepOriginalPVC now fails, before the VM is halted, when the <disk>-orig-<hash> PVC it would keep the disk under already exists and was not created by this RestoreJob (for instance one left by a deleted RestoreJob of the same name), instead of skipping the rename and restoring without keeping the current disk. Any other failure to keep a disk, including a race in which the rename read its just-created -orig PVC back through the cache and failed, now holds the restore back and retries it before its DataVolume is deleted. Previously such a restore reported success without restoring the disk. The PV's reclaim policy before the switch to Retain is recorded in the backups.cozystack.io/original-reclaim-policy annotation on the PV and the -orig PVC; set it back on the PV before deleting the -orig PVC, or its volume is not reclaimed. fix(backups): an in-place VMInstance RestoreJob with keepOriginalPVC=false now deletes each original disk PVC, and waits for it to be gone, before Velero restores it. Previously the PVC was left in place, Velero skipped restoring its data, the RestoreJob still reported Succeeded, and the disk's HelmRelease could never recreate its DataVolume ("Destination PVC already exists"). A Velero RestoreJob now also fails before touching the application when its Velero Backup is gone or not Completed, for example after the Backup expired at its TTL. ```
2 parents 92dc76d + 8fcf668 commit 9ad9ac0

6 files changed

Lines changed: 904 additions & 112 deletions

File tree

‎examples/backups/vmi/scenario-user-restore.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,22 @@ The backup controller performs the following steps automatically:
3939
After the Velero Restore completes, the HelmReleases resume and the VM boots
4040
with the restored disk data while retaining its original IP and MAC addresses.
4141

42+
### Reclaiming the kept disks
43+
44+
With `keepOriginalPVC: true`, the default, step 3 switches each disk's PV to the `Retain` reclaim policy so that it outlives the PVC it is renamed from. Nothing removes the `-orig` PVCs or sets the policy back afterwards: they stay until an operator deletes them.
45+
46+
The policy the PV had before the restore is recorded in the `backups.cozystack.io/original-reclaim-policy` annotation, on the PV and on the `-orig` PVC. Set it back on the PV before deleting the `-orig` PVC, or a PV that was `Delete` stays `Retain` once released and its volume is never reclaimed. A PVC renamed by an older controller carries no annotation; the snippet then leaves the PV on `Retain` for you to decide.
47+
48+
```bash
49+
PVC=vm-disk-ubuntu-source-orig-1a2b
50+
PV=$(kubectl -n "$NAMESPACE" get pvc "$PVC" -o jsonpath='{.spec.volumeName}')
51+
POLICY=$(kubectl get pv "$PV" -o jsonpath='{.metadata.annotations.backups\.cozystack\.io/original-reclaim-policy}')
52+
kubectl patch pv "$PV" --type=merge -p "{\"spec\":{\"persistentVolumeReclaimPolicy\":\"${POLICY:-Retain}\"}}"
53+
kubectl -n "$NAMESPACE" delete pvc "$PVC"
54+
```
55+
56+
The `<hash>` is derived from the RestoreJob name, so an `-orig` PVC left behind by a deleted RestoreJob takes the name a new RestoreJob of the same name would use. That RestoreJob fails before it halts the VM, naming the PVC, rather than restoring without keeping the current disk. Reclaim the leftover PVC as above, or give the new RestoreJob another name.
57+
4258
## Method 2: Cross-Namespace Restore (Copy)
4359

4460
```bash

‎internal/backupcontroller/restorejob_controller.go‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,10 @@ type RestoreJobReconciler struct {
3737
Scheme *runtime.Scheme
3838
Recorder record.EventRecorder
3939
CredentialsConfig BackupCredentialsConfig
40+
// APIReader is the manager's uncached reader. The keepOriginalPVC rename
41+
// reads back objects it has just written, which the cache may not have
42+
// seen yet. Wired in SetupWithManager.
43+
APIReader client.Reader
4044
}
4145

4246
func (r *RestoreJobReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
@@ -160,6 +164,7 @@ func (r *RestoreJobReconciler) Reconcile(ctx context.Context, req ctrl.Request)
160164

161165
// SetupWithManager registers our controller with the Manager and sets up watches.
162166
func (r *RestoreJobReconciler) SetupWithManager(mgr ctrl.Manager) error {
167+
r.APIReader = mgr.GetAPIReader()
163168
cfg := mgr.GetConfig()
164169
var err error
165170
if r.Interface, err = dynamic.NewForConfig(cfg); err != nil {
@@ -177,6 +182,16 @@ func (r *RestoreJobReconciler) SetupWithManager(mgr ctrl.Manager) error {
177182
Complete(r)
178183
}
179184

185+
// apiReader returns the uncached reader. Reconcilers built by hand in unit
186+
// tests have none and fall back to their fake Client, which has no cache to
187+
// bypass.
188+
func (r *RestoreJobReconciler) apiReader() client.Reader {
189+
if r.APIReader != nil {
190+
return r.APIReader
191+
}
192+
return r.Client
193+
}
194+
180195
// handleProjectionError classifies a credentials-projection error as
181196
// transient (requeue with Ready=False/CredentialsProjectionPending) or
182197
// terminal (markRestoreJobFailed). Mirrors the BackupJob variant — both

0 commit comments

Comments
 (0)