Skip to content

[Backport release-1.6] fix(backups): refuse a foreign -orig PVC and record the kept disk's reclaim policy - #5079

Merged
myasnikovdaniil merged 4 commits into
release-1.6from
backport/4671-vm-restore-release-1.6
Oct 23, 2026
Merged

myasnikovdaniil merged 4 commits into
release-1.6from
backport/4671-vm-restore-release-1.6

Conversation

@myasnikovdaniil

Copy link
Copy Markdown
Contributor

What this PR does

Backport of #4671 and #4672 to release-1.6.

Screenshots

Not a UI change.

Downstream repositories

The website v1.6 backup-and-recovery page tells users to delete *-orig-* PVCs to free storage, which a Retain PV does not do, and does not mention the new reclaim-policy annotation, I will open a website issue for it.

Release note

fix(backups): an in-place VM restore with keepOriginalPVC now fails, before the VM is halted, when the <disk>-orig-<hash> PVC it would keep the disk under already exists and was not created by this RestoreJob (for instance one left by a deleted RestoreJob of the same name), instead of skipping the rename and restoring without keeping the current disk. Any other failure to keep a disk, including a race in which the rename read its just-created -orig PVC back through the cache and failed, now holds the restore back and retries it before its DataVolume is deleted. Previously such a restore reported success without restoring the disk. The PV's reclaim policy before the switch to Retain is recorded in the backups.cozystack.io/original-reclaim-policy annotation on the PV and the -orig PVC; set it back on the PV before deleting the -orig PVC, or its volume is not reclaimed.

fix(backups): an in-place VMInstance RestoreJob with keepOriginalPVC=false now deletes each original disk PVC, and waits for it to be gone, before Velero restores it. Previously the PVC was left in place, Velero skipped restoring its data, the RestoreJob still reported Succeeded, and the disk's HelmRelease could never recreate its DataVolume ("Destination PVC already exists"). A Velero RestoreJob now also fails before touching the application when its Velero Backup is gone or not Completed, for example after the Backup expired at its TTL.

…eclaim policy

renamePVC returned early whenever a PVC named <dv>-orig-<hash> already
existed, without checking it came from this rename. The hash covers
only the RestoreJob name, so a RestoreJob created under the name of a
deleted one finds the -orig PVC its predecessor left behind. The rename
was skipped, the DataVolume deleted anyway, and the restore ran as if
keepOriginalPVC were false while reporting success: the live disk was
not kept, and the -orig name held an older one. Any other rename
failure ended the same way, since its error only became an event.

A -orig PVC now records the UID of the RestoreJob that created it. A
name held by any other PVC fails the restore before the VM is halted,
an interrupted rename of our own resumes instead of stopping halfway,
and a rename failure fails the restore before the DataVolume goes.

The PV is switched to Retain so it outlives the original PVC, and the
policy it had was discarded. Nothing in the controller deletes the
-orig PVC or restores that policy, so a PV that was Delete stayed
Retain and its volume leaked once an operator removed the claim. The
policy is now recorded on the PV and on the -orig PVC, and the restore
scenario documents how to set it back.

On release-1.6 the test file also imports strings and the apimachinery
errors package, which main's copy had already gained from earlier
commits that are not on this branch.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <[email protected]>
(cherry picked from commit d8d9608)
Signed-off-by: Myasnikov Daniil <[email protected]>
renamePVC created the -orig PVC, then read it back through the cached
client to learn its UID. The informer had often not seen the PVC yet, so
the read failed with NotFound. On a two-disk VM restore one disk was
kept and the other was not; with that error turned into an event, its
DataVolume was deleted, the original PVC stayed in place and Velero
skipped it, and the restore reported success without restoring the
disk. The -orig PVC was left Pending on the same PV, by then Retain.

Create already returns the UID, so the read-back goes. The other reads
the rename decides on, the -orig and original PVCs and the PV, go
through the manager's uncached reader, and the PV rebind retries on
conflict, since the PV controller writes the PV once the original PVC
is deleted.

A rename that fails for any reason other than an -orig name taken by
another PVC now requeues the restore instead of failing it. The
DataVolume is still there, and the retry finds its own -orig PVC by the
RestoreJob UID and resumes where it stopped.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <[email protected]>
(cherry picked from commit cc7f981)
Signed-off-by: Myasnikov Daniil <[email protected]>
…ut keepOriginalPVC

With keepOriginalPVC=false the restore deleted each DataVolume with
Orphan propagation and left its PVC in place for Velero to overwrite.
Velero never overwrites a PVC: its CSI restore action skips the data
mover for a claim that already exists, and existingResourcePolicy=update
fails to patch the immutable spec of a bound claim, so it only stamps
its labels on it. The RestoreJob then reports Succeeded with the disk
still holding its pre-restore data.

The claim left behind also breaks the disk for good. It has lost its
owner reference to the orphaned DataVolume, and it lacks the
cdi.kubevirt.io/storage.populatedFor annotation that the backed-up copy
carries, so the CDI webhook refuses the DataVolume the resumed
HelmRelease creates ("Destination PVC already exists"), on every
reconcile.

Delete the PVC once its DataVolume is gone, and hold the Velero Restore
until the claim has actually disappeared: one still mounted stays
Terminating behind pvc-protection, and Velero would take it for an
existing claim again.

The test file conflicted on release-1.6 only in the line the new helper
replaces, which this branch spells boolPtr(false) where main spells
new(false); the helper and both tests are taken as they are on main.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <[email protected]>
(cherry picked from commit 6ea713d)
Signed-off-by: Myasnikov Daniil <[email protected]>
Deleting the original PVC of a keepOriginalPVC=false restore is only
safe when Velero can put the disk back. The Backup artifact outlives
the Velero Backup it points to, which Velero garbage-collects at TTL,
and nothing checked it before prepareForRestore halted the VM and
deleted the disks: a restore of an expired backup lost the volume where
it used to fail with the disk intact. The Velero Backup must now exist
and be Completed before any preparation starts.

A DataVolume that survives its deletion makes CDI recreate the claim
under the original name, which Velero then skips as an existing PVC,
so the restore would end Succeeded on a freshly imported disk. A failed
DataVolume deletion now fails the restore, and the PVCs are deleted
only once every DataVolume is gone. Reading the DataVolume back needs
get on datavolumes, which the ClusterRole did not grant.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <[email protected]>
(cherry picked from commit 6d064a8)
Signed-off-by: Myasnikov Daniil <[email protected]>
@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) area/storage Issues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor) kind/bug Categorizes issue or PR as related to a bug size/XXL This PR changes 1000+ lines, ignoring generated files labels Oct 22, 2026
@myasnikovdaniil
myasnikovdaniil marked this pull request as ready for review October 23, 2026 17:40
@myasnikovdaniil
myasnikovdaniil merged commit 9ad9ac0 into release-1.6 Oct 23, 2026
78 of 80 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the backport/4671-vm-restore-release-1.6 branch October 23, 2026 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) area/storage Issues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor) kind/bug Categorizes issue or PR as related to a bug size/XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants