Repository navigation
vm-instance: a change to the disk list unplugs hotplug volumes the chart does not render #4769
Description
Since #4453, a vm-instance release that changes its disk list can unplug a hotplug volume from the running VM. This hits any hotpluggable volume that sits in VM.spec but not in .Values.disks: one added with virtctl addvolume --persist, or one that migration 59 copied from the VMI into the VM spec during the upgrade.
The platform now runs KubeVirt with DeclarativeHotplugVolumes. virt-controller rebuilds a running VMI's hotplug volumes from VM.spec and removes every hotpluggable volume the VM spec does not list. Before the gate change it did not reconcile the VMI this way, so an edit to VM.spec.volumes left the VMI alone.
The chart renders spec.template.spec.volumes only from .Values.disks and the cloud-init secret (packages/apps/vm-instance/templates/vm.yaml:139-150 at bcda5c8). Helm patches a VirtualMachine with a JSON merge patch, and a merge patch replaces a list as a whole. While the rendered list is the same in the old and new release, the patch does not touch volumes and the extra entry stays. The first release that changes the list writes the chart's list over the live one. Adding or removing a disk is enough, and so is turning cloud-init or SSH keys on or off. After that, the declarative reconcile detaches the dropped volume from the running VM.
Tenant Kubernetes nodes are not affected. CAPK creates their VirtualMachines from a KubevirtMachineTemplate, and no Helm release renders them.
The upgrade notes in #4453 cover ephemeral hotplug becoming permanent, but not this. The minimum fix is one line in the release notes next to that part. A chart fix is also possible: look up the live VM and render back the hotpluggable volumes that .Values.disks does not list. The cost is that the chart then never removes such a volume. virtctl removevolume still does, since it edits the VM spec.
Raised in the review of #4453: #4453 (review)
Activity
myasnikovdaniil commentedon Oct 29, 2026
Hit this on an upgrade from v1.6.5 to v1.7.0-rc.2, and it fires in the platform upgrade itself, the disk list does not need to change.
VMInstance had a volume hot-plugged on the VMI only. Migration 59 copied it into
VM.specand its log says the VM was patched. Then the same platform upgrade moved the vm-instance release to the new chart, after that the VM managedFields showhelm-controllerwith operationApplyowningspec.template.spec.volumesand nokubectl-patchentry left. About 6 minutes later KubeVirt unmounted the volume from virt-launcher and deleted the attachment pod. VM spec and VMI ended withcloudinitdiskonly, the PVC stayed Bound.Rendered list was
[cloudinitdisk]before and after, so the merge patch reasoning in the description does not hold here. helm-controller applies the VirtualMachine with server-side apply, and the CRD has nox-kubernetes-list-typeonspec.template.spec.volumes, so the list is atomic and an upgrade of the release writes the chart's list over the live one even when the rendered list did not change. The 1.7 upgrade moves every vm-instance release to the new chart, so on a VMInstance migration 59's copy is undone in the same upgrade.At rc.3
packages/apps/vm-instance/templates/vm.yamland migration 59 are the same as in rc.2, main too, so this is still open. I did not run this path again on rc.3 and did not check the effect inside the guest (nothing was mounted there). v1.7.0 upgrade notes have a manual step for VMInstance, the chart fix proposed here is tracked for 1.7.1 in #5165