You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
All existing vms live migrations keep failing after upgrading from kubevirt 1.6.2/cdi 1.6.3 to kubevirt 1.7/cdi 1.6.4 #16386
What happened:
All existing vms live migrations keep failing after upgrading from kubevirt 1.6.2/cdi 1.6.3 to kubevirt 1.7/cdi 1.6.4
What you expected to happen:
All existing VMs live migrations should not fail after kubevirt/cdi upgrade!
How to reproduce it (as minimally and precisely as possible):
Steps to reproduce the behavior.
Just do upgrade from an existing kubevirt cluster with running VMs after kubevirt 1.6.2/cdi 1.6.3 to kubevirt 1.7/cdi 1.6.4.
Additional context:
Error logs on this:
{"component":"virt-launcher-monitor","level":"info","msg":"2025-12-16T22:06:36.625034Z qemu-kvm: Features 0x1c0010130afffaf unsupported. Allowed features: 0x10179bfffef","pos":"virt-launcher-monitor.go:393","timestamp":"2025-12-16T22:06:37.498462Z"}
{"component":"virt-launcher-monitor","level":"info","msg":"2025-12-16T22:06:36.625083Z qemu-kvm: Failed to load virtio-net:virtio","pos":"virt-launcher-monitor.go:393","timestamp":"2025-12-16T22:06:37.498467Z"}
{"component":"virt-launcher-monitor","level":"info","msg":"2025-12-16T22:06:36.625087Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:02.0:00.0/virtio-net'","pos":"virt-launcher-monitor.go:393","timestamp":"2025-12-16T22:06:37.498472Z"}
{"component":"virt-launcher-monitor","level":"info","msg":"2025-12-16T22:06:36.630445Z qemu-kvm: load of migration failed: Operation not permitted","pos":"virt-launcher-monitor.go:393","timestamp":"2025-12-16T22:06:37.498476Z"}
All existing running VMs live migration were working before this upgrade. VMs using multus and bridge CNI only, VMs only having one interface via multus/bridge to talk to external network.
If creating a new vm, then live migration is working ok as expected. Or if restarting existing VM, then live migration after that is working ok for the restarted vm.Is this due to bug from old qemu-kvm-9.1.0-19.el9 (with kubevirt 1.6.2) to newer qemu-kvm-9.1.0-20.el9 (with kubevirt 1.7.0)?
Environment:
KubeVirt version (use virtctl version): N/A
Kubernetes version (use kubectl version): N/A
VM or VMI specifications: N/A
Cloud provider or hardware configuration: N/A
OS (e.g. from /etc/os-release): N/A
Kernel (e.g. uname -a): N/A
Install tools: N/A
Others: N/A
Reacted by dosubot, Nikita, Vadim Zharov, Andrei Kvapil, Waleed Malik and Moep90
This is a known limitation after a QEMU version upgrade in KubeVirt: when you upgrade from KubeVirt 1.6.2 (QEMU 9.1.0-19.el9) to 1.7 (QEMU 9.1.0-20.el9), live migration of VMs that were running before the upgrade can fail with errors like "Failed to load virtio-net" and "Operation not permitted". This happens because the in-memory state of a running VM is tied to the QEMU version it started with, and after the upgrade, the new QEMU can't always interpret the old state, especially for devices like virtio-net. This is not specific to your network configuration or CNI plugin.
Restarting the affected VMs after the upgrade reinitializes their state with the new QEMU version, which resolves the migration issue. New VMs started after the upgrade are also unaffected. There is no automated workaround in KubeVirt for this scenario, and it's not currently documented in the release notes or user guide [source].
To mitigate: after upgrading KubeVirt and QEMU, restart all running VMs before attempting live migration.
If you have further questions or need help with related troubleshooting, let me know!
Unfortunately no, I had to reboot all VMs, which is not good at all.
Simply upgrading kubevirt should not force users to reboot all their VMs. This should be a critical e2e test case before a new kubevirt version is ever being released to the public.
Cozystack maintainer here. We're hitting the same issue when upgrading KubeVirt from v1.6.3 to v1.8.2 (we're skipping v1.7.x because it's blocked on this same issue).
Reproduced on a test cluster: any VM that was running before the upgrade fails to live-migrate after the operator rolls out the new virt-launcher. KubeVirt's workloadUpdateMethods: [LiveMigrate, Evict] triggers the migrations automatically, and every one of them fails the same way:
virError(Code=9, Domain=10, Message='operation failed: job 'migration in' failed:
load of migration failed: Operation not permitted')
qemu-kvm: error while loading state for instance 0x0 of device '0000:00:02.0:00.0/virtio-net'
Source machine type at the time of failure: pc-q35-rhel9.6.0.
The only workaround we have for our users is cold-restarting affected VMs (virtctl restart or kubectl delete pod virt-launcher-...) — works, but means downtime for every running VM in the cluster, which is a hard sell for production tenants.
What would actually help:
A documented step-up path (1.6 → some intermediate that still uses the old QEMU device version — 1.7 in theory, but it's blocked on this very issue).
A way to pin the QEMU/virt-launcher image at the existing version on already-running VMs while still rolling out the new control plane (so live-migration keeps working until VMs are restarted naturally).
Or, ideally, a fix on the QEMU side that makes virtio-net device state version-tolerant across qemu-9.1.0-19.el9 → qemu-9.1.0-20.el9.
Happy to help test any proposed fix. We're blocking the upgrade on our side until there's a clearer story for production clusters with running VMs.
I'm working on testing upgrade from 1.5.0 up to 1.7.4 (as kubenertes 1.32 would block newer kubevirt versions) I am hitting the same problem going from 1.5.0 > 1.7.4 or from 1.6.6 > 1.7.4 show the same failure:
qemu-kvm: Features 0x1c0010130afffaf unsupported. Allowed features: 0x10179bfffef
qemu-kvm: Failed to load virtio-net:virtio
qemu-kvm: error while loading state for instance 0x0 of device 'virtio-net'
qemu-kvm: load of migration failed: Operation not permitted
Restarting 3500 workload VMs is a clear "show stopper".
Is there any possibility something could be done to avoid the workload restart?
My suggestion would be to roll your own virt-launcher image that strips the patch that disables USO, see #16386 (comment). This way you should end up with a QEMU that is able to pick up the existing VM.
@morete which host kernel are you running on? We have a feeling you're encountering this behaviour as you're using a non EL kernel while running EL userspace within virt-launcher.
@morete Addition to my previous suggestion: If you want to move to v1.7, you require a modified virt-launcher image, which re-enables USO as a temporary solution. Once you can move to v1.10+ (not released yet), you should be able to run the CS10 userspace, which also has support for USO. So with v1.10+ and the CS10 userspace, you should be able to revert back to the release images shipped by KubeVirt.
USO (UDP Segmentation Offload) support for both TAP (TUN_F_USO4/USO6) and virtio-net was introduced upstream in Linux 6.2:
Ubuntu 24.04 / 26.04: Ship with kernel $\ge$ 6.8 by default.
Ubuntu 22.04: Supports USO if running the HWE kernel ($\ge$ 6.2, e.g. 6.8 via linux-generic-hwe-22.04), though not on the original 5.15 GA kernel.
Why this caused the failure
During VM startup, QEMU probes the host kernel via ioctl(fd, TUNSETOFFLOAD, ...).
On your Ubuntu nodes (kernel $\ge$ 6.2), this probe succeeded, so QEMU offered virtio-net USO to the guest, and guests running $\ge$ 6.2 negotiated it. This set bits 54, 55, and 56 (VIRTIO_NET_F_GUEST_USO4, VIRTIO_NET_F_GUEST_USO6, and VIRTIO_NET_F_HOST_USO).
On EL9 hosts (which our CI and testing use), the kernel is 5.14 where TAP USO does not exist. The probe fails, so QEMU never advertised or enabled USO on EL9 nodes.
When upgrading to KubeVirt 1.7+, downstream QEMU patch RHEL-80313 (kvm-virtio-net-disable-USO-for-virt-rhel9.6.patch) retroactively disabled USO on pc-q35-rhel9.6.0 machine types. Target virt-launchers therefore reject the incoming migration containing bits 54–56 (the exact 0x1c0000000000000 diff in your error).
Because CI runs on EL9 kernels where USO was never negotiated, our upgrade tests did not hit this regression.
Follow-up & handling the host/userspace matrix
This highlights the underlying friction when running EL userspace inside virt-launcher on non-EL host kernels with newer guest OSes.
To start bringing transparency to these assumptions and boundaries, I've started documenting the kernel and userspace dependencies across several issues and PRs:
Longer term, as a community we need to figure out how to realistically handle this complexity. Upstream release images are built and validated against EL userspace and host kernels; if users or distributions rely on different host kernels or mismatched stacks, folks who care about those combinations will likely need to step up with CI infrastructure or engineering capacity to ship and validate aligned virt stacks for their environments.
@lyarwood I made a little mistake, on the environment/cluster where I tested this upgrade we ONLY have Ubuntu 26.04 on kernel 7. All VMs I used to test were created under kubevirt 1.5.0 on U26 and kernel 7.
Activity
dosubot commentedon Dec 19, 2025
This is a known limitation after a QEMU version upgrade in KubeVirt: when you upgrade from KubeVirt 1.6.2 (QEMU 9.1.0-19.el9) to 1.7 (QEMU 9.1.0-20.el9), live migration of VMs that were running before the upgrade can fail with errors like "Failed to load virtio-net" and "Operation not permitted". This happens because the in-memory state of a running VM is tied to the QEMU version it started with, and after the upgrade, the new QEMU can't always interpret the old state, especially for devices like virtio-net. This is not specific to your network configuration or CNI plugin.
Restarting the affected VMs after the upgrade reinitializes their state with the new QEMU version, which resolves the migration issue. New VMs started after the upgrade are also unaffected. There is no automated workaround in KubeVirt for this scenario, and it's not currently documented in the release notes or user guide [source].
To mitigate: after upgrading KubeVirt and QEMU, restart all running VMs before attempting live migration.
If you have further questions or need help with related troubleshooting, let me know!
To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
nbykov0 commentedon Jan 9, 2026
Same here.
@haiwu did you manage to solve it somehow or the bot is correct and the only way is to reboot all VMs?
haiwu commentedon Jan 9, 2026
Unfortunately no, I had to reboot all VMs, which is not good at all.
Simply upgrading kubevirt should not force users to reboot all their VMs. This should be a critical e2e test case before a new kubevirt version is ever being released to the public.
lyarwood commentedon Jan 20, 2026
/sig compute
/priority critical-urgent
fossedihelm commentedon Jan 21, 2026
@haiwu Can you share the VM spec? Thanks
haiwu commentedon Jan 21, 2026
@fossedihelm : The VM spec is similar to the following:
iholder101 commentedon Jan 21, 2026
/assign @fossedihelm
stefanSpectro commentedon Mar 31, 2026
Hi just as note we are running in to this as well.
kvaps commentedon Apr 27, 2026
Cozystack maintainer here. We're hitting the same issue when upgrading KubeVirt from
v1.6.3tov1.8.2(we're skippingv1.7.xbecause it's blocked on this same issue).Reproduced on a test cluster: any VM that was running before the upgrade fails to live-migrate after the operator rolls out the new virt-launcher. KubeVirt's
workloadUpdateMethods: [LiveMigrate, Evict]triggers the migrations automatically, and every one of them fails the same way:Source machine type at the time of failure:
pc-q35-rhel9.6.0.The only workaround we have for our users is cold-restarting affected VMs (
virtctl restartorkubectl delete pod virt-launcher-...) — works, but means downtime for every running VM in the cluster, which is a hard sell for production tenants.What would actually help:
virtio-netdevice state version-tolerant acrossqemu-9.1.0-19.el9→qemu-9.1.0-20.el9.Happy to help test any proposed fix. We're blocking the upgrade on our side until there's a clearer story for production clusters with running VMs.
aburdenthehand commentedon Apr 29, 2026
/cc @vladikr
14 remaining items
morete commentedon Sep 1, 2026
I'm working on testing upgrade from 1.5.0 up to 1.7.4 (as kubenertes 1.32 would block newer kubevirt versions) I am hitting the same problem going from 1.5.0 > 1.7.4 or from 1.6.6 > 1.7.4 show the same failure:
Restarting 3500 workload VMs is a clear "show stopper".
Is there any possibility something could be done to avoid the workload restart?
0xFelix commentedon Sep 2, 2026
My suggestion would be to roll your own
virt-launcherimage that strips the patch that disables USO, see #16386 (comment). This way you should end up with a QEMU that is able to pick up the existing VM.lyarwood commentedon Sep 2, 2026
@morete which host kernel are you running on? We have a feeling you're encountering this behaviour as you're using a non EL kernel while running EL userspace within virt-launcher.
0xFelix commentedon Sep 2, 2026
@morete Addition to my previous suggestion: If you want to move to v1.7, you require a modified
virt-launcherimage, which re-enables USO as a temporary solution. Once you can move to v1.10+ (not released yet), you should be able to run the CS10 userspace, which also has support for USO. So with v1.10+ and the CS10 userspace, you should be able to revert back to the release images shipped by KubeVirt.chore(kubevirt): update KubeVirt to v1.9.0
morete commentedon Sep 2, 2026
@lyarwood hosts are on a mix of Ubuntu 22.04 and Ubuntu 26.04, guest VMs also run Ubuntu 22 or 24.
lyarwood commentedon Sep 2, 2026
Thanks @morete, that confirms what we suspected.
USO (UDP Segmentation Offload) support for both TAP (
TUN_F_USO4/USO6) andvirtio-netwas introduced upstream in Linux 6.2:linux-generic-hwe-22.04), though not on the original 5.15 GA kernel.Why this caused the failure
During VM startup, QEMU probes the host kernel via
ioctl(fd, TUNSETOFFLOAD, ...).VIRTIO_NET_F_GUEST_USO4,VIRTIO_NET_F_GUEST_USO6, andVIRTIO_NET_F_HOST_USO).When upgrading to KubeVirt 1.7+, downstream QEMU patch RHEL-80313 (
kvm-virtio-net-disable-USO-for-virt-rhel9.6.patch) retroactively disabled USO onpc-q35-rhel9.6.0machine types. Target virt-launchers therefore reject the incoming migration containing bits 54–56 (the exact0x1c0000000000000diff in your error).Because CI runs on EL9 kernels where USO was never negotiated, our upgrade tests did not hit this regression.
Follow-up & handling the host/userspace matrix
This highlights the underlying friction when running EL userspace inside
virt-launcheron non-EL host kernels with newer guest OSes.To start bringing transparency to these assumptions and boundaries, I've started documenting the kernel and userspace dependencies across several issues and PRs:
Longer term, as a community we need to figure out how to realistically handle this complexity. Upstream release images are built and validated against EL userspace and host kernels; if users or distributions rely on different host kernels or mismatched stacks, folks who care about those combinations will likely need to step up with CI infrastructure or engineering capacity to ship and validate aligned virt stacks for their environments.
chore(kubevirt): update KubeVirt to v1.9.0
installation: document host kernel / EL userland compatibility (#1029)
morete commentedon Sep 4, 2026
@lyarwood I made a little mistake, on the environment/cluster where I tested this upgrade we ONLY have Ubuntu 26.04 on kernel 7. All VMs I used to test were created under kubevirt 1.5.0 on U26 and kernel 7.
chore(kubevirt): update KubeVirt to v1.9.0 (#4015)
fix(kubevirt): pause automatic workload updates (#166)
chore(kubevirt): step 3/4 of #121 — v1.8.4, CDI v1.65.0 (#167)
add the USO live-migration failure to the 1.3 upgrade guide and known…