Skip to content

All existing vms live migrations keep failing after upgrading from kubevirt 1.6.2/cdi 1.6.3 to kubevirt 1.7/cdi 1.6.4 #16386

Open

Description

@haiwu

What happened:
All existing vms live migrations keep failing after upgrading from kubevirt 1.6.2/cdi 1.6.3 to kubevirt 1.7/cdi 1.6.4

What you expected to happen:
All existing VMs live migrations should not fail after kubevirt/cdi upgrade!

How to reproduce it (as minimally and precisely as possible):
Steps to reproduce the behavior.
Just do upgrade from an existing kubevirt cluster with running VMs after kubevirt 1.6.2/cdi 1.6.3 to kubevirt 1.7/cdi 1.6.4.

Additional context:
Error logs on this:

{"component":"virt-launcher-monitor","level":"info","msg":"2025-12-16T22:06:36.625034Z qemu-kvm: Features 0x1c0010130afffaf unsupported. Allowed features: 0x10179bfffef","pos":"virt-launcher-monitor.go:393","timestamp":"2025-12-16T22:06:37.498462Z"}
{"component":"virt-launcher-monitor","level":"info","msg":"2025-12-16T22:06:36.625083Z qemu-kvm: Failed to load virtio-net:virtio","pos":"virt-launcher-monitor.go:393","timestamp":"2025-12-16T22:06:37.498467Z"}
{"component":"virt-launcher-monitor","level":"info","msg":"2025-12-16T22:06:36.625087Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:02.0:00.0/virtio-net'","pos":"virt-launcher-monitor.go:393","timestamp":"2025-12-16T22:06:37.498472Z"}
{"component":"virt-launcher-monitor","level":"info","msg":"2025-12-16T22:06:36.630445Z qemu-kvm: load of migration failed: Operation not permitted","pos":"virt-launcher-monitor.go:393","timestamp":"2025-12-16T22:06:37.498476Z"}

All existing running VMs live migration were working before this upgrade. VMs using multus and bridge CNI only, VMs only having one interface via multus/bridge to talk to external network.

If creating a new vm, then live migration is working ok as expected. Or if restarting existing VM, then live migration after that is working ok for the restarted vm.Is this due to bug from old qemu-kvm-9.1.0-19.el9 (with kubevirt 1.6.2) to newer qemu-kvm-9.1.0-20.el9 (with kubevirt 1.7.0)?

Environment:

  • KubeVirt version (use virtctl version): N/A
  • Kubernetes version (use kubectl version): N/A
  • VM or VMI specifications: N/A
  • Cloud provider or hardware configuration: N/A
  • OS (e.g. from /etc/os-release): N/A
  • Kernel (e.g. uname -a): N/A
  • Install tools: N/A
  • Others: N/A

Activity

  1. dosubot commented on Dec 19, 2025

    @dosubot

    This is a known limitation after a QEMU version upgrade in KubeVirt: when you upgrade from KubeVirt 1.6.2 (QEMU 9.1.0-19.el9) to 1.7 (QEMU 9.1.0-20.el9), live migration of VMs that were running before the upgrade can fail with errors like "Failed to load virtio-net" and "Operation not permitted". This happens because the in-memory state of a running VM is tied to the QEMU version it started with, and after the upgrade, the new QEMU can't always interpret the old state, especially for devices like virtio-net. This is not specific to your network configuration or CNI plugin.

    Restarting the affected VMs after the upgrade reinitializes their state with the new QEMU version, which resolves the migration issue. New VMs started after the upgrade are also unaffected. There is no automated workaround in KubeVirt for this scenario, and it's not currently documented in the release notes or user guide [source].

    To mitigate: after upgrading KubeVirt and QEMU, restart all running VMs before attempting live migration.

    If you have further questions or need help with related troubleshooting, let me know!

    To reply, just mention @dosu.


    Share context across your team and agents. Try Dosu.

    Leave Feedback Learn about kubevirt with Dosu Add Dosu to your team

  2. nbykov0 commented on Jan 9, 2026

    @nbykov0

    Same here.
    @haiwu did you manage to solve it somehow or the bot is correct and the only way is to reboot all VMs?

  3. haiwu commented on Jan 9, 2026

    @haiwu
    Author

    Unfortunately no, I had to reboot all VMs, which is not good at all.

    Simply upgrading kubevirt should not force users to reboot all their VMs. This should be a critical e2e test case before a new kubevirt version is ever being released to the public.

  4. lyarwood commented on Jan 20, 2026

    @lyarwood
    Member

    /sig compute
    /priority critical-urgent

  5. fossedihelm commented on Jan 21, 2026

    @fossedihelm
    Contributor

    @haiwu Can you share the VM spec? Thanks

  6. haiwu commented on Jan 21, 2026

    @haiwu
    Author

    @fossedihelm : The VM spec is similar to the following:

    apiVersion: kubevirt.io/v1
    kind: VirtualMachine
    metadata:
      labels:
        kubevirt.io/os: linux
        vm: testvm1
      annotations:
        kubevirt.io/allow-pod-bridge-network-live-migration: "true"
        k8s.v1.cni.cncf.io/networks: default/bridge-vlan100
      name: testvm1
      namespace: testns1
    spec:
      updateVolumesStrategy: Migration
      instancetype:
        name: insttype1
      preference:
        name: ubuntu
      dataVolumeTemplates:
      - metadata:
          name: testvm1-rootdisk
          labels:
            vm: testvm1
        spec:
          source:
            snapshot:
              name: ubuntu-001
              namespace: testns1
          storage:
            storageClassName: ceph-rbd
            resources:
              requests:
                storage: 250Gi
      runStrategy: RerunOnFailure
      template:
        metadata:
          labels:
            kubevirt.io/domain: testvm1
            vm: testvm1
          annotations:
            descheduler.alpha.kubernetes.io/evict: "true"
            hooks.kubevirt.io/hookSidecars: |
              [
                {"args": ["--version", "v1alpha3"], "configMap": {"name": "serial-hook","key": "vm_serial_number.py", "hookPath": "/usr/bin/onDefineDomain"}, "image": "internal.com/proxy.quay.io/kubevirt/sidecar-shim:v1.5.2"}
              ]
        spec:
          affinity: {}
          evictionStrategy: LiveMigrate
          domain:
            devices:
              disks:
              - disk:
                  bus: virtio
                name: root-disk
              - cdrom:
                  bus: sata
                  readonly: true
                name: cloudinitdisk
              interfaces:
              - name: eth0
                bridge: {}
              rng: {}
            machine:
              type: q35
          networks:
          - multus:
              networkName: default/bridge-vlan100
            name: eth0
          volumes:
          - name: root-disk
            dataVolume:
              name: testvm1-rootdisk
          - name: cloudinitdisk
            cloudInitNoCloud:
              userData: |-
                #cloud-config
                disable_root: false
    
  7. iholder101 commented on Jan 21, 2026

    @iholder101
    Contributor

    /assign @fossedihelm

  8. stefanSpectro commented on Mar 31, 2026

    @stefanSpectro

    Hi just as note we are running in to this as well.

  9. kvaps commented on Apr 27, 2026

    @kvaps
    Member

    Cozystack maintainer here. We're hitting the same issue when upgrading KubeVirt from v1.6.3 to v1.8.2 (we're skipping v1.7.x because it's blocked on this same issue).

    Reproduced on a test cluster: any VM that was running before the upgrade fails to live-migrate after the operator rolls out the new virt-launcher. KubeVirt's workloadUpdateMethods: [LiveMigrate, Evict] triggers the migrations automatically, and every one of them fails the same way:

    virError(Code=9, Domain=10, Message='operation failed: job 'migration in' failed:
      load of migration failed: Operation not permitted')
    qemu-kvm: error while loading state for instance 0x0 of device '0000:00:02.0:00.0/virtio-net'
    

    Source machine type at the time of failure: pc-q35-rhel9.6.0.

    The only workaround we have for our users is cold-restarting affected VMs (virtctl restart or kubectl delete pod virt-launcher-...) — works, but means downtime for every running VM in the cluster, which is a hard sell for production tenants.

    What would actually help:

    1. A documented step-up path (1.6 → some intermediate that still uses the old QEMU device version — 1.7 in theory, but it's blocked on this very issue).
    2. A way to pin the QEMU/virt-launcher image at the existing version on already-running VMs while still rolling out the new control plane (so live-migration keeps working until VMs are restarted naturally).
    3. Or, ideally, a fix on the QEMU side that makes virtio-net device state version-tolerant across qemu-9.1.0-19.el9 → qemu-9.1.0-20.el9.

    Happy to help test any proposed fix. We're blocking the upgrade on our side until there's a clearer story for production clusters with running VMs.

  10. aburdenthehand commented on Apr 29, 2026

    @aburdenthehand
    Member
  11. 14 remaining items

  12. removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Aug 25, 2026
  13. morete commented on Sep 1, 2026

    @morete

    I'm working on testing upgrade from 1.5.0 up to 1.7.4 (as kubenertes 1.32 would block newer kubevirt versions) I am hitting the same problem going from 1.5.0 > 1.7.4 or from 1.6.6 > 1.7.4 show the same failure:

    qemu-kvm: Features 0x1c0010130afffaf unsupported. Allowed features: 0x10179bfffef
    qemu-kvm: Failed to load virtio-net:virtio
    qemu-kvm: error while loading state for instance 0x0 of device 'virtio-net'
    qemu-kvm: load of migration failed: Operation not permitted
    

    Restarting 3500 workload VMs is a clear "show stopper".

    Is there any possibility something could be done to avoid the workload restart?

  14. 0xFelix commented on Sep 2, 2026

    @0xFelix
    Member

    My suggestion would be to roll your own virt-launcher image that strips the patch that disables USO, see #16386 (comment). This way you should end up with a QEMU that is able to pick up the existing VM.

  15. lyarwood commented on Sep 2, 2026

    @lyarwood
    Member

    @morete which host kernel are you running on? We have a feeling you're encountering this behaviour as you're using a non EL kernel while running EL userspace within virt-launcher.

  16. 0xFelix commented on Sep 2, 2026

    @0xFelix
    Member

    @morete Addition to my previous suggestion: If you want to move to v1.7, you require a modified virt-launcher image, which re-enables USO as a temporary solution. Once you can move to v1.10+ (not released yet), you should be able to run the CS10 userspace, which also has support for USO. So with v1.10+ and the CS10 userspace, you should be able to revert back to the release images shipped by KubeVirt.

  17. morete commented on Sep 2, 2026

    @morete

    @lyarwood hosts are on a mix of Ubuntu 22.04 and Ubuntu 26.04, guest VMs also run Ubuntu 22 or 24.

  18. lyarwood commented on Sep 2, 2026

    @lyarwood
    Member

    Thanks @morete, that confirms what we suspected.

    USO (UDP Segmentation Offload) support for both TAP (TUN_F_USO4/USO6) and virtio-net was introduced upstream in Linux 6.2:

    • Ubuntu 24.04 / 26.04: Ship with kernel $\ge$ 6.8 by default.
    • Ubuntu 22.04: Supports USO if running the HWE kernel ($\ge$ 6.2, e.g. 6.8 via linux-generic-hwe-22.04), though not on the original 5.15 GA kernel.

    Why this caused the failure

    During VM startup, QEMU probes the host kernel via ioctl(fd, TUNSETOFFLOAD, ...).

    • On your Ubuntu nodes (kernel $\ge$ 6.2), this probe succeeded, so QEMU offered virtio-net USO to the guest, and guests running $\ge$ 6.2 negotiated it. This set bits 54, 55, and 56 (VIRTIO_NET_F_GUEST_USO4, VIRTIO_NET_F_GUEST_USO6, and VIRTIO_NET_F_HOST_USO).
    • On EL9 hosts (which our CI and testing use), the kernel is 5.14 where TAP USO does not exist. The probe fails, so QEMU never advertised or enabled USO on EL9 nodes.

    When upgrading to KubeVirt 1.7+, downstream QEMU patch RHEL-80313 (kvm-virtio-net-disable-USO-for-virt-rhel9.6.patch) retroactively disabled USO on pc-q35-rhel9.6.0 machine types. Target virt-launchers therefore reject the incoming migration containing bits 54–56 (the exact 0x1c0000000000000 diff in your error).

    Because CI runs on EL9 kernels where USO was never negotiated, our upgrade tests did not hit this regression.

    Follow-up & handling the host/userspace matrix

    This highlights the underlying friction when running EL userspace inside virt-launcher on non-EL host kernels with newer guest OSes.

    To start bringing transparency to these assumptions and boundaries, I've started documenting the kernel and userspace dependencies across several issues and PRs:

    Longer term, as a community we need to figure out how to realistically handle this complexity. Upstream release images are built and validated against EL userspace and host kernels; if users or distributions rely on different host kernels or mismatched stacks, folks who care about those combinations will likely need to step up with CI infrastructure or engineering capacity to ship and validate aligned virt stacks for their environments.

  19. morete commented on Sep 4, 2026

    @morete

    @lyarwood I made a little mistake, on the environment/cluster where I tested this upgrade we ONLY have Ubuntu 26.04 on kernel 7. All VMs I used to test were created under kubevirt 1.5.0 on U26 and kernel 7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions