Skip to content

[BUG] KubeVirt live migration deadlock: migration setup skipped when target pod not yet scheduled, never retried (primary CNI) #6823

Closed

Description

@jiri-jezek1

Kube-OVN Version

v1.15.10
In this case the whole infrastructure is cozystack version 1.4.2
As CNI its used kube-ovn with cilium in chained regime - https://kubeovn.github.io/docs/v1.12.x/en/advance/with-cilium/

Kubernetes Version

v1.34.6

Operation-system/Kernel Version

"Ubuntu 24.04.4 LTS"
6.8.0-100-generic

Description

When Kube-OVN is used as the primary CNI and --enable-live-migration-optimize=true
(the default), every KubeVirt live migration fails after ~15 minutes. The migration
target pod is stuck in ContainerCreating throughout. kube-ovn-cni logs the same
error every ~40 seconds:

E ovs_linux.go:232] ovs interface <hash>_h is not ready after 30s
E handler.go:334]   configure nic eth0 for pod <vm>/<namespace> failed: ovs interface <hash>_h is not ready after 30s

The OVN NB logical switch port for the VM keeps options: {requested-chassis=<source-node>}
throughout the entire attempt — it is never updated to the target node.

Steps To Reproduce

  1. Deploy Kube-OVN as primary CNI with --enable-live-migration-optimize=true
  2. Run a KubeVirt VM on any node
  3. Create a VirtualMachineInstanceMigration to migrate the VM to a different node
  4. Observe: target pod stays in ContainerCreating indefinitely; VMIM stays in Scheduling
  5. After ~15 minutes, VMIM transitions to Failed

The failure is 100% reproducible across all VMs, all source nodes, and both migration
directions.

Current Behavior

handleAddOrUpdateVMIMigration processes the MigrationScheduling phase with this guard

if sourceNode == "" || targetPod.Spec.NodeName == "" || sourceNode == targetPod.Spec.NodeName {
    klog.Warningf("VM pod %s/%s migration setup skipped, source node: %s, target node: %s ...")
    return nil   // ← dead end: no retry, no re-enqueue
}

When a VMIM first enters Scheduling, the target pod has been created but
not yet assigned to a node — targetPod.Spec.NodeName == "". The handler skips setup
and returns nil.

The pod then gets scheduled and pod update events fire. The pod handler at pod.go:1579
recognizes the pod is a migration target (pod is in the migration job) but only performs
normal IP/route sync — it never re-triggers SetLogicalSwitchPortMigrateOptions. No
subsequent VMIM phase-change event fires (VMIM stays in Scheduling because the pod
never reaches Running), so migration setup is never attempted again.

Controller log sequence (from live leader pod logs):

# VMIM created, target pod exists but has no nodeName yet:
I kubevirt.go:132] current vmiMigration status Scheduling, vmi MigrationState is stale
W kubevirt.go:187] VM pod tenant-cloud/<target-pod> migration setup skipped,
                   source node: hdp-004-meta, target node:  (migration job UID: ...)

# Pod gets scheduled; pod handler fires repeatedly — NO migration setup called:
I pod.go:1579] pod tenant-cloud/<target-pod> is in the migration job <migration-name>
I pod.go:561]  sync pod tenant-cloud/<target-pod> allocated
  [above two lines repeat every ~40s; no kubevirt.go lines appear between them]

OVN NB LSP during the stuck migration:

options: {requested-chassis=hdp-004-meta}   ← source node, never updated to target

Manually setting requested-chassis to the target node breaks the deadlock:

kubectl exec -n cozy-kubeovn $OVN_CENTRAL -c ovn-central -- \
  ovn-nbctl --db=unix:/var/run/ovn/ovnnb_db.sock \
  set Logical_Switch_Port <vm-name>.<namespace> \
  options:requested-chassis=<target-node-hostname>

After this command the CNI succeeds on the next retry (~30s) and the migration completes
(Succeeded). Confirmed on multiple migrations.

Historical pattern — 11 consecutive failures, same warning every time:

W kubevirt.go:187] ... migration setup skipped, source: hdp-004-meta, target: (empty)
W kubevirt.go:187] ... migration setup skipped, source: hdp-004-meta, target: (empty)
W kubevirt.go:187] ... migration setup skipped, source: hdp-003-meta, target: (empty)
  [× 11 total, 100% failure rate, different VMs and source nodes]

Confirmed present in upstream release-1.15 and release-1.16 source: the same
return nil guard with no re-enqueue exists in both branches.

Expected Behavior

When --enable-live-migration-optimize=true, a KubeVirt live migration should complete
successfully without manual intervention. Specifically:

After the migration target pod is scheduled to a node (i.e. targetPod.Spec.NodeName
becomes non-empty), SetLogicalSwitchPortMigrateOptions should be called to update
requested-chassis on the OVN NB LSP to the target chassis. The CNI will then succeed
and the migration will proceed to Succeeded.

Activity

  1. dosubot commented on Jun 4, 2026

    @dosubot

    Hi @jiri-jezek1! I'm Dosu and I'm helping the Mengxin's Org team.

    Your analysis is accurate. The guard condition at kubevirt.go:186-190 returns nil (success) when targetPod.Spec.NodeName == "", which drops the event from the work queue without re-enqueue — creating the permanent deadlock you've described [1].

    This is a known timing issue that was fixed on master via PR #6066 (merged 2025-12-19), which restructured the handler to properly synchronize with target pod scheduling using kubevirt.io/migrationJobUID label selectors [2]. However, PR #6066 was never backported to release-1.15, which is why it still affects v1.15.10.

    A closely related issue #6549 reported the same "ovs interface ... is not ready after 30s" failure pattern in v1.15.8 with a secondary CNI setup — that had a different root cause (hardcoded LSP name) fixed by PR #6241, which was backported [3]. Your case is distinct because you're using Kube-OVN as primary CNI, so the LSP name is correct — it's purely the timing/re-enqueue deadlock.

    Suggested fix: Backport PR #6066 to release-1.15. The change restructures the handleAddOrUpdateVMIMigration logic significantly (moving pieces from pod.go to kubevirt.go), so a clean cherry-pick may require some conflict resolution, but it directly addresses your root cause.

    Temporary workaround (as you've already discovered): manually set requested-chassis on the LSP to the target node after the target pod is scheduled.

    To reply, just mention @dosu.


    Docs are dead. Just use Dosu.

    Leave Feedback Ask Dosu about kube-ovn Share Dosu with your team

  2. oilbeater commented on Jun 4, 2026

    @oilbeater
    Member

    @jiri-jezek1 thanks for the detailed analysis can you make a pr to fix it?

  3. added a commit that references this issue on Jun 9, 2026
    9ace597
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions