Repository navigation
Description
Kube-OVN Version
v1.15.10
In this case the whole infrastructure is cozystack version 1.4.2
As CNI its used kube-ovn with cilium in chained regime - https://kubeovn.github.io/docs/v1.12.x/en/advance/with-cilium/
Kubernetes Version
v1.34.6
Operation-system/Kernel Version
"Ubuntu 24.04.4 LTS"
6.8.0-100-generic
Description
When Kube-OVN is used as the primary CNI and --enable-live-migration-optimize=true
(the default), every KubeVirt live migration fails after ~15 minutes. The migration
target pod is stuck in ContainerCreating throughout. kube-ovn-cni logs the same
error every ~40 seconds:
E ovs_linux.go:232] ovs interface <hash>_h is not ready after 30s
E handler.go:334] configure nic eth0 for pod <vm>/<namespace> failed: ovs interface <hash>_h is not ready after 30s
The OVN NB logical switch port for the VM keeps options: {requested-chassis=<source-node>}
throughout the entire attempt — it is never updated to the target node.
Steps To Reproduce
- Deploy Kube-OVN as primary CNI with
--enable-live-migration-optimize=true - Run a KubeVirt VM on any node
- Create a
VirtualMachineInstanceMigrationto migrate the VM to a different node - Observe: target pod stays in
ContainerCreatingindefinitely; VMIM stays inScheduling - After ~15 minutes, VMIM transitions to
Failed
The failure is 100% reproducible across all VMs, all source nodes, and both migration
directions.
Current Behavior
handleAddOrUpdateVMIMigration processes the MigrationScheduling phase with this guard
if sourceNode == "" || targetPod.Spec.NodeName == "" || sourceNode == targetPod.Spec.NodeName {
klog.Warningf("VM pod %s/%s migration setup skipped, source node: %s, target node: %s ...")
return nil // ← dead end: no retry, no re-enqueue
}When a VMIM first enters Scheduling, the target pod has been created but
not yet assigned to a node — targetPod.Spec.NodeName == "". The handler skips setup
and returns nil.
The pod then gets scheduled and pod update events fire. The pod handler at pod.go:1579
recognizes the pod is a migration target (pod is in the migration job) but only performs
normal IP/route sync — it never re-triggers SetLogicalSwitchPortMigrateOptions. No
subsequent VMIM phase-change event fires (VMIM stays in Scheduling because the pod
never reaches Running), so migration setup is never attempted again.
Controller log sequence (from live leader pod logs):
# VMIM created, target pod exists but has no nodeName yet:
I kubevirt.go:132] current vmiMigration status Scheduling, vmi MigrationState is stale
W kubevirt.go:187] VM pod tenant-cloud/<target-pod> migration setup skipped,
source node: hdp-004-meta, target node: (migration job UID: ...)
# Pod gets scheduled; pod handler fires repeatedly — NO migration setup called:
I pod.go:1579] pod tenant-cloud/<target-pod> is in the migration job <migration-name>
I pod.go:561] sync pod tenant-cloud/<target-pod> allocated
[above two lines repeat every ~40s; no kubevirt.go lines appear between them]
OVN NB LSP during the stuck migration:
options: {requested-chassis=hdp-004-meta} ← source node, never updated to target
Manually setting requested-chassis to the target node breaks the deadlock:
kubectl exec -n cozy-kubeovn $OVN_CENTRAL -c ovn-central -- \
ovn-nbctl --db=unix:/var/run/ovn/ovnnb_db.sock \
set Logical_Switch_Port <vm-name>.<namespace> \
options:requested-chassis=<target-node-hostname>After this command the CNI succeeds on the next retry (~30s) and the migration completes
(Succeeded). Confirmed on multiple migrations.
Historical pattern — 11 consecutive failures, same warning every time:
W kubevirt.go:187] ... migration setup skipped, source: hdp-004-meta, target: (empty)
W kubevirt.go:187] ... migration setup skipped, source: hdp-004-meta, target: (empty)
W kubevirt.go:187] ... migration setup skipped, source: hdp-003-meta, target: (empty)
[× 11 total, 100% failure rate, different VMs and source nodes]
Confirmed present in upstream release-1.15 and release-1.16 source: the same
return nil guard with no re-enqueue exists in both branches.
Expected Behavior
When --enable-live-migration-optimize=true, a KubeVirt live migration should complete
successfully without manual intervention. Specifically:
After the migration target pod is scheduled to a node (i.e. targetPod.Spec.NodeName
becomes non-empty), SetLogicalSwitchPortMigrateOptions should be called to update
requested-chassis on the OVN NB LSP to the target chassis. The CNI will then succeed
and the migration will proceed to Succeeded.
Activity
dosubot commentedon Jun 4, 2026
Hi @jiri-jezek1! I'm Dosu and I'm helping the Mengxin's Org team.
Your analysis is accurate. The guard condition at
kubevirt.go:186-190returnsnil(success) whentargetPod.Spec.NodeName == "", which drops the event from the work queue without re-enqueue — creating the permanent deadlock you've described [1].This is a known timing issue that was fixed on
mastervia PR #6066 (merged 2025-12-19), which restructured the handler to properly synchronize with target pod scheduling usingkubevirt.io/migrationJobUIDlabel selectors [2]. However, PR #6066 was never backported torelease-1.15, which is why it still affects v1.15.10.A closely related issue #6549 reported the same
"ovs interface ... is not ready after 30s"failure pattern in v1.15.8 with a secondary CNI setup — that had a different root cause (hardcoded LSP name) fixed by PR #6241, which was backported [3]. Your case is distinct because you're using Kube-OVN as primary CNI, so the LSP name is correct — it's purely the timing/re-enqueue deadlock.Suggested fix: Backport PR #6066 to
release-1.15. The change restructures thehandleAddOrUpdateVMIMigrationlogic significantly (moving pieces frompod.gotokubevirt.go), so a clean cherry-pick may require some conflict resolution, but it directly addresses your root cause.Temporary workaround (as you've already discovered): manually set
requested-chassison the LSP to the target node after the target pod is scheduled.To reply, just mention @dosu.
Docs are dead. Just use Dosu.
oilbeater commentedon Jun 4, 2026
@jiri-jezek1 thanks for the detailed analysis can you make a pr to fix it?
fix(controller): requeue VMI migration when target pod is not yet sch…
fix(kubeovn): retry VM migration setup after scheduling (#4253)