Repository navigation
Description
Summary
vm.status.observedGeneration never reaches vm.status.desiredGeneration after a
live-updatable field (e.g., spec.template.spec.affinity) is changed on a running
VirtualMachine. Consumers cannot reliably detect that virt-controller has processed
and applied the change to the VMI.
Impact
Surfaced by a race condition in live-migration automation: a migration triggered
immediately after an affinity update could use the stale VMI affinity (before
virt-controller patched it), causing the migration target pod to land on the wrong node.
The intended fix — waiting until observedGeneration == desiredGeneration before
triggering migration — does not work because of this bug.
Root Cause
vm.status.observedGeneration is derived from the kubevirt.io/vm-generation annotation
on the VMI. The annotation is bumped by conditionallyBumpGenerationAnnotationOnVmi
(vm.go:1438), which only bumps when the controller revision (a snapshot of
vm.spec.template taken at VMI creation) is identical to the current vm.spec.template:
// vm.go:1465
if equality.Semantic.DeepEqual(revisionSpec.Spec.Template, vm.Spec.Template) {
patchedVMI, err := c.patchVmGenerationAnnotationOnVmi(vm.Generation, vmi)
...
}For live-updatable fields (affinity, tolerations, nodeSelector, CPU sockets, memory),
the controller revision retains the original values. Even after handleAffinityChangeRequest
(vm.go:775) successfully patches the VMI spec, the controller revision is
never updated, so the comparison never becomes equal — the annotation is never bumped and
observedGeneration is never updated.
The template-equality check was introduced in PR #9232 (March 2023), before
live-updatable fields existed, solely to handle the restart-required case. It was not
updated when live-updatable fields were added.
Proposed Solutions
Option A — Change semantics: bump at end of every reconciliation cycle (recommended)
Remove the template-equality check from conditionallyBumpGenerationAnnotationOnVmi
(vm.go:1438). Bump the annotation unconditionally at the end of each successful
reconciliation cycle, keeping only the early-exit guard (skip if annotation already equals
vm.Generation).
observedGeneration == desiredGeneration would mean: "virt-controller completed a full
reconciliation pass at this generation." The existing RestartRequired condition
remains the authoritative signal for pending non-live-updatable changes.
| Pros | Simpler — template-comparison removed entirely |
| Handles all live-updatable fields without enumerating them | |
Clean separation of concerns: generation = "controller ran", RestartRequired = "VMI out of sync" |
|
| Future live-updatable fields need no changes here | |
| Cons | Semantic change to a public API field — external consumers relying on the current meaning must be updated; must be clearly communicated in API docs and release notes |
Unit tests in vm_test.go require updates |
Option B — Extend conditionallyBumpGenerationAnnotationOnVmi to account for live-updatable fields
When the template-equality check fails, add a second check: copy the live-updatable field
values from the VMI's current spec into a copy of the controller revision template,
then compare against the current VM template. If equal, all changes were in live-updatable
fields that have already been applied to the VMI — safe to bump.
| Pros | Preserves existing semantics — no risk to external consumers |
Minimal blast radius — only conditionallyBumpGenerationAnnotationOnVmi changes |
|
| Cons | Must enumerate all live-updatable fields; syncRestartRequired already has this list with a maintenance warning — this adds a second place to keep in sync |
Scope is wider than it appears: must also cover fields neutralized outside the IsVMRolloutStrategyLiveUpdate gate (network interfaces, volumes, disks, firmware UUID) |
|
| Bump takes effect on the subsequent reconciliation cycle, not the same one as the live-update patch | |
| The underlying conflation of "generation tracking" and "template consistency" remains in the design |
Relevant Code
All links are to commit b8b04c190d (main as of this writing).
| Location | Relevance |
|---|---|
vm.go:1438 — conditionallyBumpGenerationAnnotationOnVmi |
where the fix goes |
vm.go:1465 — template-equality check |
the check that causes the bug |
vm.go:775 — handleAffinityChangeRequest |
patches the VMI, runs after the bump check |
vm.go:2428 — syncGenerationInfo |
reads annotation, writes vm.status.observedGeneration |
vm.go:2988 — syncRestartRequired |
correctly neutralizes live-updatable fields — the pattern Solution B would replicate |
vm_test.go — generation tracking tests |
unit tests that cover this area; need updating for Option A |
Additional Observation (out of scope for this fix)
The kubevirt.io/vm-generation VMI annotation is an unnecessary intermediary: both the
write and the read are performed exclusively by the VM controller; no other KubeVirt
component reads it. vm.status.observedGeneration could be maintained directly in
vm.status, removing an extra API patch call per bump and the backfill path
(patchVmGenerationFromControllerRevision). However, the annotation is visible on VMI
objects and may be consumed by external tooling — this should be a separate effort
with its own deprecation and API review, not bundled with this fix.
History
- PR #9232 (March 2023): introduced these fields; no VEP or design doc; template-equality check present from day one but predates live-updatable fields
- PR #15298 (March 2026): attempted to replace the annotation approach by creating ControllerRevisions on every live spec change — would have resolved this issue
- PR #17466 (April 2026): reverted watch/vm: Handle VM generation updates #15298 because other subsystems depend on ControllerRevisions being boot-time snapshots; generation tracking simplification was noted for separate follow-up
Activity
EdDev commentedon Aug 3, 2026
/sig compute
lyarwood commentedon Aug 5, 2026
/cc
tests: add envtest test for VM/VMI generation tracking
tests: add envtest test for VM/VMI generation tracking
fix(vm-instance-rd): disable helm readiness wait on VMInstance releases
fix(vm-instance-rd): disable helm readiness wait on VMInstance releases
fix(vm-instance): disable helm readiness wait on the VMInstance relea…
tests: add envtest test for VM/VMI generation tracking
tests: add integration tests for VM/VMI generation tracking
tests: add integration tests for VM/VMI generation tracking
tests: add integration tests for VM/VMI generation tracking
tests: add integration tests for VM/VMI generation tracking
tests: add integration tests for VM/VMI generation tracking
tillo commentedon Oct 27, 2026
Another trigger with the same root cause, and this one needs no user change at all:
virt-controller's own PCI-topology writeback.
syncPCITopologyAnnotationsToVM(called fromsync()) copieskubevirt.io/pci-topology-version, andkubevirt.io/pci-interface-slot-countwhen the VMI carries it,from the VMI into
vm.spec.template.metadata.annotationswhen the template has none. Thatraises
metadata.generationby 1. The ControllerRevision was created instartVMIbeforethe annotation existed on the template. So the
equality.Semantic.DeepEqualinconditionallyBumpGenerationAnnotationOnVmifails,kubevirt.io/vm-generationon the VMI isnever bumped, and
observedGenerationstays one behinddesiredGenerationuntil the VMrestarts. The only difference between the revision template and the VM template is a value
the controller copied from that same VMI.
Observed on v1.7.4 (Harvester v1.8.2): 6 of 10 VMs, all at once, right after the upgrade.
virt-handler's
detectPCITopologyAndAnnotateannotated the running VMIs, and managedFieldsshow
virt-controlleras the writer off:spec/f:template/f:metadata/f:annotations/f:kubevirt.io/pci-topology-version.On release-1.9 and main the virt-handler detection is gone, but the path is still reachable:
startVMI→createVMRevision(vm)(vm.go L1309), from a template without the annotation;pci-topology-version: v3on the new VMI. This is the"Pre-v2 VM (no annotation)" and "v2 VM (without annotation)" cases in docs/pci-topology.md;
sync()→syncPCITopologyAnnotationsToVM(L3359) writes it into the template, and generation goes up by 1;conditionallyBumpGenerationAnnotationOnVmi(L3275): the revision has no annotation, so no bump.So every VM created before the VM webhook began setting the annotation reports a phantom
pending change after its first boot on the new version.
Repro on main:
spec.template.metadata.annotations["kubevirt.io/pci-topology-version"]from the template,which is how a VM created before the VM webhook set it looks.
{.metadata.generation} {.status.observedGeneration} {.status.desiredGeneration}and the VMI's
kubevirt.io/vm-generation.named in
vmi.status.virtualMachineRevisionName(.data.spec.template) againstvm.spec.template: the only difference is the annotation above.This bears on the proposals in this issue. Option B copies live-updatable spec fields from
the VMI into the revision before comparing, so as written it would miss this case. The
differing field is template metadata, written by the controller. Option A (bump at the end
of a successful reconcile, with
RestartRequiredas the pending-change signal) covers it. If Bis preferred, the two PCI-topology annotations need to be in its copy-from-VMI set too, or
syncPCITopologyAnnotationsToVMcould bump the VMI'skubevirt.io/vm-generationin the samesync once the VM update succeeds. The value cannot be a change the VMI lacks, because it came
from the VMI.
Workaround we used, safe only after the template diff above shows nothing else:
Environment: KubeVirt v1.7.4 (Harvester v1.8.2), Kubernetes v1.35.7 (rke2), bare metal, Linux guests.
fix(vm-instance): disable helm readiness wait on the VMInstance relea…
tests: add integration tests for VM/VMI generation tracking