Skip to content

vm.status.observedGeneration does not converge after live-updatable VM spec changes, breaking the generation sync signal #18700

Open

Description

@EdDev

Summary

vm.status.observedGeneration never reaches vm.status.desiredGeneration after a
live-updatable field (e.g., spec.template.spec.affinity) is changed on a running
VirtualMachine. Consumers cannot reliably detect that virt-controller has processed
and applied the change to the VMI.

Impact

Surfaced by a race condition in live-migration automation: a migration triggered
immediately after an affinity update could use the stale VMI affinity (before
virt-controller patched it), causing the migration target pod to land on the wrong node.
The intended fix — waiting until observedGeneration == desiredGeneration before
triggering migration — does not work because of this bug.

Root Cause

vm.status.observedGeneration is derived from the kubevirt.io/vm-generation annotation
on the VMI. The annotation is bumped by conditionallyBumpGenerationAnnotationOnVmi
(vm.go:1438), which only bumps when the controller revision (a snapshot of
vm.spec.template taken at VMI creation) is identical to the current vm.spec.template:

// vm.go:1465
if equality.Semantic.DeepEqual(revisionSpec.Spec.Template, vm.Spec.Template) {
    patchedVMI, err := c.patchVmGenerationAnnotationOnVmi(vm.Generation, vmi)
    ...
}

For live-updatable fields (affinity, tolerations, nodeSelector, CPU sockets, memory),
the controller revision retains the original values. Even after handleAffinityChangeRequest
(vm.go:775) successfully patches the VMI spec, the controller revision is
never updated, so the comparison never becomes equal — the annotation is never bumped and
observedGeneration is never updated.

The template-equality check was introduced in PR #9232 (March 2023), before
live-updatable fields existed, solely to handle the restart-required case. It was not
updated when live-updatable fields were added.

Proposed Solutions

Option A — Change semantics: bump at end of every reconciliation cycle (recommended)

Remove the template-equality check from conditionallyBumpGenerationAnnotationOnVmi
(vm.go:1438). Bump the annotation unconditionally at the end of each successful
reconciliation cycle, keeping only the early-exit guard (skip if annotation already equals
vm.Generation).

observedGeneration == desiredGeneration would mean: "virt-controller completed a full
reconciliation pass at this generation."
The existing RestartRequired condition
remains the authoritative signal for pending non-live-updatable changes.

Pros Simpler — template-comparison removed entirely
Handles all live-updatable fields without enumerating them
Clean separation of concerns: generation = "controller ran", RestartRequired = "VMI out of sync"
Future live-updatable fields need no changes here
Cons Semantic change to a public API field — external consumers relying on the current meaning must be updated; must be clearly communicated in API docs and release notes
Unit tests in vm_test.go require updates

Option B — Extend conditionallyBumpGenerationAnnotationOnVmi to account for live-updatable fields

When the template-equality check fails, add a second check: copy the live-updatable field
values from the VMI's current spec into a copy of the controller revision template,
then compare against the current VM template. If equal, all changes were in live-updatable
fields that have already been applied to the VMI — safe to bump.

Pros Preserves existing semantics — no risk to external consumers
Minimal blast radius — only conditionallyBumpGenerationAnnotationOnVmi changes
Cons Must enumerate all live-updatable fields; syncRestartRequired already has this list with a maintenance warning — this adds a second place to keep in sync
Scope is wider than it appears: must also cover fields neutralized outside the IsVMRolloutStrategyLiveUpdate gate (network interfaces, volumes, disks, firmware UUID)
Bump takes effect on the subsequent reconciliation cycle, not the same one as the live-update patch
The underlying conflation of "generation tracking" and "template consistency" remains in the design

Relevant Code

All links are to commit b8b04c190d (main as of this writing).

Location Relevance
vm.go:1438 — conditionallyBumpGenerationAnnotationOnVmi where the fix goes
vm.go:1465 — template-equality check the check that causes the bug
vm.go:775 — handleAffinityChangeRequest patches the VMI, runs after the bump check
vm.go:2428 — syncGenerationInfo reads annotation, writes vm.status.observedGeneration
vm.go:2988 — syncRestartRequired correctly neutralizes live-updatable fields — the pattern Solution B would replicate
vm_test.go — generation tracking tests unit tests that cover this area; need updating for Option A

Additional Observation (out of scope for this fix)

The kubevirt.io/vm-generation VMI annotation is an unnecessary intermediary: both the
write and the read are performed exclusively by the VM controller; no other KubeVirt
component reads it. vm.status.observedGeneration could be maintained directly in
vm.status, removing an extra API patch call per bump and the backfill path
(patchVmGenerationFromControllerRevision). However, the annotation is visible on VMI
objects and may be consumed by external tooling — this should be a separate effort
with its own deprecation and API review, not bundled with this fix.

History

  • PR #9232 (March 2023): introduced these fields; no VEP or design doc; template-equality check present from day one but predates live-updatable fields
  • PR #15298 (March 2026): attempted to replace the annotation approach by creating ControllerRevisions on every live spec change — would have resolved this issue
  • PR #17466 (April 2026): reverted watch/vm: Handle VM generation updates #15298 because other subsystems depend on ControllerRevisions being boot-time snapshots; generation tracking simplification was noted for separate follow-up

Activity

  1. EdDev commented on Aug 3, 2026

    @EdDev
    MemberAuthor

    /sig compute

  2. lyarwood commented on Aug 5, 2026

    @lyarwood
    Member

    /cc

  3. tillo commented on Oct 27, 2026

    @tillo

    Another trigger with the same root cause, and this one needs no user change at all:
    virt-controller's own PCI-topology writeback.

    syncPCITopologyAnnotationsToVM (called from sync()) copies
    kubevirt.io/pci-topology-version, and kubevirt.io/pci-interface-slot-count when the VMI carries it,
    from the VMI into vm.spec.template.metadata.annotations when the template has none. That
    raises metadata.generation by 1. The ControllerRevision was created in startVMI before
    the annotation existed on the template. So the equality.Semantic.DeepEqual in
    conditionallyBumpGenerationAnnotationOnVmi fails, kubevirt.io/vm-generation on the VMI is
    never bumped, and observedGeneration stays one behind desiredGeneration until the VM
    restarts. The only difference between the revision template and the VM template is a value
    the controller copied from that same VMI.

    Observed on v1.7.4 (Harvester v1.8.2): 6 of 10 VMs, all at once, right after the upgrade.
    virt-handler's detectPCITopologyAndAnnotate annotated the running VMIs, and managedFields
    show virt-controller as the writer of
    f:spec/f:template/f:metadata/f:annotations/f:kubevirt.io/pci-topology-version.

    On release-1.9 and main the virt-handler detection is gone, but the path is still reachable:

    • startVMI → createVMRevision(vm) (vm.go L1309), from a template without the annotation;
    • the VMI mutating webhook sets pci-topology-version: v3 on the new VMI. This is the
      "Pre-v2 VM (no annotation)" and "v2 VM (without annotation)" cases in docs/pci-topology.md;
    • sync() → syncPCITopologyAnnotationsToVM (L3359) writes it into the template, and generation goes up by 1;
    • conditionallyBumpGenerationAnnotationOnVmi (L3275): the revision has no annotation, so no bump.

    So every VM created before the VM webhook began setting the annotation reports a phantom
    pending change after its first boot on the new version.

    Repro on main:

    1. Create a hotplug-capable VM (q35, defaults). Remove
      spec.template.metadata.annotations["kubevirt.io/pci-topology-version"] from the template,
      which is how a VM created before the VM webhook set it looks.
    2. Start it. Watch {.metadata.generation} {.status.observedGeneration} {.status.desiredGeneration}
      and the VMI's kubevirt.io/vm-generation.
    3. Generation goes up by 1 and observedGeneration does not follow. Diff the ControllerRevision
      named in vmi.status.virtualMachineRevisionName (.data.spec.template) against
      vm.spec.template: the only difference is the annotation above.

    This bears on the proposals in this issue. Option B copies live-updatable spec fields from
    the VMI into the revision before comparing, so as written it would miss this case. The
    differing field is template metadata, written by the controller. Option A (bump at the end
    of a successful reconcile, with RestartRequired as the pending-change signal) covers it. If B
    is preferred, the two PCI-topology annotations need to be in its copy-from-VMI set too, or
    syncPCITopologyAnnotationsToVM could bump the VMI's kubevirt.io/vm-generation in the same
    sync once the VM update succeeds. The value cannot be a change the VMI lacks, because it came
    from the VMI.

    Workaround we used, safe only after the template diff above shows nothing else:

    kubectl patch vmi <name> --type=json -p '[
      {"op":"test","path":"/metadata/annotations/kubevirt.io~1vm-generation","value":"<old>"},
      {"op":"replace","path":"/metadata/annotations/kubevirt.io~1vm-generation","value":"<vm .metadata.generation>"}]'
    

    Environment: KubeVirt v1.7.4 (Harvester v1.8.2), Kubernetes v1.35.7 (rke2), bare metal, Linux guests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions