kubevirt: the cluster-wide evictionStrategy: LiveMigrate applies to VMs whose disks cannot live-migrate #4360

What happens

packages/system/kubevirt/templates/kubevirt-cr.yaml:126 sets evictionStrategy: LiveMigrate cluster-wide. A VM whose boot disk is ReadWriteOnce cannot satisfy that strategy, and nothing rejects the combination: the VM starts and runs normally, and the contradiction only shows up as a condition on the VMI.

The VMI below comes from the e2e run on #3426, which is not merged, but the bug is not that PR's and does not depend on it — see the next section. I am quoting it because it is the clearest capture I have, out of the crust-gather snapshot in that run's cozyreport artifact:

status:
  conditions:
  - type: LiveMigratable
    status: "False"
    reason: DisksNotLiveMigratable
    message: "cannot migrate VMI: PVC site-router-a-boot is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode)"
  nodeName: srv1
  phase: Running
  launcherContainerImageVersion: quay.io/kubevirt/virt-launcher:v1.9.0

There is no PodDisruptionBudget in that namespace, while the same snapshot captured 13 of them across other namespaces, which is consistent with virt-controller's disruption-budget controller skipping a VMI it cannot migrate.

Why this is not one application's problem

The access mode is decided by the CDI StorageProfile of the chosen StorageClass rather than by the DataVolume, so whether a given VM ends up migratable depends on which StorageClass the user picked. packages/apps/kubernetes-nodes is the one place that reasoned about this: it overrides the strategy to evictionStrategy: External at templates/nodegroup.yaml:204, and its comment at line 95 explains that worker VMs live-migrate so their disks need RWX, which it arranges by falling back to a replicated DRBD-backed class.

packages/apps/vm-instance/templates/vm.yaml sets no evictionStrategy at all, so it inherits LiveMigrate from the platform CR. On any StorageClass whose profile gives RWO, a VMInstance on current main lands in exactly the state quoted above, with no unmerged code involved. That is the reason I am filing this separately rather than as a review comment on #3426: the default is set by the platform chart, the fallout reaches an application that shipped long ago, and fixing it inside one pending feature branch would leave vm-instance exactly as it is now. I expect the same for any future application that ships a KubeVirt VM and does not think to override the default.

What I have not established

I have not drained a node hosting such a VM, so I am not claiming which of the two failure modes happens: the eviction waiting on a migration that can never be created, or the VM being shut down and the workload dropping with no failover. Both are bad in different ways and the answer decides how urgent this is. Settling it needs a stand, not a log, and I would rather have the behaviour measured than guessed at in this issue.

What is established is that the contradictory state is reachable today, is reached silently, and that nothing in CI exercises a drain against it.

Possible directions

Making the platform default conditional is the obvious one, but the platform CR cannot see a workload's storage. Options that seem worth weighing: leave the cluster default alone and require every chart shipping a VM to declare a strategy explicitly, the way kubernetes-nodes does; or have a chart derive the strategy from whether its StorageClass profile offers RWX; or surface LiveMigratable=False as a warning on the application status so an operator learns about it before a maintenance window rather than during one.

I do not have a preference yet and would rather agree on the direction than open a PR against a guess.

Found via

Reviewing #3426, which ships a gateway VM that inherits the default. Run carrying the artifact the VMI above was read from: https://github.com/cozystack/cozystack/actions/runs/34513841738