Repository navigation
Conversation
…ncel KubeVirt stopped creating a PodDisruptionBudget per VMI in v1.6.0. Evictions of virt-launcher pods are now denied by the virt-launcher-eviction-interceptor.kubevirt.io webhook, which marks the VMI with status.evacuationNodeName, and the evacuation controller creates kubevirt-evacuation-* migrations. Describe that, show what kubectl drain prints while it retries, and what happens to a LiveMigrate VMI that is not live migratable. Uncordoning a node does not stop an evacuation that already started: new evacuation migrations keep being created while the VMI is still on the node named in evacuationNodeName. Add a section on checking that field and on clearing it with virtctl evacuate-cancel (v1.9.1 or later). Signed-off-by: YoungjuKim <[email protected]> Assisted-by: Claude <[email protected]>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @fjvbn2003. Thanks for your PR. I'm waiting for a kubevirt member to verify that this patch is reasonable to test. If it is, they should reply with Tip We noticed you've done this a few times! Consider joining the org to skip this step and gain Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
What this PR does / why we need it:
docs/cluster_admin/node_maintenance.mdstill says that "a PodDisruptionBudget is created for each VMI which has an evictionStrategy defined". KubeVirt stopped doing that in v1.6.0 (kubevirt/kubevirt#13764, "KubeVirt doesn't use PDBs anymore"). The page also does not say how to stop an evacuation after the node is made schedulable again.Changes:
virt-launcher-eviction-interceptor.kubevirt.iowebhook denies the eviction and setsstatus.evacuationNodeName, and the evacuation controller createskubevirt-evacuation-*migrations. Add thekubectl drainoutput you see while it retries, and what happens to aLiveMigrateVMI that is not live migratable.kubectl uncordondoes not stop an evacuation that already started. The new section shows how to checkstatus.evacuationNodeName, how to clear it withvirtctl evacuate-cancel(v1.9.1 or later), that thekubevirt.io/draintaint blocks the cancel, and that a running migration has to be deleted separately.Sources (kubevirt v1.9.0, the same code is on main):
pkg/virt-api/webhooks/validating-webhook/admitters/pod-eviction-admitter.goL127-183: deny messages,markVMIsets/status/evacuationNodeName.pkg/virt-controller/watch/drain/evacuation/evacuation.goL326-334 and L416-424: VMIs marked for eviction keep being selected while they are still on that node, whether or not the node is schedulable. L344:GenerateName: "kubevirt-evacuation-".pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go: only deletes old PDBs now.pkg/virt-api/rest/evacuate_cancel.go(v1.9.1): JSON patchtest+removeof/status/evacuationNodeName, rejected while the node has the drain taint. Nothing touches running migrations.What I measured on a test cluster (KubeVirt v1.9.0, Kubernetes 1.36, two workers):
evictionStrategy: LiveMigrateVMIs:kubectl get pdb -AgaveNo resources foundbefore and during the drain. The drain output had the denials shown in the new example, and a VMI on aReadWriteOncePVC was denied withVMI <name> is configured with an eviction strategy but is not live-migratable.kubevirt-evacuation-*migrations for that VMI were still created after the node was schedulable again (once 48 s after an uncordon, once about 5 minutes after a later one). When the load stopped, the last one succeeded and the VMI left the node.virtctl evacuate-cancel vmi <name>with virtctl v1.9.0 failed withInternal error occurred: the server rejected our request due to an error in our request(virt-api:Failed to patching VMI, 500). The same PUT with body{"evacuationNodeName":"<node>"}returned 200 and cleared the field, which matches the virtctl fix above. I did not rerun this with a v1.9.1 binary.I did not build the site locally. The page only uses Markdown that it already uses elsewhere (fenced blocks,
!!! note).Special notes for your reviewer:
This does not overlap with the open live migration docs PRs (they only touch
docs/compute/live_migration.md). The new text links to its "Canceling a live migration" heading, which those PRs keep.Prepared with help from Claude (AI); the measurements come from my own nested test cluster.