Skip to content

docs, node maintenance: describe the eviction webhook and evacuate-cancel - #1051

Open
fjvbn2003 wants to merge 1 commit into
kubevirt:mainfrom
fjvbn2003:node-maintenance-eviction-webhook
Open

fjvbn2003 wants to merge 1 commit into
kubevirt:mainfrom
fjvbn2003:node-maintenance-eviction-webhook

Conversation

@fjvbn2003

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

docs/cluster_admin/node_maintenance.md still says that "a PodDisruptionBudget is created for each VMI which has an evictionStrategy defined". KubeVirt stopped doing that in v1.6.0 (kubevirt/kubevirt#13764, "KubeVirt doesn't use PDBs anymore"). The page also does not say how to stop an evacuation after the node is made schedulable again.

Changes:

  1. Replace the PDB paragraph with what happens now. The virt-launcher-eviction-interceptor.kubevirt.io webhook denies the eviction and sets status.evacuationNodeName, and the evacuation controller creates kubevirt-evacuation-* migrations. Add the kubectl drain output you see while it retries, and what happens to a LiveMigrate VMI that is not live migratable.
  2. Add "Canceling an evacuation" under "Re-enabling a Node after Eviction". kubectl uncordon does not stop an evacuation that already started. The new section shows how to check status.evacuationNodeName, how to clear it with virtctl evacuate-cancel (v1.9.1 or later), that the kubevirt.io/drain taint blocks the cancel, and that a running migration has to be deleted separately.

Sources (kubevirt v1.9.0, the same code is on main):

  • pkg/virt-api/webhooks/validating-webhook/admitters/pod-eviction-admitter.go L127-183: deny messages, markVMI sets /status/evacuationNodeName.
  • pkg/virt-controller/watch/drain/evacuation/evacuation.go L326-334 and L416-424: VMIs marked for eviction keep being selected while they are still on that node, whether or not the node is schedulable. L344: GenerateName: "kubevirt-evacuation-".
  • pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go: only deletes old PDBs now.
  • pkg/virt-api/rest/evacuate_cancel.go (v1.9.1): JSON patch test + remove of /status/evacuationNodeName, rejected while the node has the drain taint. Nothing touches running migrations.
  • The virtctl fix that sends the node name is virtctl, evacuate: Provide required argument kubevirt#17008. It was backported to release-1.9 ([release-1.9] virtctl, evacuate: Provide required argument kubevirt#18864) and first released in v1.9.1.

What I measured on a test cluster (KubeVirt v1.9.0, Kubernetes 1.36, two workers):

  • Draining a node with evictionStrategy: LiveMigrate VMIs: kubectl get pdb -A gave No resources found before and during the drain. The drain output had the denials shown in the new example, and a VMI on a ReadWriteOnce PVC was denied with VMI <name> is configured with an eviction strategy but is not live-migratable.
  • A VMI running a memory-heavy workload kept failing to converge, and I uncordoned the node after the drain timed out. New kubevirt-evacuation-* migrations for that VMI were still created after the node was schedulable again (once 48 s after an uncordon, once about 5 minutes after a later one). When the load stopped, the last one succeeded and the VMI left the node.
  • virtctl evacuate-cancel vmi <name> with virtctl v1.9.0 failed with Internal error occurred: the server rejected our request due to an error in our request (virt-api: Failed to patching VMI, 500). The same PUT with body {"evacuationNodeName":"<node>"} returned 200 and cleared the field, which matches the virtctl fix above. I did not rerun this with a v1.9.1 binary.

I did not build the site locally. The page only uses Markdown that it already uses elsewhere (fenced blocks, !!! note).

Special notes for your reviewer:

This does not overlap with the open live migration docs PRs (they only touch docs/compute/live_migration.md). The new text links to its "Canceling a live migration" heading, which those PRs keep.


Prepared with help from Claude (AI); the measurements come from my own nested test cluster.

…ncel

KubeVirt stopped creating a PodDisruptionBudget per VMI in v1.6.0. Evictions
of virt-launcher pods are now denied by the
virt-launcher-eviction-interceptor.kubevirt.io webhook, which marks the VMI
with status.evacuationNodeName, and the evacuation controller creates
kubevirt-evacuation-* migrations.

Describe that, show what kubectl drain prints while it retries, and what
happens to a LiveMigrate VMI that is not live migratable.

Uncordoning a node does not stop an evacuation that already started: new
evacuation migrations keep being created while the VMI is still on the node
named in evacuationNodeName. Add a section on checking that field and on
clearing it with virtctl evacuate-cancel (v1.9.1 or later).

Signed-off-by: YoungjuKim <[email protected]>
Assisted-by: Claude <[email protected]>
@kubevirt-prow kubevirt-prow Bot added the dco-signoff: yes Indicates the PR's author has DCO signed all their commits. label Oct 30, 2026
@kubevirt-prow

kubevirt-prow Bot commented Oct 30, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign iholder101 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubevirt-prow
kubevirt-prow Bot requested review from fabiand and vladikr October 30, 2026 21:21
@kubevirt-prow kubevirt-prow Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 30, 2026
@kubevirt-prow

kubevirt-prow Bot commented Oct 30, 2026

Copy link
Copy Markdown

Hi @fjvbn2003. Thanks for your PR.

I'm waiting for a kubevirt member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubevirt-prow kubevirt-prow Bot added the size/M label Oct 30, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has DCO signed all their commits. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant