Skip to content

virt-operator: stop GETting the disabled ContainerPath webhook on every reconcile - #19499

Merged
kubevirt-prow[bot] merged 1 commit into
kubevirt:mainfrom
fjvbn2003:operator-skip-containerpath-webhook-get
Oct 20, 2026
Merged

kubevirt-prow[bot] merged 1 commit into
kubevirt:mainfrom
fjvbn2003:operator-skip-containerpath-webhook-get

Conversation

@fjvbn2003

Copy link
Copy Markdown
Contributor

What this PR does

Before this PR:

With the ContainerPathVolumes feature gate disabled (the default), every virt-operator reconcile calls
createOrDeleteContainerPathVolumesWebhook, misses the virt-launcher-pod-mutator webhook in the informer
cache (it does not exist), and falls back to GET mutatingwebhookconfigurations/virt-launcher-pod-mutator,
which returns 404.

Measured on a homelab cluster (KubeVirt v1.9.0, Kubernetes v1.34.10, leader virt-operator up for 59 days),
from the operator's own kubevirt_rest_client_requests_total:

request count
GET mutatingwebhookconfigurations → 404 843,995
LIST nodes → 200 (once per reconcile, for comparison) 843,995
all operator REST calls 15,509,446
all non-2xx operator REST calls 843,999

So there is one 404 per reconcile. These 404s are 5.4% of all operator requests and all but 4 of its non-2xx
responses. They keep a constant ~5% floor under the error ratio that VirtOperatorRESTErrorsBurst is built
on (far below that alert's 80% threshold, so no alert fires, but it hides real errors on dashboards).

After this PR:

  • The KubeVirt controller waits for the MutatingWebhook informer to sync before it starts workers (it was
    the only webhook informer missing from hasSynced; ValidationWebhook is already there).
  • With the cache authoritative, a cache miss means there is nothing to delete, so no API call is made.
  • When the gate is disabled after being enabled, the webhook is still found in the cache and deleted as before.

References

Why we need it and why it was done in this way

The following tradeoffs were made:

  • If someone removes the app.kubernetes.io/managed-by label from the webhook by hand, the operator no longer
    sees it and will not delete it when the gate is off. That matches how the operator treats its other
    label-selected objects.

The following alternatives were considered:

  • Keep the GET only until the first successful sync. This needs extra state, and adding the informer to
    hasSynced gives the same guarantee with one line.

Links to places where the discussion took place: none

Special notes for your reviewer

  • The unit test change extends the existing "should not create virt-launcher-pod-mutator webhook when disabled"
    spec: it fails if any get action for this webhook name is recorded on the fake clientset.
  • Test results (Go 1.27.2, linux/amd64, GOFLAGS=-mod=vendor, on main @ 6e80bca): the focused spec -ginkgo.focus "ContainerPathVolumes feature gate" fails with only the test change applied (3 ran, 1 failed) and passes with the fix; go test ./pkg/virt-operator/... passes. One earlier run of ./pkg/virt-operator/resource/apply/ reported a TestInstallStrategy failure that did not reproduce on re-runs with or without this change.

Checklist

  • Design: A VEP is not required (bug fix, no API change)
  • PR: The PR description is expressive enough and will help future contributors
  • Code: Write code that humans can understand and Keep it simple
  • Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
  • Upgrade: no upgrade impact (the operator only reads less)
  • Testing: unit test extended
  • Documentation: not required
  • Community: Announcement to kubevirt-dev was considered
  • AI Contributions: The PR abides by the KubeVirt AI Contribution Policy.

AI disclosure: I used Claude (Anthropic) to help trace the cause from the metrics and to draft the change and
the test. I reviewed every line and measured the numbers above on my own cluster. The commit carries an
Assisted-by trailer.

Release note

virt-operator no longer queries the API server for the ContainerPathVolumes mutating webhook on every reconcile when the feature gate is disabled.

🤖 Generated with Claude Code

@kubevirt-prow kubevirt-prow Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Oct 17, 2026
@coderabbitai

coderabbitai Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: kubevirt/kubevirt/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 112ffff9-a221-488f-bde6-8bdff7a8f630

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kubevirt-prow

kubevirt-prow Bot commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

Hi @fjvbn2003. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@akalenyu akalenyu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!
/test all
/cc @mhenriks

// reconcile for every cluster that does not use ContainerPathVolumes.
_, exists, _ := r.stores.MutatingWebhookCache.Get(webhook)
if !exists {
// Also check directly in case cache is not populated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if this was ever a big deal since we would just queue it up when event handler notices it appears/disappears?
I think we can drop the comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, the informer requeues on add/delete anyway, so the comment overstated it. Dropped it.

@kubevirt-prow
kubevirt-prow Bot requested a review from mhenriks October 19, 2026 07:54
When the ContainerPathVolumes feature gate is disabled, the operator
looks up the virt-launcher-pod-mutator MutatingWebhookConfiguration in
its informer cache and, on a cache miss, sends a GET to the API server
in case the cache is not populated yet. On clusters that do not use
the feature gate the webhook never exists, so every reconcile sends a
GET that returns 404.

Make the controller wait for the MutatingWebhook informer to sync, as
it already does for the ValidatingWebhook informer, and trust the
cache. The webhook is always created with the managed-by label that
the informer selects on, so a webhook the operator created is always
visible there.

Assisted-by: Claude <[email protected]>
Signed-off-by: YoungjuKim <[email protected]>
@fjvbn2003
fjvbn2003 force-pushed the operator-skip-containerpath-webhook-get branch from 356c37d to 47a37b0 Compare October 19, 2026 13:13
@kubevirt-prow kubevirt-prow Bot added the lgtm Indicates that a PR is ready to be merged. label Oct 19, 2026
@mhenriks

Copy link
Copy Markdown
Member

/approve
Thanks @fjvbn2003!

@kubevirt-prow

kubevirt-prow Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mhenriks

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubevirt-prow kubevirt-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 19, 2026
@kubevirt-commenter-bot

Copy link
Copy Markdown

Required labels detected, running phase 2 presubmits:
/test pull-kubevirt-e2e-k8s-1.36-windows2016
/test pull-kubevirt-e2e-kind-1.37-vgpu
/test pull-kubevirt-e2e-k8s-1.37-ipv6-sig-network
/test pull-kubevirt-e2e-k8s-1.36-sig-network
/test pull-kubevirt-e2e-k8s-1.36-sig-storage
/test pull-kubevirt-e2e-k8s-1.36-sig-compute
/test pull-kubevirt-e2e-k8s-1.36-sig-operator
/test pull-kubevirt-e2e-k8s-1.37-sig-network

@kubevirt-prow
kubevirt-prow Bot merged commit 6443e16 into kubevirt:main Oct 20, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/operator dco-signoff: yes Indicates the PR's author has DCO signed all their commits. lgtm Indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/control-plane size/S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants