Repository navigation
kubernetes: tenant workers on kubevirt should inherit topology zone/region labels from their management-cluster node #5305
Description
Problem / motivation
Tenant Kubernetes workers on the kubevirt substrate never get topology.kubernetes.io/zone or topology.kubernetes.io/region, even when the management-cluster node hosting the worker VM carries them. Inside the tenant cluster, zone-aware scheduling (topologySpreadConstraints on topology.kubernetes.io/zone, zonal anti-affinity, zone-aware storage or load balancing) has nothing to key on, so a tenant cannot spread its replicas across the failure domains its workers actually run in.
The code to do this already ships. The kubevirt cloud-controller-manager we build (kubevirt/cloud-provider-kubevirt at a0acf33) defaults to instancesV2.zoneAndRegionEnabled: true: on node initialisation it reads the zone and region labels of the infra node the worker VMI runs on and sets them on the tenant Node, and templates/kccm/kccm_cluster_role.yaml already grants it get nodes on the management cluster. It never runs, though. The cloud node controller only initialises Nodes carrying the node.cloudprovider.kubernetes.io/uninitialized taint, which the kubelet sets only with --cloud-provider=external, and _talosconfigtemplate.tpl sets that flag on the proxmox substrate only. On proxmox the zone and region labels reach tenant Nodes through the Proxmox CCM; on kubevirt they never do.
Proposed solution
Tenant workers on the kubevirt substrate inherit topology.kubernetes.io/zone and topology.kubernetes.io/region from the management-cluster node hosting their VM, and keep them accurate when the VM moves.
Two parts:
- Initial labels: run the kubevirt worker kubelet with
cloud-provider: externalso KCCM initialises each Node and applies zone and region throughInstancesV2. This needs KCCM to reliably clear theuninitializedtaint on every worker (including scale-from-zero and the window before KCCM is up), otherwise workers stay unschedulable; that is the reason the template comment gives for keeping the flag off today. - Keeping them current: the cloud node controller sets these labels once, at initialisation, and only updates addresses afterwards. Worker VMs live-migrate (
kubevirt.io/allow-pod-bridge-network-live-migration), so after a migration across zones the tenant Node keeps a stale zone. Something has to resync the labels whenvmi.status.nodeNamechanges, for instance in KCCM (we already carry a VMI informer patch,vmi-informer.diff) or in a small controller beside it.
Whether this is always on or opt-in per cluster is open; a management cluster whose nodes carry no topology labels would see no change either way.
Scope check
- This affects a single component / package (if it spans many, consider a design proposal)
- I can describe the expected behavior concretely (if not, consider a Discussion first)
It touches packages/apps/kubernetes-nodes (kubelet flag) and the KCCM in packages/apps/kubernetes (resync on migration).
Alternatives considered
- Static zone labels per pool, once worker pools can carry arbitrary labels (feat(kubernetes): add labels and taints to worker pools #4927): works only if each pool is pinned to one zone on the management cluster, and the label is a declaration rather than a fact, so it drifts as soon as a VM lands elsewhere.
- Setting the labels from Talos machine config at boot: the zone is only known after the VM is scheduled, and it would still go stale on live migration.
Additional context
- [Epic] Topology and affinity settings for applications in Cozystack #1949 covers placement of tenant workloads on the management cluster; this issue is the other direction, exposing the management-cluster placement inside the tenant cluster.
- feat(kubernetes): add labels and taints to worker pools #4927 adds
labelsandtaintstoKubernetesNodes; its validation already keeps the zone labels for the Proxmox CCM on that substrate, and the same reservation would apply here.
Activity
mattia-eleuteri commentedon Nov 13, 2026
Some findings on why the kubevirt path runs without
cloud-provider: external, and what turning it on would cost.Why the flag is off. It was never set on kubevirt: it arrived with the Proxmox CCM (d902421), where the CSI cannot start without the zone and region labels. The template comment says the kubevirt path "has no cloud provider" and that the flag would taint every worker with nothing to remove the taint. That premise does not hold: every kubevirt tenant cluster already runs KCCM, so the real question is whether KCCM can be relied on to clear the taint.
Checked on a test management cluster (read-only):
node_controllerandnode_lifecycle_controller. The lifecycle controller already acts on every tenant Node today (deletes a Node whose VMI is gone, taints a shut-down one); the node controller idles because no Node carries theuninitializedtaint.InstanceMetadataresolves the VMI throughInstanceByVMINamewithout theinstance-idlabel.spec.providerIDtokubevirt://<vmi>, the same value KCCM would compute, so there is no providerID conflict.defaultinterface with the pod IP, which is what KCCM would write as the Node'sInternalIP.topology.kubernetes.io/zonebut noregion, so tenant Nodes would get a zone and an empty region.node.kubernetes.io/instance-typewould stay empty too, since the worker VMIs carry nokubevirt.io/instancetype-nameannotation.What the change costs:
TalosConfigTemplate, so the upgrade that introduces it replaces every worker in every pool of every tenant cluster, as the automatic GPUkernelModulesdid.NoScheduleuntil KCCM initialises it, which covers autoscaler scale-up (including from zero), MachineHealthCheck replacements, and thenodeStartupTimeoutbudget. The KCCM on that cluster logs repeated leader-election failures (timeouts and DNS errors reaching the tenant apiserver), so this is not hypothetical.node.cloudprovider.kubernetes.io/uninitializedas a startup taint rather than counting the node as unusable during scale-up.No chicken-and-egg problem: KCCM runs in the management cluster, not on the workers, and Cilium and the system DaemonSets tolerate the taint.
So the chart change is one line, but it rolls every kubevirt worker and makes worker startup depend on KCCM availability. The roll cannot be avoided; the dependency should be validated (autoscaler behaviour, KCCM resilience) before the flag becomes the default, which argues for an opt-in first.