Repository navigation
feat(kubernetes): add labels and taints to worker pools - #4927
Draft
Aleksei Sviridkin (lexfrei) wants to merge 1 commit into
Draft
Aleksei Sviridkin (lexfrei) wants to merge 1 commit into
Aleksei Sviridkin (lexfrei) wants to merge 1 commit into
Conversation
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
area/kubernetes
Merged
2 tasks done
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/kubernetes-kairos-pools
branch
from
October 18, 2026 20:28
12296ff to
0637aa4
Compare
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/kubernetes-nodes-labels-taints
branch
2 times, most recently
from
October 18, 2026 21:19
8e4cc75 to
af8fd13
Compare
1 of 12 tasks
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/kubernetes-kairos-pools
branch
from
October 27, 2026 17:51
0637aa4 to
dff2ea6
Compare
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/kubernetes-nodes-labels-taints
branch
2 times, most recently
from
October 27, 2026 21:57
700b686 to
edb15a6
Compare
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/kubernetes-kairos-pools
branch
2 times, most recently
from
October 29, 2026 13:55
5b53c4f to
7d309f6
Compare
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/kubernetes-nodes-labels-taints
branch
from
October 29, 2026 13:55
edb15a6 to
66c6a82
Compare
A pool can now carry its own node labels and taints, for Talos and for Kairos workers alike, for example to keep ordinary workloads off GPU workers with an nvidia.com/gpu NoSchedule taint, which the GPU operator's daemonsets tolerate by default. A taint the GPU addons do not tolerate keeps their device plugin off the pool and leaves it without GPUs, so the taints description says what they tolerate. Labels go where something will apply them. Cluster API syncs labels from a Machine to its Node only under node-role.kubernetes.io, node-restriction.kubernetes.io and node.cluster.x-k8s.io. Node roles already have their own field, roles, so a node-role.kubernetes.io label is refused rather than giving a pool two ways to set one; labels in the other two domains go on the MachineDeployment template and change in place. Every other label goes through the bootstrap config: on Talos, machine.nodeLabels, which machined writes to the Node with the kubelet's credentials; on Kairos, kubeadm's node-labels, where the kubelet admits no other kubernetes.io or k8s.io label than the kubelet.kubernetes.io and node.kubernetes.io domains and a few well-known keys. The chart applies that rule on both OSes. Of the well-known keys it refuses hostname, os and arch, which describe the node itself, and the deprecated beta zone, region and instance-type keys, which the cloud-node controller copies over the current ones. The current zone, region and instance-type keys are accepted on KubeVirt and refused on Proxmox, whose cloud controller manager sets them. Taints are registered by the kubelet when the node joins: Talos through the kubelet's registerWithTaints, Kairos through kubeadm's nodeRegistration.taints. Cluster API v1.13 can also manage Machine taints, but behind an alpha feature gate that is off. Both are part of the bootstrap config, so a change rolls the pool and every new machine carries them. The MachineDeployment also announces the pool's labels and taints to the cluster-autoscaler through its capacity labels and taints annotations. A pool scaled to zero has no Node to read them from, and without them a pod selecting the pool's label never scales it up, and a tainted pool is scaled up for pods that cannot tolerate it. Keys and values are held to the label and taint character sets, which also keeps them safe in the Talos reconcile script, and tenant label keys are quoted there so a key such as null stays a string. The values schema also declares the taint key format and the three effects, so Helm reports a malformed taint against its field before any template renders. A pool without labels or taints renders as before. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
Aleksei Sviridkin (lexfrei)
force-pushed
the
feat/kubernetes-nodes-labels-taints
branch
from
October 29, 2026 15:56
66c6a82 to
f317a98
Compare
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Worker pools can now carry their own node labels and taints, for Talos and Kairos workers alike. Without a taint, GPU workers take ordinary workloads too. A
nvidia.com/gpu:NoScheduletaint keeps them off and is tolerated by the GPU Operator's own DaemonSets by default; any other taint, or HAMi, needs the add-on tolerations set throughaddons.gpuOperator.valuesOverrideoraddons.hami.valuesOverride, or the pool is left without GPUs. The override replaces the whole tolerations list, so it has to keep thenvidia.com/gpuentry too.KubernetesNodesgets two fields,labelsandtaints.Stacked on #4911 (Kairos worker pools), the base branch is
feat/kubernetes-kairos-pools.Labels in the domains Cluster API syncs to Nodes (
node-restriction.kubernetes.io,node.cluster.x-k8s.io) go on the MachineDeployment template. Cluster API syncs those to the Nodes, so they change in place. Every other label and every taint goes through the pool's bootstrap config, so changing them rolls the pool. Taints use kubelet registration because Cluster API's own taint propagation is still behind an alpha feature gate.Some values are refused before anything is applied. A malformed taint effect or key fails when Helm loads the values, against the field; other keys and values Kubernetes would reject fail at render time instead of in the reconcile Job. A
node-role.kubernetes.io/<role>label is refused with a pointer toroles, which already sets node roles. Hostname, os and arch describe the node itself, so a pool can't override them. The deprecated beta zone and instance-type labels are refused too, and on Proxmox the zone and instance-type labels belong to its cloud controller manager. Taints in Kubernetes or Cluster API namespaces are refused. A pool without labels or taints renders byte for byte as before.I tested it on a test cluster with one Talos and one Kairos pool. On both, a label in a Cluster API domain reached the Node without a rollout. A kubelet label and a taint rolled the pool, and the new Node carried both. A pod without the toleration stayed Pending, a pod with it ran on that Node. The Kairos node was also recreated once before the test, because the default worker disk changed in the test overlay. That rollout is unrelated to this change.
Reviewed by Opus. The last two passes on the same commit found nothing.
A decision record for the kubernetes-nodes-split proposal explains why this replaces the proposal's Talos-only
userMachineConfiglayer. It goes to cozystack/community separately.Screenshots
No UI changes.
Downstream repositories
The trigger map reaches two repositories. Follow-ups are not opened yet, they are listed here for a maintainer to decide.
kubernetes-nodesgetslabelsandtaints. The generated reference page picks them up from the README.kubernetes-nodesvalues.schema.json, if the provider models that kind.Release note