docs(kubernetes-nodes-split): record labels and taints as chart fields by lexfrei · Pull Request #103 · cozystack/community

Aleksei Sviridkin (lexfrei) · 2026-10-15T08:45:34Z

This records a change of course in the proposal that split worker pools out of the kubernetes app. The proposal planned pool labels and taints as part of a Talos userMachineConfig layer, and that layer was never built. A pool could carry only its roles and the gpu label and had no taints, so GPU workers took ordinary workloads too.

The decision is to make labels and taints two typed chart fields that work for Talos and Kairos workers alike. The record also says why the Talos overlay, Cluster API taint propagation and syncing every label through --additional-sync-machine-labels lost. The implementation is cozystack/cozystack#4927, checked on a test cluster with one Talos pool and one Kairos pool. The proposal README gets a Decisions section that links the record and notes that its user layer was not built.

Before review

If this revises a merged design proposal: it adds a decision record under that proposal's decisions/ directory, or says below why none is needed.

This is the decision record for a change of course in kubernetes-nodes-split.

DCO

Commits are signed off (git commit --signoff).

coderabbitai · 2026-10-15T08:45:44Z

Important

🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: 97e7514e-659c-414f-89e0-01064071313a

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Andrei Kvapil (kvaps)

LGTM as of c6969a1. Two typed fields for both OSes is better than a Talos-only overlay, and the reasons for not using MachineTaintPropagation and --additional-sync-machine-labels are clear.

nits:

"Decided in" should probably be this PR. cozystack/cozystack#4927 is a draft, it's where it's implemented.
The implementation also writes the autoscaler labels and taints annotations and refuses a gpu label on a pool with gpus. One line in Consequences would be enough.
This PR touches the README, and two lines there are going stale with cozystack/cozystack#4911: "No Ubuntu + kubeadm path remains" and "only kubevirt-talos". Fine to leave them for the Kairos proposal.

myasnikovdaniil · 2026-10-23T14:06:41Z

Hi! We removed Status and Date from proposal headers in #111. Could you remove these fields when you next update your proposal? Decision records keep their metadata.

Sorry for the extra work, I wanted to mention this now so it doesn't come up late in review.

Timofei Larkin (lllamnyp) · 2026-10-29T12:31:37Z

+
+## Decision
+
+`KubernetesNodes` gains two chart fields that apply to every worker OS: `labels`, a string map, and `taints`, a list of taints with a key, an optional value and an effect. Labels in the domains Cluster API syncs go on the MachineDeployment template and reach the Nodes in place. Every other label and every taint goes through the pool's bootstrap config: `machine.nodeLabels` and the kubelet's `registerWithTaints` on Talos, kubeadm's `node-labels` and `nodeRegistration.taints` on Kairos. The chart refuses, at render time, label keys and values Kubernetes would reject, labels that describe the node itself or that another writer owns, and taints in namespaces reserved for Kubernetes and Cluster API. The rules live in the `kubernetes-nodes.labelsAndTaints` helper of `packages/apps/kubernetes-nodes/templates/_helpers.tpl` and are pinned by `packages/apps/kubernetes-nodes/tests/labels_taints_test.yaml`.


Taints need a tolerating side that this record doesn't cover. The HAMi device plugin that Cozystack installs in tenant clusters has no tolerations (devicePlugin.tolerations: []), so a GPU pool tainted the way Context motivates would run no device plugin and expose no GPUs. The kubevirt-csi node plugin is fine (operator: Exists). Could Consequences say who tolerates tenant taints? Either platform DaemonSets that must run on every node tolerate operator: Exists, or the chart documents a GPU taint key that its addons tolerate.

Added to Consequences: node-wide add-ons have to tolerate a pool taint. The kubevirt-csi node plugin does, while gpu-operator tolerates only nvidia.com/gpu with NoSchedule and the HAMi device plugin, which replaces it when HAMi is on, tolerates nothing. So a NoSchedule or NoExecute taint they don't tolerate leaves a GPU pool without GPUs. The taints field description in cozystack/cozystack#4927 says the same, and making the add-ons tolerate tenant taints is left as follow-up.

Timofei Larkin (lllamnyp) · 2026-10-29T12:31:37Z

+
+## Decision
+
+`KubernetesNodes` gains two chart fields that apply to every worker OS: `labels`, a string map, and `taints`, a list of taints with a key, an optional value and an effect. Labels in the domains Cluster API syncs go on the MachineDeployment template and reach the Nodes in place. Every other label and every taint goes through the pool's bootstrap config: `machine.nodeLabels` and the kubelet's `registerWithTaints` on Talos, kubeadm's `node-labels` and `nodeRegistration.taints` on Kairos. The chart refuses, at render time, label keys and values Kubernetes would reject, labels that describe the node itself or that another writer owns, and taints in namespaces reserved for Kubernetes and Cluster API. The rules live in the `kubernetes-nodes.labelsAndTaints` helper of `packages/apps/kubernetes-nodes/templates/_helpers.tpl` and are pinned by `packages/apps/kubernetes-nodes/tests/labels_taints_test.yaml`.


Nit: the effect enum and the key pattern fit in the API schema (+kubebuilder:validation:Enum on effect). Then a bad pool is rejected at apply instead of surfacing later as a failed release. Maybe "refuses at admission where the schema can express it, at render time otherwise"?

The enum on effect and the key pattern are in the schema now (#4927). One correction to "rejected at apply": cozystack-api uses the values schema only for defaults and doesn't validate the spec against it, so the schema fires when Helm loads the values, before any template renders, and the error names the field. The record says it that way.

Timofei Larkin (lllamnyp) · 2026-10-29T12:31:38Z

+## Consequences
+
+- One field set works for Talos and Kairos pools, and a pool without labels or taints renders exactly as before.
+- Labels outside the Cluster API domains and all taints are part of the bootstrap config, so changing them rolls the pool. The field descriptions say which labels change in place.


The same labels field applies in place or rolls the pool depending on the key's prefix, so a one-label edit on a GPU pool replaces every node. Worth one line recommending node-restriction.kubernetes.io/ keys for scheduling labels: they change in place, and a kubelet can't set them on itself, so they're also the safe choice for isolation.

Added to Consequences as the recommended choice for scheduling and isolation labels, with both reasons: Cluster API syncs them in place, and a kubelet can't set them on its own Node.

Timofei Larkin (lllamnyp) · 2026-10-29T12:31:38Z

+
+- One field set works for Talos and Kairos pools, and a pool without labels or taints renders exactly as before.
+- Labels outside the Cluster API domains and all taints are part of the bootstrap config, so changing them rolls the pool. The field descriptions say which labels change in place.
+- The chart applies one set of per-key label rules to both OSes, including where Talos alone would accept more. The exact set of refused keys is in the helper and its test, not restated here, because it tracks the Kubernetes, Talos and cloud-controller versions the platform ships.


Which keys does Talos accept that the Kairos path refuses? Context says machined writes machine.nodeLabels with the kubelet's credentials, so NodeRestriction applies to both. Separately: labels accepts node-role.kubernetes.io/<x> for any role not in roles, which gives two ways to declare a role. Refusing that domain in labels and pointing to roles would keep one.

None, the claim was wrong. Both paths write with the kubelet's identity, so NodeRestriction limits both the same way, and the sentence now says that. node-role.kubernetes.io/ keys are refused in labels with a pointer to roles, so a role has one way in (#4927 now refuses them and its tests were flipped).

Timofei Larkin (lllamnyp) · 2026-10-29T12:31:38Z


 The two layers are merged at render time; the user never writes raw Talos YAML for cluster-critical fields. The chart guarantees the machineconfig results in a worker that joins the right Kamaji control-plane.

+The user layer was not built. Registry mirrors and the image-factory schematic became chart fields of their own, and labels and taints became chart fields for every worker OS; see [decision 0001](./decisions/0001-pool-labels-and-taints-are-chart-fields-for-every-os.md).


The past tense reads as current state, but on main neither field exists yet and Talos is the only worker OS (the implementation and Kairos pools are both still drafts). Suggest "are being added as chart fields (cozystack/cozystack#4927)" until that merges, since the record's header already tracks implementation state.

Changed to "are being added as chart fields (cozystack/cozystack#4927)".

The proposal put per-pool labels and taints into a Talos userMachineConfig layer that was never built, and kubernetes-nodes is gaining Kairos workers, which have no Talos machine config. Record the decision to make labels and taints typed chart fields for every worker OS instead, why the userMachineConfig overlay, Cluster API taint propagation and syncing every label through --additional-sync-machine-labels lost, and when to revisit. The record also says which rules the chart applies: role labels come only from `roles`, a malformed taint fails when Helm loads the values, and the chart owns the `gpu` label on a GPU pool. It names the cost of a taint: node-wide add-ons must tolerate it, and the GPU device plugins tolerate at most nvidia.com/gpu with NoSchedule, so a tainted GPU pool can end up without GPUs. The proposal links the record and notes that its user layer was not built. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>

Aleksei Sviridkin (lexfrei) · 2026-10-29T19:46:38Z

Andrei Kvapil (@kvaps) The record changed since your approval: it takes lllamnyp's five points, your three nits (Decided in is this PR, plus the autoscaler annotations and the refused gpu label), and drops the os: kairos field that #4911 no longer has. Could you take another look at 02dc629?

I'd also merge this after cozystack/cozystack#4927. The record names rules that the implementation must keep, and once it merges its prose is frozen.

Andrei Kvapil (kvaps)

LGTM as of 02dc629. Agree to merge it after cozystack/cozystack#4927.

Andrei Kvapil (kvaps) previously approved these changes Oct 22, 2026

View reviewed changes

Andrei Kvapil (kvaps) mentioned this pull request Oct 22, 2026

docs(design-proposals): add GPUDevice proposal #99

Merged

2 tasks done

Timofei Larkin (lllamnyp) reviewed Oct 29, 2026

View reviewed changes

Timofei Larkin (lllamnyp) mentioned this pull request Oct 29, 2026

docs(design-proposals): add WorkerClass and Kairos worker pools #115

Merged

2 tasks done

Aleksei Sviridkin (lexfrei) dismissed Andrei Kvapil (kvaps)’s stale review via 02dc629 October 29, 2026 18:46

Aleksei Sviridkin (lexfrei) force-pushed the design/kubernetes-nodes-labels-taints branch from c6969a1 to 02dc629 Compare October 29, 2026 18:46

Andrei Kvapil (kvaps) approved these changes Nov 2, 2026

View reviewed changes

Aleksei Sviridkin (lexfrei) merged commit ff067a9 into main Nov 3, 2026
2 checks passed

Aleksei Sviridkin (lexfrei) deleted the design/kubernetes-nodes-labels-taints branch November 3, 2026 08:18

Aleksei Sviridkin (lexfrei) mentioned this pull request Nov 10, 2026

kubernetes node pools: decide who tolerates tenant taints for node-wide add-ons #129

Open


		## Decision

		`KubernetesNodes` gains two chart fields that apply to every worker OS: `labels`, a string map, and `taints`, a list of taints with a key, an optional value and an effect. Labels in the domains Cluster API syncs go on the MachineDeployment template and reach the Nodes in place. Every other label and every taint goes through the pool's bootstrap config: `machine.nodeLabels` and the kubelet's `registerWithTaints` on Talos, kubeadm's `node-labels` and `nodeRegistration.taints` on Kairos. The chart refuses, at render time, label keys and values Kubernetes would reject, labels that describe the node itself or that another writer owns, and taints in namespaces reserved for Kubernetes and Cluster API. The rules live in the `kubernetes-nodes.labelsAndTaints` helper of `packages/apps/kubernetes-nodes/templates/_helpers.tpl` and are pinned by `packages/apps/kubernetes-nodes/tests/labels_taints_test.yaml`.


		The two layers are merged at render time; the user never writes raw Talos YAML for cluster-critical fields. The chart guarantees the machineconfig results in a worker that joins the right Kamaji control-plane.

		The user layer was not built. Registry mirrors and the image-factory schematic became chart fields of their own, and labels and taints became chart fields for every worker OS; see [decision 0001](./decisions/0001-pool-labels-and-taints-are-chart-fields-for-every-os.md).

Conversation

Aleksei Sviridkin (lexfrei) commented Oct 15, 2026

Before review

DCO

Uh oh!

coderabbitai Bot commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Andrei Kvapil (kvaps) left a comment

Choose a reason for hiding this comment

Uh oh!

myasnikovdaniil commented Oct 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aleksei Sviridkin (lexfrei) commented Oct 29, 2026

Uh oh!

Andrei Kvapil (kvaps) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

coderabbitai Bot commented Oct 15, 2026 •

edited

Loading