|
| 1 | +# 0001. Pool labels and taints are `kubernetes-nodes` chart fields for every worker OS, not a Talos `userMachineConfig` layer |
| 2 | + |
| 3 | +- **Number:** `0001` |
| 4 | +- **Date:** `2026-10-22` |
| 5 | +- **Status:** Accepted |
| 6 | +- **Deciders:** `@lexfrei, @kvaps` |
| 7 | +- **Proposal:** [`../README.md`](../README.md) |
| 8 | +- **Decided in:** [`cozystack/community#103`](https://github.com/cozystack/community/pull/103) |
| 9 | +- **Implemented in:** not yet ([`cozystack/cozystack#4927`](https://github.com/cozystack/cozystack/pull/4927), open) |
| 10 | + |
| 11 | +## Context |
| 12 | + |
| 13 | +The proposal described a two-layer Talos machine config per pool: a chart-managed system layer and a `userMachineConfig` user layer in `values.yaml` for "extra kubelet args, extra labels/taints, registry mirror overrides, Talos image-factory schematic" ([Talos machineconfig: template + user overlay](../README.md#talos-machineconfig-template--user-overlay)). The split shipped without the user layer. Registry mirrors and the image-factory schematic became their own chart fields (`talos.registryMirrors`, `osImage`), and `userMachineConfig` was never added. The only labels a pool carried were its `roles`, as `node-role.kubernetes.io/<role>`, and `gpu: "on"` on a GPU pool. It had no taints, so GPU workers took ordinary workloads too. |
| 14 | + |
| 15 | +A pool was also about to stop being always Talos. `kubernetes-nodes` was gaining Kairos workers ([`cozystack/cozystack#4911`](https://github.com/cozystack/cozystack/pull/4911), which the implementation is built on), which join with `kubeadm join` through the Kairos Cluster API bootstrap provider and have no Talos machine config at all. |
| 16 | + |
| 17 | +Two mechanisms put labels on a Node, with different limits. Cluster API syncs labels from a Machine to its Node only for `node-role.kubernetes.io` and the `node-restriction.kubernetes.io` and `node.cluster.x-k8s.io` domains ([`util/labels.GetManagedLabels`](https://github.com/kubernetes-sigs/cluster-api/blob/v1.13.7/util/labels/helpers.go) in v1.13.7). Everything else goes through the worker's bootstrap config. On Kairos that is kubelet `--node-labels`, which admits only a restricted set of `kubernetes.io` and `k8s.io` keys ([`IsKubeletLabel`](https://github.com/kubernetes/kubernetes/blob/v1.35.6/staging/src/k8s.io/kubelet/pkg/apis/well_known_labels.go)). On Talos it is `machine.nodeLabels`, which machined writes to the Node with the kubelet's credentials. Cluster API v1.13 can also manage Machine taints, but behind the alpha `MachineTaintPropagation` feature gate, which is off by default ([`feature/feature.go`](https://github.com/kubernetes-sigs/cluster-api/blob/v1.13.7/feature/feature.go)). |
| 18 | + |
| 19 | +## Decision |
| 20 | + |
| 21 | +`KubernetesNodes` gains two chart fields that apply to every worker OS: `labels`, a string map, and `taints`, a list of taints with a key, an optional value and an effect. Labels in the `node-restriction.kubernetes.io` and `node.cluster.x-k8s.io` domains, which Cluster API syncs, go on the MachineDeployment template and reach the Nodes in place. Every other label and every taint goes through the pool's bootstrap config: `machine.nodeLabels` and the kubelet's `registerWithTaints` on Talos, kubeadm's `node-labels` and `nodeRegistration.taints` on Kairos. The chart refuses label keys and values Kubernetes would reject, labels that describe the node itself or that another writer owns, and taints in namespaces reserved for Kubernetes and Cluster API. The values schema rejects a malformed taint effect or key when Helm loads the values, before any template renders, and names the field; the rest is refused at render time. Admission does not check it: cozystack-api uses the schema for defaults only. Role labels (`node-role.kubernetes.io/`) are refused in `labels`, because `roles` already sets them and a role should have one way in. The rules live in the `kubernetes-nodes.labelsAndTaints` helper of `packages/apps/kubernetes-nodes/templates/_helpers.tpl` and are pinned by `packages/apps/kubernetes-nodes/tests/labels_taints_test.yaml`. |
| 22 | + |
| 23 | +## Why not the alternatives |
| 24 | + |
| 25 | +All three were weighed in [`cozystack/community#103`](https://github.com/cozystack/community/pull/103); [`cozystack/cozystack#4927`](https://github.com/cozystack/cozystack/pull/4927) implements the decision. |
| 26 | + |
| 27 | +- **A Talos `userMachineConfig` overlay, as the proposal described** ([proposal section](../README.md#talos-machineconfig-template--user-overlay)). It only exists for Talos, so a Kairos pool would need a second, different mechanism for the same thing. It also hands the tenant raw machine-config YAML, which the chart then has to merge and keep from overriding cluster-critical fields. Two typed fields do not have that problem. |
| 28 | +- **Machine `spec.taints` with Cluster API taint propagation.** It would let taints change without rolling the pool, but in Cluster API v1.13 it sits behind an alpha feature gate that is off by default ([`feature/feature.go`](https://github.com/kubernetes-sigs/cluster-api/blob/v1.13.7/feature/feature.go)). Turning it on is a platform-wide change to the core provider, and an alpha API can still change shape. This lost on maturity, not on design, so it is the first thing to revisit. |
| 29 | +- **All labels through the MachineDeployment template, with Cluster API's `--additional-sync-machine-labels`** ([metadata propagation](https://github.com/kubernetes-sigs/cluster-api/blob/v1.13.7/docs/book/src/reference/api/metadata-propagation.md)). It would make every label change in place. The flag takes regexes and applies to every tenant cluster the core provider manages. Tenant label keys are free-form, so a regex that covers them is in effect `.*` for every tenant, while a narrower prefix would force tenants into a Cozystack-owned prefix. This lost on a judgement call: registering the rest through the bootstrap config keeps the platform configuration unchanged. |
| 30 | + |
| 31 | +## Consequences |
| 32 | + |
| 33 | +- One field set works for Talos and Kairos pools, and a pool without labels or taints renders exactly as before. |
| 34 | +- Labels outside the Cluster API domains and all taints are part of the bootstrap config, so changing them rolls the pool. The field descriptions say which labels change in place. |
| 35 | +- Both paths write the remaining labels with the kubelet's identity, so the same NodeRestriction limits apply, and the chart applies one set of per-key label rules to both OSes. The exact set of refused keys is in the helper and its test, not restated here, because it tracks the Kubernetes, Talos and cloud-controller versions the platform ships. |
| 36 | +- For scheduling and isolation labels, `node-restriction.kubernetes.io/` keys are the recommended choice. Cluster API syncs them in place, and a kubelet cannot set them on its own Node. |
| 37 | +- A taint needs a tolerating side. Add-ons that must run on every node have to tolerate it. The kubevirt-csi node plugin tolerates everything (`operator: Exists`). The gpu-operator device plugin tolerates only `nvidia.com/gpu` with `NoSchedule`, and the HAMi device plugin, which replaces it when HAMi is enabled, tolerates no taint (`devicePlugin.tolerations: []`). A `NoSchedule` or `NoExecute` taint those plugins do not tolerate leaves a GPU pool without GPUs. A `PreferNoSchedule` taint only steers the scheduler and keeps nothing off the node. The field description says so, and making node-wide add-ons tolerate tenant taints is follow-up work. |
| 38 | +- The chart also announces the pool's labels and taints to cluster-autoscaler through the MachineDeployment capacity annotations, so scale from zero sees them, and it refuses a `gpu` label on a pool with `gpus`, because it sets that label itself. |
| 39 | +- The proposal's `userMachineConfig` user layer is not planned. Further per-pool settings follow the same pattern: a typed chart field that each OS path renders. |
| 40 | +- The user layer also listed extra kubelet arguments. This record does not decide them, and the chart still has no general way to pass them. |
| 41 | + |
| 42 | +## Revisit if |
| 43 | + |
| 44 | +Cluster API's `MachineTaintPropagation` reaches beta or is enabled on the platform for another reason. Taints could then move to Machine `spec.taints` and change without a rollout. |
0 commit comments