Skip to content

Commit 02dc629

Browse files
committed
docs(kubernetes-nodes-split): record labels and taints as chart fields
The proposal put per-pool labels and taints into a Talos userMachineConfig layer that was never built, and kubernetes-nodes is gaining Kairos workers, which have no Talos machine config. Record the decision to make labels and taints typed chart fields for every worker OS instead, why the userMachineConfig overlay, Cluster API taint propagation and syncing every label through --additional-sync-machine-labels lost, and when to revisit. The record also says which rules the chart applies: role labels come only from `roles`, a malformed taint fails when Helm loads the values, and the chart owns the `gpu` label on a GPU pool. It names the cost of a taint: node-wide add-ons must tolerate it, and the GPU device plugins tolerate at most nvidia.com/gpu with NoSchedule, so a tainted GPU pool can end up without GPUs. The proposal links the record and notes that its user layer was not built. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 5ccd161 commit 02dc629

2 files changed

Lines changed: 50 additions & 0 deletions

File tree

‎design-proposals/kubernetes-nodes-split/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,10 @@ Hybrid clusters — workers that live outside the Cozystack management cluster (
1717
- **Phase 3 (hybrid clusters)** lives in a separate draft proposal: [`kubernetes-nodes-hybrid-clusters`](../kubernetes-nodes-hybrid-clusters/) ([PR #9](https://github.com/cozystack/community/pull/9)). None of the design here forecloses Phase 3; the package split is exactly what makes Phase 3 expressible cleanly.
1818
- **Companion: [`cross-cluster-tenant-mesh`](../cross-cluster-tenant-mesh/)** (PR #7). Independent of this proposal; relevant once tenants need to reach services across cluster boundaries.
1919

20+
## Decisions
21+
22+
- [0001. Pool labels and taints are `kubernetes-nodes` chart fields for every worker OS, not a Talos `userMachineConfig` layer](./decisions/0001-pool-labels-and-taints-are-chart-fields-for-every-os.md) — why the `userMachineConfig` user layer below was not built.
23+
2024
## Context
2125

2226
### The current shape
@@ -121,6 +125,8 @@ Each `kubernetes-nodes` HelmRelease generates a single Talos machineconfig for i
121125

122126
The two layers are merged at render time; the user never writes raw Talos YAML for cluster-critical fields. The chart guarantees the machineconfig results in a worker that joins the right Kamaji control-plane.
123127

128+
The user layer was not built. Registry mirrors and the image-factory schematic became chart fields of their own, and labels and taints are being added as chart fields for every worker OS ([`cozystack/cozystack#4927`](https://github.com/cozystack/cozystack/pull/4927)); see [decision 0001](./decisions/0001-pool-labels-and-taints-are-chart-fields-for-every-os.md).
129+
124130
### `kubevirt-ccm` stays in `kubernetes` package
125131

126132
`kubevirt-ccm` is logically a cluster-level property, not a per-pool one. It remains in the `kubernetes` package and is rendered once per tenant cluster, independent of how many `kubernetes-nodes` HelmReleases attach to it.
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# 0001. Pool labels and taints are `kubernetes-nodes` chart fields for every worker OS, not a Talos `userMachineConfig` layer
2+
3+
- **Number:** `0001`
4+
- **Date:** `2026-10-22`
5+
- **Status:** Accepted
6+
- **Deciders:** `@lexfrei, @kvaps`
7+
- **Proposal:** [`../README.md`](../README.md)
8+
- **Decided in:** [`cozystack/community#103`](https://github.com/cozystack/community/pull/103)
9+
- **Implemented in:** not yet ([`cozystack/cozystack#4927`](https://github.com/cozystack/cozystack/pull/4927), open)
10+
11+
## Context
12+
13+
The proposal described a two-layer Talos machine config per pool: a chart-managed system layer and a `userMachineConfig` user layer in `values.yaml` for "extra kubelet args, extra labels/taints, registry mirror overrides, Talos image-factory schematic" ([Talos machineconfig: template + user overlay](../README.md#talos-machineconfig-template--user-overlay)). The split shipped without the user layer. Registry mirrors and the image-factory schematic became their own chart fields (`talos.registryMirrors`, `osImage`), and `userMachineConfig` was never added. The only labels a pool carried were its `roles`, as `node-role.kubernetes.io/<role>`, and `gpu: "on"` on a GPU pool. It had no taints, so GPU workers took ordinary workloads too.
14+
15+
A pool was also about to stop being always Talos. `kubernetes-nodes` was gaining Kairos workers ([`cozystack/cozystack#4911`](https://github.com/cozystack/cozystack/pull/4911), which the implementation is built on), which join with `kubeadm join` through the Kairos Cluster API bootstrap provider and have no Talos machine config at all.
16+
17+
Two mechanisms put labels on a Node, with different limits. Cluster API syncs labels from a Machine to its Node only for `node-role.kubernetes.io` and the `node-restriction.kubernetes.io` and `node.cluster.x-k8s.io` domains ([`util/labels.GetManagedLabels`](https://github.com/kubernetes-sigs/cluster-api/blob/v1.13.7/util/labels/helpers.go) in v1.13.7). Everything else goes through the worker's bootstrap config. On Kairos that is kubelet `--node-labels`, which admits only a restricted set of `kubernetes.io` and `k8s.io` keys ([`IsKubeletLabel`](https://github.com/kubernetes/kubernetes/blob/v1.35.6/staging/src/k8s.io/kubelet/pkg/apis/well_known_labels.go)). On Talos it is `machine.nodeLabels`, which machined writes to the Node with the kubelet's credentials. Cluster API v1.13 can also manage Machine taints, but behind the alpha `MachineTaintPropagation` feature gate, which is off by default ([`feature/feature.go`](https://github.com/kubernetes-sigs/cluster-api/blob/v1.13.7/feature/feature.go)).
18+
19+
## Decision
20+
21+
`KubernetesNodes` gains two chart fields that apply to every worker OS: `labels`, a string map, and `taints`, a list of taints with a key, an optional value and an effect. Labels in the `node-restriction.kubernetes.io` and `node.cluster.x-k8s.io` domains, which Cluster API syncs, go on the MachineDeployment template and reach the Nodes in place. Every other label and every taint goes through the pool's bootstrap config: `machine.nodeLabels` and the kubelet's `registerWithTaints` on Talos, kubeadm's `node-labels` and `nodeRegistration.taints` on Kairos. The chart refuses label keys and values Kubernetes would reject, labels that describe the node itself or that another writer owns, and taints in namespaces reserved for Kubernetes and Cluster API. The values schema rejects a malformed taint effect or key when Helm loads the values, before any template renders, and names the field; the rest is refused at render time. Admission does not check it: cozystack-api uses the schema for defaults only. Role labels (`node-role.kubernetes.io/`) are refused in `labels`, because `roles` already sets them and a role should have one way in. The rules live in the `kubernetes-nodes.labelsAndTaints` helper of `packages/apps/kubernetes-nodes/templates/_helpers.tpl` and are pinned by `packages/apps/kubernetes-nodes/tests/labels_taints_test.yaml`.
22+
23+
## Why not the alternatives
24+
25+
All three were weighed in [`cozystack/community#103`](https://github.com/cozystack/community/pull/103); [`cozystack/cozystack#4927`](https://github.com/cozystack/cozystack/pull/4927) implements the decision.
26+
27+
- **A Talos `userMachineConfig` overlay, as the proposal described** ([proposal section](../README.md#talos-machineconfig-template--user-overlay)). It only exists for Talos, so a Kairos pool would need a second, different mechanism for the same thing. It also hands the tenant raw machine-config YAML, which the chart then has to merge and keep from overriding cluster-critical fields. Two typed fields do not have that problem.
28+
- **Machine `spec.taints` with Cluster API taint propagation.** It would let taints change without rolling the pool, but in Cluster API v1.13 it sits behind an alpha feature gate that is off by default ([`feature/feature.go`](https://github.com/kubernetes-sigs/cluster-api/blob/v1.13.7/feature/feature.go)). Turning it on is a platform-wide change to the core provider, and an alpha API can still change shape. This lost on maturity, not on design, so it is the first thing to revisit.
29+
- **All labels through the MachineDeployment template, with Cluster API's `--additional-sync-machine-labels`** ([metadata propagation](https://github.com/kubernetes-sigs/cluster-api/blob/v1.13.7/docs/book/src/reference/api/metadata-propagation.md)). It would make every label change in place. The flag takes regexes and applies to every tenant cluster the core provider manages. Tenant label keys are free-form, so a regex that covers them is in effect `.*` for every tenant, while a narrower prefix would force tenants into a Cozystack-owned prefix. This lost on a judgement call: registering the rest through the bootstrap config keeps the platform configuration unchanged.
30+
31+
## Consequences
32+
33+
- One field set works for Talos and Kairos pools, and a pool without labels or taints renders exactly as before.
34+
- Labels outside the Cluster API domains and all taints are part of the bootstrap config, so changing them rolls the pool. The field descriptions say which labels change in place.
35+
- Both paths write the remaining labels with the kubelet's identity, so the same NodeRestriction limits apply, and the chart applies one set of per-key label rules to both OSes. The exact set of refused keys is in the helper and its test, not restated here, because it tracks the Kubernetes, Talos and cloud-controller versions the platform ships.
36+
- For scheduling and isolation labels, `node-restriction.kubernetes.io/` keys are the recommended choice. Cluster API syncs them in place, and a kubelet cannot set them on its own Node.
37+
- A taint needs a tolerating side. Add-ons that must run on every node have to tolerate it. The kubevirt-csi node plugin tolerates everything (`operator: Exists`). The gpu-operator device plugin tolerates only `nvidia.com/gpu` with `NoSchedule`, and the HAMi device plugin, which replaces it when HAMi is enabled, tolerates no taint (`devicePlugin.tolerations: []`). A `NoSchedule` or `NoExecute` taint those plugins do not tolerate leaves a GPU pool without GPUs. A `PreferNoSchedule` taint only steers the scheduler and keeps nothing off the node. The field description says so, and making node-wide add-ons tolerate tenant taints is follow-up work.
38+
- The chart also announces the pool's labels and taints to cluster-autoscaler through the MachineDeployment capacity annotations, so scale from zero sees them, and it refuses a `gpu` label on a pool with `gpus`, because it sets that label itself.
39+
- The proposal's `userMachineConfig` user layer is not planned. Further per-pool settings follow the same pattern: a typed chart field that each OS path renders.
40+
- The user layer also listed extra kubelet arguments. This record does not decide them, and the chart still has no general way to pass them.
41+
42+
## Revisit if
43+
44+
Cluster API's `MachineTaintPropagation` reaches beta or is enabled on the platform for another reason. Taints could then move to Machine `spec.taints` and change without a rollout.

0 commit comments

Comments
 (0)