Skip to content

Commit b507cb5

Browse files
feat(kubernetes): allow a per-node-group Talos schematic
A Talos schematic is a fixed set of system extensions baked into one image, and Talos refuses to finish booting when an extension service in it cannot start. ext-nvidia-persistenced and ext-nvidia-cdi-gen require an NVIDIA card, so a node with no GPU that boots an NVIDIA schematic fails startAllServices and reboots, indefinitely. talos.schematicID was cluster-wide, so a cluster mixing GPU and non-GPU node groups had no correct value: the GPU group needs the extensions and every other group is broken by them. Setting the NVIDIA schematic for the sake of the GPU group put every other group into a reboot cycle of about 70 minutes per node. Nothing reports it — kubelet starts before the failing phase, so the node holds Ready and only the pod restart counters, identical across a node, betray it. Seen in production on 1.6.0. Add an optional per-node-group schematicID falling back to talos.schematicID, so the NVIDIA schematic can be scoped to the group that has the cards. Both consumers resolve it through one helper: the boot disk image the DataVolume pulls, and the installer image in the TalosConfigTemplate. They have to agree, or an in-place Talos upgrade swaps a node's extension set out from under it. Deliberately not derived from gpus, unlike kernelModules in the previous commit: a schematic ID is an opaque image-factory digest, so the chart cannot know which one carries the NVIDIA extensions or synthesise one. Unset renders byte-identically to before, so the content-hash-named KubevirtMachineTemplate keeps its name and no worker is rolled. Setting it does roll that group's workers, which is inherent — changing a node's boot image means replacing the node. Refs: cozystack#3563 Assisted-By: Claude <[email protected]> Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent 41f4cdc commit b507cb5

19 files changed

Lines changed: 508 additions & 60 deletions

File tree

‎api/apps/v1alpha1/kubernetes/types.go‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetesnodes/types.go‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/kubernetes-nodes/README.md‎

Lines changed: 25 additions & 24 deletions
Large diffs are not rendered by default.

‎packages/apps/kubernetes-nodes/templates/_helpers.tpl‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,50 @@ creates `/dev/nvidia0`, and then finds no devices.
107107
{{- end -}}
108108
{{- end }}
109109

110+
{{/*
111+
Effective Talos image-factory schematic ID for this worker pool.
112+
113+
Byte-for-byte the same helper as the parent kubernetes chart's, for the same
114+
reason as kubernetes.kernelModules: both charts render the boot disk image and
115+
the installer image, and tests/render-parity.sh compares both.
116+
117+
Takes a context carrying .group (the pool) and .Values, returns the pool's own
118+
schematicID when set, otherwise the cluster-wide talos.schematicID.
119+
120+
Why this needs to be per-pool at all: a schematic is a fixed set of system
121+
extensions baked into one image, and Talos refuses to finish booting when an
122+
extension service in it cannot start. `ext-nvidia-persistenced` and
123+
`ext-nvidia-cdi-gen` need an NVIDIA card, so a node with no GPU that boots the
124+
NVIDIA schematic fails `startAllServices` and reboots, every time, forever.
125+
Observed in production as a ~70 minute reboot cycle on a non-GPU pool in a
126+
cluster whose only GPU pool had forced the cluster-wide schematic to an NVIDIA
127+
one. The nodes stay `Ready` throughout — kubelet starts before the failing
128+
phase — so `kubectl get nodes` shows nothing and only the pod restart counters
129+
betray it.
130+
131+
With one schematic per cluster, a cluster that mixes GPU and non-GPU pools has
132+
no correct value to set: the GPU pool needs the extensions and every other pool
133+
is broken by them. This field is what makes such a cluster expressible.
134+
135+
Both consumers of the schematic have to agree on it, or an in-place upgrade
136+
silently swaps a node's extension set: the boot disk image the DataVolume pulls
137+
(nodegroup.yaml) and the installer image in the TalosConfigTemplate
138+
(talos-reconcile-job.yaml). Both call this helper.
139+
140+
Deliberately NOT auto-derived from `gpus`, unlike kernelModules: a schematic ID
141+
is an opaque image-factory digest, so the chart cannot know which one carries
142+
the NVIDIA extensions, or synthesise one. That part stays the operator's to
143+
supply.
144+
145+
Leaving it unset renders byte-identical output to a chart without this field,
146+
so the pool's content-hash-named KubevirtMachineTemplate keeps its name and its
147+
workers are not rolled. Setting it does roll that pool's workers, which is
148+
inherent: changing a node's boot image means replacing the node.
149+
*/}}
150+
{{- define "kubernetes.schematicID" -}}
151+
{{- .group.schematicID | default .Values.talos.schematicID -}}
152+
{{- end }}
153+
110154
{{/*
111155
Reconstruct the parent CAPI cluster name from the linkage value.
112156

‎packages/apps/kubernetes-nodes/templates/nodegroup.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@
4141
"resources" .Values.resources
4242
"gpus" .Values.gpus
4343
"kubelet" .Values.kubelet
44+
"schematicID" .Values.schematicID
4445
}}
4546
{{- define "kubevirtmachinetemplate" -}}
4647
spec:
@@ -82,7 +83,7 @@ spec:
8283
ignores configdrive and never applies the machineconfig. */}}
8384
source:
8485
http:
85-
url: {{ $.Values.talos.imageFactoryURL | trimSuffix "/" }}/image/{{ $.Values.talos.schematicID }}/{{ $.Values.talos.version }}/openstack-amd64.raw.xz
86+
url: {{ $.Values.talos.imageFactoryURL | trimSuffix "/" }}/image/{{ include "kubernetes.schematicID" . }}/{{ $.Values.talos.version }}/openstack-amd64.raw.xz
8687
storage:
8788
resources:
8889
requests:

‎packages/apps/kubernetes-nodes/templates/talos-reconcile-job.yaml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -332,7 +332,11 @@ spec:
332332
memory.available: "256Mi"
333333
install:
334334
disk: /dev/vda
335-
image: {{ $.Values.talos.installerRepository | trimSuffix "/" }}/{{ $.Values.talos.schematicID }}:{{ .talosVersion }}
335+
{{- /* Must resolve to the same schematic as the boot disk image in
336+
nodegroup.yaml: this installer is what an in-place Talos
337+
upgrade runs, so a mismatch would swap the node's extension
338+
set out from under it. Both call kubernetes.schematicID. */}}
339+
image: {{ $.Values.talos.installerRepository | trimSuffix "/" }}/{{ include "kubernetes.schematicID" . }}:{{ .talosVersion }}
336340
wipe: false
337341
features:
338342
rbac: true
@@ -400,6 +404,7 @@ spec:
400404
"gpus" .Values.gpus
401405
"kubelet" .Values.kubelet
402406
"kernelModules" .Values.kernelModules
407+
"schematicID" .Values.schematicID
403408
}}
404409
{{- /* Mirror the kubelet-reservation computation from nodegroup.yaml so the
405410
worker machineconfig this Job applies carries the same numbers the

‎packages/apps/kubernetes-nodes/tests/render-parity.sh‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -229,8 +229,28 @@ kernelModules: []
229229
kubelet: {}
230230
EOF
231231

232+
# --- Case: per-pool Talos schematic override. Covers both consumers at once — the
233+
# boot disk image is in the KubevirtMachineTemplate compared above, the installer
234+
# image is in the machine config — so a chart that overrode one and not the other
235+
# fails here. ---
236+
cat >"$WORK/case-schematic.yaml" <<'EOF'
237+
minReplicas: 0
238+
maxReplicas: 3
239+
instanceType: ""
240+
diskSize: 20Gi
241+
storageClass: replicated
242+
roles: []
243+
resources:
244+
cpu: "4"
245+
memory: 8Gi
246+
gpus:
247+
- name: nvidia.com/AD102GL_L40S
248+
schematicID: "aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999"
249+
kubelet: {}
250+
EOF
251+
232252
RC=0
233-
for c in resources gpu kubelet kernelmodules kernelmodules-optout; do
253+
for c in resources gpu kubelet kernelmodules kernelmodules-optout schematic; do
234254
write_values "$WORK/case-${c}.yaml"
235255
diff_kinds "$c" || RC=1
236256
done
Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
suite: per-pool Talos schematic tests
2+
3+
# Mirror of packages/apps/kubernetes/tests/schematic_per_nodegroup_test.yaml. Both
4+
# charts render the worker boot disk image and the Talos installer image, so both
5+
# need the override and both need it pinned; tests/render-parity.sh additionally
6+
# compares the two charts' output against each other.
7+
#
8+
# A Talos schematic is a fixed set of system extensions baked into one image, and
9+
# Talos refuses to finish booting when an extension service in it cannot start.
10+
# `ext-nvidia-persistenced` and `ext-nvidia-cdi-gen` need an NVIDIA card, so a node
11+
# with no GPU that boots the NVIDIA schematic fails `startAllServices` and reboots —
12+
# observed in production as a ~70 minute cycle on every node of a non-GPU pool, in a
13+
# cluster whose GPU pool had forced the cluster-wide `talos.schematicID` to an NVIDIA
14+
# one. The nodes report `Ready` throughout, because kubelet starts before the failing
15+
# phase, so nothing surfaces in `kubectl get nodes`.
16+
#
17+
# Assertions are scoped with `template:` because helm-unittest evaluates every
18+
# assertion against every template in the suite, so a `kind`-based selector fails on
19+
# whichever template does not contain that kind.
20+
21+
templates:
22+
- templates/nodegroup.yaml
23+
- templates/talos-reconcile-job.yaml
24+
release:
25+
name: kubernetes-nodes-myk8s-md0
26+
namespace: tenant-test
27+
set:
28+
cluster: myk8s
29+
_cluster:
30+
cluster-domain: cozy.local
31+
version: "v1.35"
32+
minReplicas: 0
33+
maxReplicas: 3
34+
instanceType: ""
35+
diskSize: 20Gi
36+
storageClass: replicated
37+
roles: []
38+
resources:
39+
cpu: "4"
40+
memory: 8Gi
41+
gpus: []
42+
43+
tests:
44+
- it: applies a per-pool schematic to both the boot image and the installer
45+
set:
46+
schematicID: "aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999"
47+
asserts:
48+
- equal:
49+
path: spec.template.spec.virtualMachineTemplate.spec.dataVolumeTemplates[0].spec.source.http.url
50+
value: https://factory.talos.dev/image/aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999/v1.13.6/openstack-amd64.raw.xz
51+
template: templates/nodegroup.yaml
52+
documentSelector:
53+
path: kind
54+
value: KubevirtMachineTemplate
55+
# The installer must match the boot image. A mismatch is worse than a wrong
56+
# value: the node boots one extension set and upgrades into another.
57+
- matchRegex:
58+
path: spec.template.spec.containers[0].command[2]
59+
pattern: 'image: factory\.talos\.dev/installer/aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999:v1\.13\.6'
60+
template: templates/talos-reconcile-job.yaml
61+
documentSelector:
62+
path: kind
63+
value: Job
64+
65+
- it: falls back to the cluster-wide schematic when the pool does not set one
66+
set:
67+
talos:
68+
schematicID: "1111111111111111111111111111111111111111111111111111111111111111"
69+
asserts:
70+
- equal:
71+
path: spec.template.spec.virtualMachineTemplate.spec.dataVolumeTemplates[0].spec.source.http.url
72+
value: https://factory.talos.dev/image/1111111111111111111111111111111111111111111111111111111111111111/v1.13.6/openstack-amd64.raw.xz
73+
template: templates/nodegroup.yaml
74+
documentSelector:
75+
path: kind
76+
value: KubevirtMachineTemplate
77+
- matchRegex:
78+
path: spec.template.spec.containers[0].command[2]
79+
pattern: 'image: factory\.talos\.dev/installer/1111111111111111111111111111111111111111111111111111111111111111:v1\.13\.6'
80+
template: templates/talos-reconcile-job.yaml
81+
documentSelector:
82+
path: kind
83+
value: Job
84+
85+
- it: treats an empty per-pool schematic as unset
86+
set:
87+
talos:
88+
schematicID: "1111111111111111111111111111111111111111111111111111111111111111"
89+
schematicID: ""
90+
asserts:
91+
# An empty string inherits rather than rendering an image URL with an empty
92+
# schematic path segment, which would 404 at the factory and leave the VM
93+
# stuck importing forever. This is also the chart default, so it is the path
94+
# every existing pool takes.
95+
- equal:
96+
path: spec.template.spec.virtualMachineTemplate.spec.dataVolumeTemplates[0].spec.source.http.url
97+
value: https://factory.talos.dev/image/1111111111111111111111111111111111111111111111111111111111111111/v1.13.6/openstack-amd64.raw.xz
98+
template: templates/nodegroup.yaml
99+
documentSelector:
100+
path: kind
101+
value: KubevirtMachineTemplate

‎packages/apps/kubernetes-nodes/values.schema.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,11 @@
108108
}
109109
}
110110
},
111+
"schematicID": {
112+
"description": "Per-pool override for `talos.schematicID`, applied to both the worker boot disk image and the Talos installer image. When empty, the cluster-wide `talos.schematicID` applies. A schematic is a fixed set of Talos system extensions, and Talos refuses to finish booting when an extension service in it cannot start: `ext-nvidia-persistenced` and `ext-nvidia-cdi-gen` require an NVIDIA card, so a pool with no GPU that boots an NVIDIA schematic fails `startAllServices` and reboots roughly every 70 minutes, indefinitely, while still reporting `Ready` (kubelet starts before the failing phase). A cluster mixing GPU and non-GPU pools therefore has no correct cluster-wide value, and this field is what makes it expressible: set the NVIDIA schematic on the GPU pool only. Changing it replaces the pool's boot image and so rolls its workers.",
113+
"type": "string",
114+
"default": ""
115+
},
111116
"kernelModules": {
112117
"description": "Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia`, `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The module still has to be in the image: which extension supplies it is set by `talos.schematicID`, and on Blackwell (GB202) it must be the open-kernel-modules extension.",
113118
"type": "array",

‎packages/apps/kubernetes-nodes/values.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,9 @@ resources: {}
5959
## @param {[]GPU} gpus - List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).
6060
gpus: []
6161

62+
## @param {string} [schematicID] - Per-pool override for `talos.schematicID`, applied to both the worker boot disk image and the Talos installer image. When empty, the cluster-wide `talos.schematicID` applies. A schematic is a fixed set of Talos system extensions, and Talos refuses to finish booting when an extension service in it cannot start: `ext-nvidia-persistenced` and `ext-nvidia-cdi-gen` require an NVIDIA card, so a pool with no GPU that boots an NVIDIA schematic fails `startAllServices` and reboots roughly every 70 minutes, indefinitely, while still reporting `Ready` (kubelet starts before the failing phase). A cluster mixing GPU and non-GPU pools therefore has no correct cluster-wide value, and this field is what makes it expressible: set the NVIDIA schematic on the GPU pool only. Changing it replaces the pool's boot image and so rolls its workers.
63+
schematicID: ""
64+
6265
## @param {[]KernelModule} [kernelModules] - Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia`, `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The module still has to be in the image: which extension supplies it is set by `talos.schematicID`, and on Blackwell (GB202) it must be the open-kernel-modules extension.
6366

6467
## @param {Kubelet} [kubelet] - Kubelet resource reservations for this pool.

0 commit comments

Comments
 (0)