Skip to content

Commit 669fb24

Browse files
feat(kubernetes): default gpu-operator to an OS-provided driver on Talos
Closes the gpu-operator half of cozystack#3563. Tenant clusters have run Talos workers since 1.6.0, where the NVIDIA driver comes from a system extension baked into the worker image rather than from the operator's driver container. The addon default still described the Ubuntu shape, so a GPU node group came up with the operator fighting the OS for the driver and no path at all for CUDA libraries to reach a tenant pod. Three things change, each fixing one silent failure: driver.enabled and toolkit.enabled go false. The driver container would load its own module and clash with the extension's; the toolkit container would rewrite /etc/containerd/config.toml, which Talos owns and which already registers the nvidia runtime with enable_cdi = true and cdi_spec_dirs = ['/run/cdi']. validator.driver.env points DRIVER_ROOT and NVIDIA_DRIVER_ROOT at / and disables /dev/char symlink creation. The driver validator otherwise looks for a driver container install, finds none, never writes its sentinel, and leaves the device plugin blocked behind toolkit-validation — the loop on "Attempting to validate a driver container installation". devicePlugin.env selects cdi-annotations and points NVIDIA_CTK_PATH at the host's nvidia-ctk. Tenant pods carry no runtimeClassName and did not need one on Ubuntu, where the toolkit made nvidia the default containerd runtime; Talos does not, so CDI is the only route for the CUDA libraries. DEVICE_ID_STRATEGY, NVIDIA_DRIVER_ROOT and CDI_ANNOTATION_PREFIX are set and tested alongside it: CDI device names key on the UUID strategy, the plugin resolves the driver from NVIDIA_DRIVER_ROOT, and the annotation prefix has to be the one containerd reads, so a regression in any of them breaks device resolution on its own. The previous default fed NVreg_NvLinkDisable=1 to the driver container through the nvidia-kernel-module-params ConfigMap. With that container off the ConfigMap is mounted by nothing, so the default is removed. On Talos a module parameter belongs to the worker machine config, which the kernelModules field of the KubernetesNodes app (packages/apps/kubernetes-nodes, cozystack#3571) writes. The wrapper chart keeps the ConfigMap for an operator who re-enables the driver container. Every value here is the configuration validated on a production Talos GPU node group, minus the parts that only mattered while the driver container was in play. Not covered: HAMi. When addons.hami.enabled the operator's device plugin stays off and HAMi serves the GPUs, and HAMi's own plugin needs equivalent CDI wiring on Talos that this default does not attempt. Refs: cozystack#3563 Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent a1b32f2 commit 669fb24

3 files changed

Lines changed: 149 additions & 38 deletions

File tree

‎packages/apps/kubernetes/templates/helmreleases/gpu-operator.yaml‎

Lines changed: 56 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,68 @@
11
{{- define "cozystack.defaultGpuOperatorValues" -}}
22
{{- /*
3-
GPU passthrough default: cozystack passes individual SXM GPUs (e.g.
4-
H200) into tenant worker VMs WITHOUT the NVSwitches. The nvidia driver
5-
then waits forever for an NVLink fabric that can never come up (no
6-
NVSwitch / fabricmanager), leaving Fabric State: In Progress — every
7-
CUDA call fails with "system not yet initialized" and the
8-
nvidia-operator-validator crashloops. Loading the kernel module with
9-
NVreg_NvLinkDisable=1 flips Fabric State to N/A so CUDA initializes.
10-
This is correct for per-GPU passthrough since there is no in-VM
11-
NVLink/NVSwitch peering anyway.
3+
Default for the Talos workers of tenant clusters, where the NVIDIA driver
4+
comes from a Talos system extension baked into the worker image. Nothing
5+
here loads the module or sets its parameters: that is the kernelModules
6+
field of the worker pool, in the KubernetesNodes app
7+
(packages/apps/kubernetes-nodes).
128

13-
The upstream chart only references an existing ConfigMap by name; the
14-
wrapper chart (packages/system/gpu-operator) renders that ConfigMap
15-
when kernelModuleConfig.create is true. Overridable via
16-
addons.gpuOperator.valuesOverride (mergeOverwrite below keeps user
17-
values winning).
9+
driver and toolkit are off. The driver container would load its own module
10+
and clash with the extension's, and the toolkit container rewrites
11+
/etc/containerd/config.toml, which Talos owns and which already registers
12+
the nvidia runtime with enable_cdi = true and cdi_spec_dirs = ['/run/cdi'].
13+
14+
The driver validator is pointed at the host root, where the extension puts
15+
the driver. Looking for a driver container install instead, it never writes
16+
its sentinel and the device plugin stays blocked behind toolkit-validation.
17+
The /dev/char symlinks belong to the extension, so it must not create them.
18+
19+
Tenant pods carry no runtimeClassName and Talos does not make nvidia the
20+
default containerd runtime, so CUDA libraries reach a pod only through CDI.
21+
The device plugin annotates pods for CDI injection and generates the spec
22+
into /run/cdi with the host's nvidia-ctk. CDI device names key on the uuid
23+
strategy, and the annotation prefix is pinned to the one containerd reads.
24+
Injection is decided at pod admission, so a pod created before its node
25+
works receives nothing and has to be recreated.
26+
27+
mergeOverwrite below lets addons.gpuOperator.valuesOverride win, but it
28+
replaces a list wholesale: an override of devicePlugin.env that adds one
29+
variable drops the defaults above, and CDI with them. Restate the full list.
1830
*/}}
19-
kernelModuleConfig:
20-
create: true
2131
gpu-operator:
2232
driver:
23-
kernelModuleConfig:
24-
name: nvidia-kernel-module-params
25-
{{- if .Values.addons.hami.enabled }}
33+
enabled: false
34+
toolkit:
35+
enabled: false
36+
validator:
37+
driver:
38+
env:
39+
- name: DRIVER_ROOT
40+
value: "/"
41+
- name: NVIDIA_DRIVER_ROOT
42+
value: "/"
43+
- name: DISABLE_DEV_CHAR_SYMLINK_CREATION
44+
value: "true"
2645
devicePlugin:
46+
{{- if .Values.addons.hami.enabled }}
47+
{{- /* HAMi ships its own device plugin and serves the GPUs itself; the
48+
operator's must stay off so the two do not both advertise
49+
nvidia.com/gpu. The env below is inert while this is false, and is
50+
emitted anyway so the values stay coherent if HAMi is turned off
51+
later. HAMi's own plugin needs its own CDI wiring on Talos, which
52+
this default does not attempt. */}}
2753
enabled: false
2854
{{- end }}
55+
env:
56+
- name: DEVICE_LIST_STRATEGY
57+
value: cdi-annotations
58+
- name: DEVICE_ID_STRATEGY
59+
value: uuid
60+
- name: NVIDIA_DRIVER_ROOT
61+
value: "/"
62+
- name: NVIDIA_CTK_PATH
63+
value: /usr/local/bin/nvidia-ctk
64+
- name: CDI_ANNOTATION_PREFIX
65+
value: cdi.k8s.io/
2966
{{- end }}
3067

3168
{{- if and .Values.addons.gpuOperator.enabled .Values._namespace.etcd }}

‎packages/apps/kubernetes/tests/gpu_operator_hami_test.yaml‎

Lines changed: 86 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,10 @@ tests:
1818
path: spec.values.gpu-operator.devicePlugin.enabled
1919
value: false
2020

21-
- it: should default the NvLinkDisable kernel module config even when hami is disabled
21+
# On Talos the driver comes from a system extension: the operator must not own
22+
# the driver or containerd, the validator must look at the host root, and the
23+
# device plugin must route CUDA libraries through CDI.
24+
- it: configures the operator for a driver provided by the OS
2225
set:
2326
addons:
2427
gpuOperator:
@@ -29,34 +32,102 @@ tests:
2932
valuesOverride: {}
3033
asserts:
3134
- equal:
32-
path: spec.values.kernelModuleConfig.create
33-
value: true
35+
path: spec.values.gpu-operator.driver.enabled
36+
value: false
3437
- equal:
35-
path: spec.values.gpu-operator.driver.kernelModuleConfig.name
36-
value: nvidia-kernel-module-params
38+
path: spec.values.gpu-operator.toolkit.enabled
39+
value: false
40+
# Without these the validator never writes its sentinel and the device
41+
# plugin stays blocked behind toolkit-validation.
42+
- contains:
43+
path: spec.values.gpu-operator.validator.driver.env
44+
content:
45+
name: DRIVER_ROOT
46+
value: "/"
47+
- contains:
48+
path: spec.values.gpu-operator.validator.driver.env
49+
content:
50+
name: DISABLE_DEV_CHAR_SYMLINK_CREATION
51+
value: "true"
52+
# Without the CDI settings the pod gets the device and no CUDA libraries.
53+
- contains:
54+
path: spec.values.gpu-operator.devicePlugin.env
55+
content:
56+
name: DEVICE_LIST_STRATEGY
57+
value: cdi-annotations
58+
- contains:
59+
path: spec.values.gpu-operator.devicePlugin.env
60+
content:
61+
name: NVIDIA_CTK_PATH
62+
value: /usr/local/bin/nvidia-ctk
63+
# CDI device names key on the UUID strategy, the plugin resolves the driver
64+
# from NVIDIA_DRIVER_ROOT, and the annotation prefix has to be the one
65+
# containerd reads.
66+
- contains:
67+
path: spec.values.gpu-operator.devicePlugin.env
68+
content:
69+
name: DEVICE_ID_STRATEGY
70+
value: uuid
71+
- contains:
72+
path: spec.values.gpu-operator.devicePlugin.env
73+
content:
74+
name: NVIDIA_DRIVER_ROOT
75+
value: "/"
76+
- contains:
77+
path: spec.values.gpu-operator.devicePlugin.env
78+
content:
79+
name: CDI_ANNOTATION_PREFIX
80+
value: cdi.k8s.io/
81+
# The driver-validation init container reads NVIDIA_DRIVER_ROOT alongside
82+
# DRIVER_ROOT.
83+
- contains:
84+
path: spec.values.gpu-operator.validator.driver.env
85+
content:
86+
name: NVIDIA_DRIVER_ROOT
87+
value: "/"
88+
89+
# With the driver container off nothing mounts this ConfigMap. Module
90+
# parameters belong to the worker pool's kernelModules in the KubernetesNodes
91+
# app (packages/apps/kubernetes-nodes), and an inert second place to set them
92+
# would only mislead.
93+
- it: does not render the driver-container kernel module ConfigMap
94+
set:
95+
addons:
96+
gpuOperator:
97+
enabled: true
98+
valuesOverride: {}
99+
hami:
100+
enabled: false
101+
valuesOverride: {}
102+
asserts:
103+
- notExists:
104+
path: spec.values.kernelModuleConfig
37105
- notExists:
38-
path: spec.values.gpu-operator.devicePlugin
106+
path: spec.values.gpu-operator.driver.kernelModuleConfig
39107

40-
- it: should let user override the kernel module config name
108+
# mergeOverwrite replaces a list rather than merging it, so an override adding
109+
# one variable to devicePlugin.env drops the CDI defaults with it and has to
110+
# restate the full list.
111+
- it: lets an operator replace the device plugin env wholesale
41112
set:
42113
addons:
43114
gpuOperator:
44115
enabled: true
45116
valuesOverride:
46117
gpu-operator:
47-
driver:
48-
kernelModuleConfig:
49-
name: my-custom-kmc
118+
devicePlugin:
119+
env:
120+
- name: DEVICE_LIST_STRATEGY
121+
value: envvar
50122
hami:
51123
enabled: false
52124
valuesOverride: {}
53125
asserts:
54126
- equal:
55-
path: spec.values.gpu-operator.driver.kernelModuleConfig.name
56-
value: my-custom-kmc
57-
- equal:
58-
path: spec.values.kernelModuleConfig.create
59-
value: true
127+
path: spec.values.gpu-operator.devicePlugin.env
128+
value:
129+
- name: DEVICE_LIST_STRATEGY
130+
value: envvar
60131

61132
- it: should apply hami defaults when valuesOverride key is omitted
62133
set:

‎packages/system/gpu-operator/values.yaml‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,10 +32,13 @@ gpu-operator:
3232
# true this wrapper chart renders the ConfigMap into the release
3333
# namespace so the params actually land. The driver entrypoint reads
3434
# ${base}/nvidia.conf and appends each line to the nvidia modprobe
35-
# options. Used by tenant Kubernetes clusters to set
36-
# NVreg_NvLinkDisable=1 (see packages/apps/kubernetes gpu-operator
37-
# HelmRelease defaults); off by default for host-level passthrough/vgpu
38-
# variants where the in-host driver is disabled.
35+
# options, so it only does anything when driver.enabled is true.
36+
#
37+
# Tenant Kubernetes clusters run this chart with the driver container off
38+
# and take module parameters from the worker pool's kernelModules in the
39+
# KubernetesNodes app (packages/apps/kubernetes-nodes). An operator who
40+
# re-enables the driver container through valuesOverride also has to set
41+
# create: true here.
3942
kernelModuleConfig:
4043
create: false
4144
name: nvidia-kernel-module-params

0 commit comments

Comments
 (0)