Skip to content

Commit d105bbe

Browse files
feat(kubernetes): default gpu-operator to an OS-provided driver on Talos
Closes the gpu-operator half of #3563. Tenant clusters have run Talos workers since 1.6.0, where the NVIDIA driver comes from a system extension baked into the worker image rather than from the operator's driver container. The addon default still described the Ubuntu shape, so a GPU node group came up with the operator fighting the OS for the driver and no path at all for CUDA libraries to reach a tenant pod. Three things change, each fixing one silent failure: driver.enabled and toolkit.enabled go false. The driver container would load its own module and clash with the extension's; the toolkit container would rewrite /etc/containerd/config.toml, which Talos owns and which already registers the nvidia runtime with enable_cdi = true and cdi_spec_dirs = ['/run/cdi']. validator.driver.env points DRIVER_ROOT and NVIDIA_DRIVER_ROOT at / and disables /dev/char symlink creation. The driver validator otherwise looks for a driver container install, finds none, never writes its sentinel, and leaves the device plugin blocked behind toolkit-validation — the loop on "Attempting to validate a driver container installation". devicePlugin.env selects cdi-annotations and points NVIDIA_CTK_PATH at the host's nvidia-ctk. Tenant pods carry no runtimeClassName and did not need one on Ubuntu, where the toolkit made nvidia the default containerd runtime; Talos does not, so CDI is the only route for the CUDA libraries. The previous default fed NVreg_NvLinkDisable=1 to the driver container through the nvidia-kernel-module-params ConfigMap. With that container off the ConfigMap is mounted by nothing, so the default is removed and the parameter travels through the worker machine config instead. The wrapper chart keeps the ConfigMap feature for an operator who re-enables the driver container, with its comment corrected to say nothing uses it by default any more. Every value here is the configuration validated on a production Talos GPU node group, minus the parts that only mattered while the driver container was in play. Not covered: HAMi. When addons.hami.enabled the operator's device plugin stays off and HAMi serves the GPUs, and HAMi's own plugin needs equivalent CDI wiring on Talos that this default does not attempt. Refs: #3563 Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent a1b32f2 commit d105bbe

3 files changed

Lines changed: 157 additions & 38 deletions

File tree

‎packages/apps/kubernetes/templates/helmreleases/gpu-operator.yaml‎

Lines changed: 79 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,91 @@
11
{{- define "cozystack.defaultGpuOperatorValues" -}}
22
{{- /*
3-
GPU passthrough default: cozystack passes individual SXM GPUs (e.g.
4-
H200) into tenant worker VMs WITHOUT the NVSwitches. The nvidia driver
5-
then waits forever for an NVLink fabric that can never come up (no
6-
NVSwitch / fabricmanager), leaving Fabric State: In Progress — every
7-
CUDA call fails with "system not yet initialized" and the
8-
nvidia-operator-validator crashloops. Loading the kernel module with
9-
NVreg_NvLinkDisable=1 flips Fabric State to N/A so CUDA initializes.
10-
This is correct for per-GPU passthrough since there is no in-VM
11-
NVLink/NVSwitch peering anyway.
3+
OS-provided-driver default, for the Talos workers that tenant clusters have
4+
run since 1.6.0. The driver is supplied by a Talos system extension baked
5+
into the worker image, not by the operator's driver container, and that
6+
changes what every other component has to be told.
127

13-
The upstream chart only references an existing ConfigMap by name; the
14-
wrapper chart (packages/system/gpu-operator) renders that ConfigMap
15-
when kernelModuleConfig.create is true. Overridable via
16-
addons.gpuOperator.valuesOverride (mergeOverwrite below keeps user
17-
values winning).
8+
This replaces the previous default, which assumed the Ubuntu shape: the
9+
operator's driver container compiling and loading the module inside the
10+
worker VM, with NVreg_NvLinkDisable=1 fed to it through the
11+
nvidia-kernel-module-params ConfigMap. On Talos that container is off, so
12+
the ConfigMap was mounted by nothing and the parameter never applied. It now
13+
travels through the machine config instead, as
14+
nodeGroups.<name>.kernelModules, which is where a Talos node takes module
15+
parameters from.
16+
17+
driver.enabled: false — the module comes from the extension. The operator's
18+
driver container would try to load its own and clash with it.
19+
20+
toolkit.enabled: false — the toolkit container rewrites
21+
/etc/containerd/config.toml, which on Talos is machine-config-owned. Talos
22+
already registers the `nvidia` containerd runtime and sets
23+
enable_cdi = true with cdi_spec_dirs = ['/run/cdi'].
24+
25+
validator.driver.env — the driver validator looks for a driver container's
26+
install and finds none, so it never writes its sentinel and the device
27+
plugin stays blocked behind toolkit-validation. Pointing DRIVER_ROOT and
28+
NVIDIA_DRIVER_ROOT at / makes it inspect the host root, where the extension
29+
put the driver. DISABLE_DEV_CHAR_SYMLINK_CREATION stops it trying to create
30+
/dev/char symlinks that the extension already owns.
31+
32+
devicePlugin.env — tenant pods carry no runtimeClassName and did not need
33+
one on Ubuntu, where the toolkit made `nvidia` the default containerd
34+
runtime. Talos does not, so CUDA libraries reach a pod only through CDI:
35+
cdi-annotations makes the plugin annotate pods for CDI injection, and
36+
NVIDIA_CTK_PATH points it at the host's nvidia-ctk so it can generate the
37+
spec into the /run/cdi that containerd already reads. DEVICE_ID_STRATEGY
38+
uuid is what CDI device names key on. CDI_ANNOTATION_PREFIX matches the
39+
plugin's own default and is pinned so an upstream change cannot move it out
40+
from under containerd. NOTE: CDI injection is decided at pod admission, so
41+
pods created before a node is working never receive anything and have to be
42+
recreated.
43+
44+
Every value here is the configuration validated in production on a Talos
45+
GPU node group, minus the parts that only mattered while the driver
46+
container was in play.
47+
48+
Overridable via addons.gpuOperator.valuesOverride — mergeOverwrite below
49+
keeps operator values winning. Mind that mergeOverwrite replaces a list
50+
wholesale rather than merging it: an override that sets devicePlugin.env to
51+
add one variable drops the defaults above with it, and CDI stops working.
52+
Restate the full list when overriding it.
1853
*/}}
19-
kernelModuleConfig:
20-
create: true
2154
gpu-operator:
2255
driver:
23-
kernelModuleConfig:
24-
name: nvidia-kernel-module-params
25-
{{- if .Values.addons.hami.enabled }}
56+
enabled: false
57+
toolkit:
58+
enabled: false
59+
validator:
60+
driver:
61+
env:
62+
- name: DRIVER_ROOT
63+
value: "/"
64+
- name: NVIDIA_DRIVER_ROOT
65+
value: "/"
66+
- name: DISABLE_DEV_CHAR_SYMLINK_CREATION
67+
value: "true"
2668
devicePlugin:
69+
{{- if .Values.addons.hami.enabled }}
70+
{{- /* HAMi ships its own device plugin and serves the GPUs itself; the
71+
operator's must stay off so the two do not both advertise
72+
nvidia.com/gpu. The env below is inert while this is false, and is
73+
emitted anyway so the values stay coherent if HAMi is turned off
74+
later. HAMi's own plugin needs its own CDI wiring on Talos, which
75+
this default does not attempt. */}}
2776
enabled: false
2877
{{- end }}
78+
env:
79+
- name: DEVICE_LIST_STRATEGY
80+
value: cdi-annotations
81+
- name: DEVICE_ID_STRATEGY
82+
value: uuid
83+
- name: NVIDIA_DRIVER_ROOT
84+
value: "/"
85+
- name: NVIDIA_CTK_PATH
86+
value: /usr/local/bin/nvidia-ctk
87+
- name: CDI_ANNOTATION_PREFIX
88+
value: cdi.k8s.io/
2989
{{- end }}
3090

3191
{{- if and .Values.addons.gpuOperator.enabled .Values._namespace.etcd }}

‎packages/apps/kubernetes/tests/gpu_operator_hami_test.yaml‎

Lines changed: 69 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,14 @@ tests:
1818
path: spec.values.gpu-operator.devicePlugin.enabled
1919
value: false
2020

21-
- it: should default the NvLinkDisable kernel module config even when hami is disabled
21+
# The default targets the Talos workers tenant clusters have run since 1.6.0,
22+
# where the driver comes from a system extension rather than the operator's
23+
# driver container. These three asserts are the whole contract: the operator
24+
# must not try to own the driver or containerd, the driver validator must look
25+
# at the host root instead of a driver container that does not exist, and the
26+
# device plugin must route CUDA libraries through CDI because tenant pods carry
27+
# no runtimeClassName and Talos does not make `nvidia` the default runtime.
28+
- it: configures the operator for a driver provided by the OS
2229
set:
2330
addons:
2431
gpuOperator:
@@ -29,34 +36,81 @@ tests:
2936
valuesOverride: {}
3037
asserts:
3138
- equal:
32-
path: spec.values.kernelModuleConfig.create
33-
value: true
39+
path: spec.values.gpu-operator.driver.enabled
40+
value: false
3441
- equal:
35-
path: spec.values.gpu-operator.driver.kernelModuleConfig.name
36-
value: nvidia-kernel-module-params
42+
path: spec.values.gpu-operator.toolkit.enabled
43+
value: false
44+
# gap 3: without these the validator never writes its sentinel and the
45+
# device plugin stays blocked behind toolkit-validation.
46+
- contains:
47+
path: spec.values.gpu-operator.validator.driver.env
48+
content:
49+
name: DRIVER_ROOT
50+
value: "/"
51+
- contains:
52+
path: spec.values.gpu-operator.validator.driver.env
53+
content:
54+
name: DISABLE_DEV_CHAR_SYMLINK_CREATION
55+
value: "true"
56+
# gap 4: without these the pod gets the device and no CUDA libraries.
57+
- contains:
58+
path: spec.values.gpu-operator.devicePlugin.env
59+
content:
60+
name: DEVICE_LIST_STRATEGY
61+
value: cdi-annotations
62+
- contains:
63+
path: spec.values.gpu-operator.devicePlugin.env
64+
content:
65+
name: NVIDIA_CTK_PATH
66+
value: /usr/local/bin/nvidia-ctk
67+
68+
# The previous default fed NVreg_NvLinkDisable=1 to the operator's driver
69+
# container through the nvidia-kernel-module-params ConfigMap. With the driver
70+
# container off that ConfigMap is mounted by nothing, so the parameter now
71+
# travels through the machine config as nodeGroups.<name>.kernelModules. Pin
72+
# its absence so the dead default cannot come back and offer a second, inert
73+
# source for the same setting.
74+
- it: no longer renders the driver-container kernel module ConfigMap
75+
set:
76+
addons:
77+
gpuOperator:
78+
enabled: true
79+
valuesOverride: {}
80+
hami:
81+
enabled: false
82+
valuesOverride: {}
83+
asserts:
84+
- notExists:
85+
path: spec.values.kernelModuleConfig
3786
- notExists:
38-
path: spec.values.gpu-operator.devicePlugin
87+
path: spec.values.gpu-operator.driver.kernelModuleConfig
3988

40-
- it: should let user override the kernel module config name
89+
# mergeOverwrite replaces a list rather than merging it, so an operator adding
90+
# one variable to devicePlugin.env silently drops the CDI defaults with it.
91+
# That is Helm's list semantics and cannot be fixed here, so pin it: the
92+
# behaviour is documented in the template and a reader who finds this test
93+
# knows the override has to restate the full list.
94+
- it: lets an operator replace the device plugin env wholesale
4195
set:
4296
addons:
4397
gpuOperator:
4498
enabled: true
4599
valuesOverride:
46100
gpu-operator:
47-
driver:
48-
kernelModuleConfig:
49-
name: my-custom-kmc
101+
devicePlugin:
102+
env:
103+
- name: DEVICE_LIST_STRATEGY
104+
value: envvar
50105
hami:
51106
enabled: false
52107
valuesOverride: {}
53108
asserts:
54109
- equal:
55-
path: spec.values.gpu-operator.driver.kernelModuleConfig.name
56-
value: my-custom-kmc
57-
- equal:
58-
path: spec.values.kernelModuleConfig.create
59-
value: true
110+
path: spec.values.gpu-operator.devicePlugin.env
111+
value:
112+
- name: DEVICE_LIST_STRATEGY
113+
value: envvar
60114

61115
- it: should apply hami defaults when valuesOverride key is omitted
62116
set:

‎packages/system/gpu-operator/values.yaml‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,10 +32,15 @@ gpu-operator:
3232
# true this wrapper chart renders the ConfigMap into the release
3333
# namespace so the params actually land. The driver entrypoint reads
3434
# ${base}/nvidia.conf and appends each line to the nvidia modprobe
35-
# options. Used by tenant Kubernetes clusters to set
36-
# NVreg_NvLinkDisable=1 (see packages/apps/kubernetes gpu-operator
37-
# HelmRelease defaults); off by default for host-level passthrough/vgpu
38-
# variants where the in-host driver is disabled.
35+
# options, so it only does anything when driver.enabled is true.
36+
#
37+
# No variant enables it any more. Tenant Kubernetes clusters used to, to set
38+
# NVreg_NvLinkDisable=1, but their workers run Talos since 1.6.0: the driver
39+
# comes from a system extension, the operator's driver container is off, and
40+
# that parameter now travels through the worker machine config instead
41+
# (nodeGroups.<name>.kernelModules in packages/apps/kubernetes). Kept for an
42+
# operator who re-enables the driver container through valuesOverride, which
43+
# then also has to set create: true here.
3944
kernelModuleConfig:
4045
create: false
4146
name: nvidia-kernel-module-params

0 commit comments

Comments
 (0)