Skip to content

Commit 924d28c

Browse files
authored
feat(kubernetes): default gpu-operator to an OS-provided driver on Talos (#3585)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Closes the gpu-operator half of #3563 — gap 3 (the operator cannot validate a driver it did not install) and gap 4 (nothing routes CUDA libraries to a tenant pod). Tenant clusters have run Talos workers since 1.6.0, where the NVIDIA driver comes from a system extension baked into the worker image rather than from the operator's driver container. The addon default still described the Ubuntu shape, so a GPU node group came up with the operator fighting the OS for the driver and no path at all for the CUDA libraries. Three changes, each closing one silent failure: | Change | Without it | |---|---| | `driver.enabled: false`, `toolkit.enabled: false` | the driver container loads its own module and clashes with the extension's; the toolkit container rewrites `/etc/containerd/config.toml`, which Talos owns and which already registers the `nvidia` runtime with `enable_cdi = true` and `cdi_spec_dirs = ['/run/cdi']` | | `validator.driver.env`: `DRIVER_ROOT=/`, `NVIDIA_DRIVER_ROOT=/`, `DISABLE_DEV_CHAR_SYMLINK_CREATION=true` | the driver validator looks for a driver-container install, finds none, never writes its sentinel, and the device plugin stays blocked behind `toolkit-validation` — the loop on `Attempting to validate a driver container installation` (**gap 3**) | | `devicePlugin.env`: `DEVICE_LIST_STRATEGY=cdi-annotations`, `NVIDIA_CTK_PATH=/usr/local/bin/nvidia-ctk`, plus `DEVICE_ID_STRATEGY`, `NVIDIA_DRIVER_ROOT`, `CDI_ANNOTATION_PREFIX` | tenant pods carry no `runtimeClassName` and did not need one on Ubuntu, where the toolkit made `nvidia` the default containerd runtime. Talos does not, so CDI is the only route: the pod gets the device and no CUDA libraries (**gap 4**) | Every value is the configuration validated on a production Talos GPU node group, minus the parts that only mattered while the driver container was in play. No DaemonSet, no new image, no new package — the fix is values. ### The removed default The previous default fed `NVreg_NvLinkDisable=1` to the driver container through the `nvidia-kernel-module-params` ConfigMap. With that container off the ConfigMap is mounted by nothing, so this removes the default. On Talos the parameter belongs to the worker machine config, which the `kernelModules` field of the worker pool writes: that field is top-level in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), added by #3571, and its automatic set for a pool holding an `nvidia.com/*` GPU already loads `nvidia` with `NVreg_NvLinkDisable=1`. The wrapper chart keeps the ConfigMap feature for an operator who re-enables the driver container through `valuesOverride`. A test pins the absence so the dead default cannot come back and offer a second, inert source for the same setting. ### Relationship to #3571 Independent files, no conflict, but only useful together: this makes `driver.enabled: false` correct, and #3571 is what actually loads the modules the extension ships. Merged alone, this changes a GPU node group from "operator fights the OS" to "operator stays out of the way and the modules still are not loaded" — better, but not yet working. #3571 alone leaves the operator fighting for the driver. Either order is safe; both are needed for a GPU node group to work. ### What is verified, and what is not Verified here: the rendered HelmRelease values, that `valuesOverride` still wins over each new key, and that the removed default is gone. The individual values are the ones running in production today, applied by hand through `addons.gpuOperator.valuesOverride`. Not verified here: this exact default exercised end-to-end as the chart's own output. That needs a Talos GPU node group whose modules load, which needs #3571 released. I would rather say so than imply a green e2e that does not exist — happy to hold this as draft until I have run a single-node L40S cluster against the chart default, and I will report the result on this PR either way. ### Not covered HAMi. When `addons.hami.enabled` the operator's device plugin stays off and HAMi serves the GPUs; HAMi's own plugin needs equivalent CDI wiring on Talos, which this default does not attempt. Flagging rather than guessing — the production cluster this was derived from runs HAMi disabled, so I have no validated configuration for that path. Worth its own issue if a maintainer confirms the shape. Also not in scope, and unchanged from the merge base: `talos.installerRepository` and `talos.version` reach the worker machine config unvalidated (being tracked separately by @lexfrei), and the `nvidia-operator-validator` chain for the *management* cluster's own passthrough/vgpu variants, which never load the host driver at all. ### Screenshots No UI changes. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#643 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(kubernetes): The GPU Operator addon of tenant Kubernetes clusters now defaults to a driver provided by the operating system, which is the shape of the Talos workers these clusters have run since 1.6.0. `driver` and `toolkit` are disabled (the driver comes from a Talos system extension, and containerd configuration belongs to the machine config), the driver validator is pointed at the host root so it stops looping on "Attempting to validate a driver container installation", and the device plugin is switched to CDI (`DEVICE_LIST_STRATEGY=cdi-annotations`) so tenant pods receive the CUDA libraries — previously they received the GPU device and nothing else, since Talos does not make `nvidia` the default containerd runtime. The previous default, which fed `NVreg_NvLinkDisable=1` to the operator's driver container through a ConfigMap, is removed because that container is disabled; on Talos the parameter is carried by the worker machine config, through the `kernelModules` field of the worker pool in the KubernetesNodes app (`packages/apps/kubernetes-nodes`), which loads `nvidia` with `NVreg_NvLinkDisable=1` by default on a pool holding an `nvidia.com/*` GPU. Operators who overrode `addons.gpuOperator.valuesOverride` to work around this by hand can drop those overrides. NOTE: an override that sets `devicePlugin.env` replaces the list rather than merging it, so restate the full list when overriding it or CDI stops working. HAMi is not covered: when `addons.hami.enabled` the operator's device plugin stays off and HAMi's own plugin needs equivalent CDI wiring, which this change does not attempt. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Configuration** * GPU workloads on Talos workers now use host-provided NVIDIA drivers and tooling by default; the GPU Operator’s driver and toolkit are disabled. * Device discovery uses CDI annotations and UUID-based device IDs. When HAMi is enabled, the Operator’s device plugin is disabled. * Host-root validation no longer creates `/dev/char` symlinks. * Driver kernel-module configuration is no longer created by default. * **Documentation** * Clarified when kernel-module configuration applies and what settings are needed to re-enable the driver. * Noted that `valuesOverride` replaces lists wholesale. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents d466fd9 + 669fb24 commit 924d28c

3 files changed

Lines changed: 149 additions & 38 deletions

File tree

‎packages/apps/kubernetes/templates/helmreleases/gpu-operator.yaml‎

Lines changed: 56 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,68 @@
11
{{- define "cozystack.defaultGpuOperatorValues" -}}
22
{{- /*
3-
GPU passthrough default: cozystack passes individual SXM GPUs (e.g.
4-
H200) into tenant worker VMs WITHOUT the NVSwitches. The nvidia driver
5-
then waits forever for an NVLink fabric that can never come up (no
6-
NVSwitch / fabricmanager), leaving Fabric State: In Progress — every
7-
CUDA call fails with "system not yet initialized" and the
8-
nvidia-operator-validator crashloops. Loading the kernel module with
9-
NVreg_NvLinkDisable=1 flips Fabric State to N/A so CUDA initializes.
10-
This is correct for per-GPU passthrough since there is no in-VM
11-
NVLink/NVSwitch peering anyway.
3+
Default for the Talos workers of tenant clusters, where the NVIDIA driver
4+
comes from a Talos system extension baked into the worker image. Nothing
5+
here loads the module or sets its parameters: that is the kernelModules
6+
field of the worker pool, in the KubernetesNodes app
7+
(packages/apps/kubernetes-nodes).
128

13-
The upstream chart only references an existing ConfigMap by name; the
14-
wrapper chart (packages/system/gpu-operator) renders that ConfigMap
15-
when kernelModuleConfig.create is true. Overridable via
16-
addons.gpuOperator.valuesOverride (mergeOverwrite below keeps user
17-
values winning).
9+
driver and toolkit are off. The driver container would load its own module
10+
and clash with the extension's, and the toolkit container rewrites
11+
/etc/containerd/config.toml, which Talos owns and which already registers
12+
the nvidia runtime with enable_cdi = true and cdi_spec_dirs = ['/run/cdi'].
13+
14+
The driver validator is pointed at the host root, where the extension puts
15+
the driver. Looking for a driver container install instead, it never writes
16+
its sentinel and the device plugin stays blocked behind toolkit-validation.
17+
The /dev/char symlinks belong to the extension, so it must not create them.
18+
19+
Tenant pods carry no runtimeClassName and Talos does not make nvidia the
20+
default containerd runtime, so CUDA libraries reach a pod only through CDI.
21+
The device plugin annotates pods for CDI injection and generates the spec
22+
into /run/cdi with the host's nvidia-ctk. CDI device names key on the uuid
23+
strategy, and the annotation prefix is pinned to the one containerd reads.
24+
Injection is decided at pod admission, so a pod created before its node
25+
works receives nothing and has to be recreated.
26+
27+
mergeOverwrite below lets addons.gpuOperator.valuesOverride win, but it
28+
replaces a list wholesale: an override of devicePlugin.env that adds one
29+
variable drops the defaults above, and CDI with them. Restate the full list.
1830
*/}}
19-
kernelModuleConfig:
20-
create: true
2131
gpu-operator:
2232
driver:
23-
kernelModuleConfig:
24-
name: nvidia-kernel-module-params
25-
{{- if .Values.addons.hami.enabled }}
33+
enabled: false
34+
toolkit:
35+
enabled: false
36+
validator:
37+
driver:
38+
env:
39+
- name: DRIVER_ROOT
40+
value: "/"
41+
- name: NVIDIA_DRIVER_ROOT
42+
value: "/"
43+
- name: DISABLE_DEV_CHAR_SYMLINK_CREATION
44+
value: "true"
2645
devicePlugin:
46+
{{- if .Values.addons.hami.enabled }}
47+
{{- /* HAMi ships its own device plugin and serves the GPUs itself; the
48+
operator's must stay off so the two do not both advertise
49+
nvidia.com/gpu. The env below is inert while this is false, and is
50+
emitted anyway so the values stay coherent if HAMi is turned off
51+
later. HAMi's own plugin needs its own CDI wiring on Talos, which
52+
this default does not attempt. */}}
2753
enabled: false
2854
{{- end }}
55+
env:
56+
- name: DEVICE_LIST_STRATEGY
57+
value: cdi-annotations
58+
- name: DEVICE_ID_STRATEGY
59+
value: uuid
60+
- name: NVIDIA_DRIVER_ROOT
61+
value: "/"
62+
- name: NVIDIA_CTK_PATH
63+
value: /usr/local/bin/nvidia-ctk
64+
- name: CDI_ANNOTATION_PREFIX
65+
value: cdi.k8s.io/
2966
{{- end }}
3067

3168
{{- if and .Values.addons.gpuOperator.enabled .Values._namespace.etcd }}

‎packages/apps/kubernetes/tests/gpu_operator_hami_test.yaml‎

Lines changed: 86 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,10 @@ tests:
1818
path: spec.values.gpu-operator.devicePlugin.enabled
1919
value: false
2020

21-
- it: should default the NvLinkDisable kernel module config even when hami is disabled
21+
# On Talos the driver comes from a system extension: the operator must not own
22+
# the driver or containerd, the validator must look at the host root, and the
23+
# device plugin must route CUDA libraries through CDI.
24+
- it: configures the operator for a driver provided by the OS
2225
set:
2326
addons:
2427
gpuOperator:
@@ -29,34 +32,102 @@ tests:
2932
valuesOverride: {}
3033
asserts:
3134
- equal:
32-
path: spec.values.kernelModuleConfig.create
33-
value: true
35+
path: spec.values.gpu-operator.driver.enabled
36+
value: false
3437
- equal:
35-
path: spec.values.gpu-operator.driver.kernelModuleConfig.name
36-
value: nvidia-kernel-module-params
38+
path: spec.values.gpu-operator.toolkit.enabled
39+
value: false
40+
# Without these the validator never writes its sentinel and the device
41+
# plugin stays blocked behind toolkit-validation.
42+
- contains:
43+
path: spec.values.gpu-operator.validator.driver.env
44+
content:
45+
name: DRIVER_ROOT
46+
value: "/"
47+
- contains:
48+
path: spec.values.gpu-operator.validator.driver.env
49+
content:
50+
name: DISABLE_DEV_CHAR_SYMLINK_CREATION
51+
value: "true"
52+
# Without the CDI settings the pod gets the device and no CUDA libraries.
53+
- contains:
54+
path: spec.values.gpu-operator.devicePlugin.env
55+
content:
56+
name: DEVICE_LIST_STRATEGY
57+
value: cdi-annotations
58+
- contains:
59+
path: spec.values.gpu-operator.devicePlugin.env
60+
content:
61+
name: NVIDIA_CTK_PATH
62+
value: /usr/local/bin/nvidia-ctk
63+
# CDI device names key on the UUID strategy, the plugin resolves the driver
64+
# from NVIDIA_DRIVER_ROOT, and the annotation prefix has to be the one
65+
# containerd reads.
66+
- contains:
67+
path: spec.values.gpu-operator.devicePlugin.env
68+
content:
69+
name: DEVICE_ID_STRATEGY
70+
value: uuid
71+
- contains:
72+
path: spec.values.gpu-operator.devicePlugin.env
73+
content:
74+
name: NVIDIA_DRIVER_ROOT
75+
value: "/"
76+
- contains:
77+
path: spec.values.gpu-operator.devicePlugin.env
78+
content:
79+
name: CDI_ANNOTATION_PREFIX
80+
value: cdi.k8s.io/
81+
# The driver-validation init container reads NVIDIA_DRIVER_ROOT alongside
82+
# DRIVER_ROOT.
83+
- contains:
84+
path: spec.values.gpu-operator.validator.driver.env
85+
content:
86+
name: NVIDIA_DRIVER_ROOT
87+
value: "/"
88+
89+
# With the driver container off nothing mounts this ConfigMap. Module
90+
# parameters belong to the worker pool's kernelModules in the KubernetesNodes
91+
# app (packages/apps/kubernetes-nodes), and an inert second place to set them
92+
# would only mislead.
93+
- it: does not render the driver-container kernel module ConfigMap
94+
set:
95+
addons:
96+
gpuOperator:
97+
enabled: true
98+
valuesOverride: {}
99+
hami:
100+
enabled: false
101+
valuesOverride: {}
102+
asserts:
103+
- notExists:
104+
path: spec.values.kernelModuleConfig
37105
- notExists:
38-
path: spec.values.gpu-operator.devicePlugin
106+
path: spec.values.gpu-operator.driver.kernelModuleConfig
39107

40-
- it: should let user override the kernel module config name
108+
# mergeOverwrite replaces a list rather than merging it, so an override adding
109+
# one variable to devicePlugin.env drops the CDI defaults with it and has to
110+
# restate the full list.
111+
- it: lets an operator replace the device plugin env wholesale
41112
set:
42113
addons:
43114
gpuOperator:
44115
enabled: true
45116
valuesOverride:
46117
gpu-operator:
47-
driver:
48-
kernelModuleConfig:
49-
name: my-custom-kmc
118+
devicePlugin:
119+
env:
120+
- name: DEVICE_LIST_STRATEGY
121+
value: envvar
50122
hami:
51123
enabled: false
52124
valuesOverride: {}
53125
asserts:
54126
- equal:
55-
path: spec.values.gpu-operator.driver.kernelModuleConfig.name
56-
value: my-custom-kmc
57-
- equal:
58-
path: spec.values.kernelModuleConfig.create
59-
value: true
127+
path: spec.values.gpu-operator.devicePlugin.env
128+
value:
129+
- name: DEVICE_LIST_STRATEGY
130+
value: envvar
60131

61132
- it: should apply hami defaults when valuesOverride key is omitted
62133
set:

‎packages/system/gpu-operator/values.yaml‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,10 +32,13 @@ gpu-operator:
3232
# true this wrapper chart renders the ConfigMap into the release
3333
# namespace so the params actually land. The driver entrypoint reads
3434
# ${base}/nvidia.conf and appends each line to the nvidia modprobe
35-
# options. Used by tenant Kubernetes clusters to set
36-
# NVreg_NvLinkDisable=1 (see packages/apps/kubernetes gpu-operator
37-
# HelmRelease defaults); off by default for host-level passthrough/vgpu
38-
# variants where the in-host driver is disabled.
35+
# options, so it only does anything when driver.enabled is true.
36+
#
37+
# Tenant Kubernetes clusters run this chart with the driver container off
38+
# and take module parameters from the worker pool's kernelModules in the
39+
# KubernetesNodes app (packages/apps/kubernetes-nodes). An operator who
40+
# re-enables the driver container through valuesOverride also has to set
41+
# create: true here.
3942
kernelModuleConfig:
4043
create: false
4144
name: nvidia-kernel-module-params

0 commit comments

Comments
 (0)