Repository navigation
Commit d105bbe
committed
feat(kubernetes): default gpu-operator to an OS-provided driver on Talos
Closes the gpu-operator half of #3563. Tenant clusters have run Talos
workers since 1.6.0, where the NVIDIA driver comes from a system extension
baked into the worker image rather than from the operator's driver
container. The addon default still described the Ubuntu shape, so a GPU
node group came up with the operator fighting the OS for the driver and no
path at all for CUDA libraries to reach a tenant pod.
Three things change, each fixing one silent failure:
driver.enabled and toolkit.enabled go false. The driver container would
load its own module and clash with the extension's; the toolkit container
would rewrite /etc/containerd/config.toml, which Talos owns and which
already registers the nvidia runtime with enable_cdi = true and
cdi_spec_dirs = ['/run/cdi'].
validator.driver.env points DRIVER_ROOT and NVIDIA_DRIVER_ROOT at / and
disables /dev/char symlink creation. The driver validator otherwise looks
for a driver container install, finds none, never writes its sentinel, and
leaves the device plugin blocked behind toolkit-validation — the loop on
"Attempting to validate a driver container installation".
devicePlugin.env selects cdi-annotations and points NVIDIA_CTK_PATH at the
host's nvidia-ctk. Tenant pods carry no runtimeClassName and did not need
one on Ubuntu, where the toolkit made nvidia the default containerd
runtime; Talos does not, so CDI is the only route for the CUDA libraries.
The previous default fed NVreg_NvLinkDisable=1 to the driver container
through the nvidia-kernel-module-params ConfigMap. With that container off
the ConfigMap is mounted by nothing, so the default is removed and the
parameter travels through the worker machine config instead. The wrapper
chart keeps the ConfigMap feature for an operator who re-enables the
driver container, with its comment corrected to say nothing uses it by
default any more.
Every value here is the configuration validated on a production Talos GPU
node group, minus the parts that only mattered while the driver container
was in play.
Not covered: HAMi. When addons.hami.enabled the operator's device plugin
stays off and HAMi serves the GPUs, and HAMi's own plugin needs equivalent
CDI wiring on Talos that this default does not attempt.
Refs: #3563
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <[email protected]>1 parent a1b32f2 commit d105bbe
3 files changed
Lines changed: 157 additions & 38 deletions
File tree
- packages
- apps/kubernetes
- templates/helmreleases
- tests
- system/gpu-operator
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
4 | | - | |
5 | | - | |
6 | | - | |
7 | | - | |
8 | | - | |
9 | | - | |
10 | | - | |
11 | | - | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
12 | 7 | | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | | - | |
17 | | - | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
18 | 53 | | |
19 | | - | |
20 | | - | |
21 | 54 | | |
22 | 55 | | |
23 | | - | |
24 | | - | |
25 | | - | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
26 | 68 | | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
27 | 76 | | |
28 | 77 | | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
29 | 89 | | |
30 | 90 | | |
31 | 91 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
22 | 29 | | |
23 | 30 | | |
24 | 31 | | |
| |||
29 | 36 | | |
30 | 37 | | |
31 | 38 | | |
32 | | - | |
33 | | - | |
| 39 | + | |
| 40 | + | |
34 | 41 | | |
35 | | - | |
36 | | - | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
37 | 86 | | |
38 | | - | |
| 87 | + | |
39 | 88 | | |
40 | | - | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
41 | 95 | | |
42 | 96 | | |
43 | 97 | | |
44 | 98 | | |
45 | 99 | | |
46 | 100 | | |
47 | | - | |
48 | | - | |
49 | | - | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
50 | 105 | | |
51 | 106 | | |
52 | 107 | | |
53 | 108 | | |
54 | 109 | | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
60 | 114 | | |
61 | 115 | | |
62 | 116 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
39 | 44 | | |
40 | 45 | | |
41 | 46 | | |
| |||
0 commit comments