Skip to content

Commit f0f1431

Browse files
authored
feat(kubernetes): enable GPU passthrough out-of-the-box (NvLinkDisable + gpu=on) (#2780)
## What this PR does Makes GPU passthrough work out-of-the-box for tenant Kubernetes clusters (the `kubernetes` app), so operators no longer have to hand-fix it. Two defaults: 1. **Tenant gpu-operator loads the nvidia driver with `NVreg_NvLinkDisable=1`.** 2. **Nodes in GPU node-groups are labeled `gpu=on` automatically.** ### Root cause (validated empirically on a live cluster) cozystack passes through individual SXM GPUs (e.g. H200) into tenant worker VMs **without** the NVSwitches. Such a GPU's driver then waits for an NVLink fabric that can never come up — there is no NVSwitch and fabricmanager has nothing to manage — so `nvidia-smi` reports `Fabric State: In Progress` indefinitely. As a result every CUDA call fails with `system not yet initialized`, `nvidia-operator-validator` (cuda-validation) crashloops, and no GPUs are advertised. Loading the kernel module with `NVreg_NvLinkDisable=1` flips `Fabric State` to `N/A`, CUDA initializes, and the validators pass. This is the correct setting for per-GPU passthrough because there is no in-VM NVLink/NVSwitch peering — the GPU is alone in the VM, so there is nothing to disable that the workload needs. The second issue: HAMi's `hami-device-plugin` DaemonSet has `nodeSelector: gpu=on` and stays at `DESIRED=0` because GPU worker nodes lack that label, so `nvidia.com/gpu` is never advertised. Labeling the GPU nodes `gpu=on` lets the device plugin schedule and advertise the GPUs. ### How it works - **`gpu=on` label:** node-groups in `packages/apps/kubernetes/templates/cluster.yaml` that declare `gpus` get `--node-labels=gpu=on` appended to their kubelet args (in the `KubeadmConfigTemplate` `joinConfiguration`). Non-GPU node-groups are untouched. - **`NVreg_NvLinkDisable=1`:** the upstream gpu-operator chart only references an existing ConfigMap by name (`driver.kernelModuleConfig.name`) and never creates it. The gpu-operator wrapper chart (`packages/system/gpu-operator`) now renders that ConfigMap (opt-in via `kernelModuleConfig.create`) so it deploys into the tenant `cozy-gpu-operator` namespace alongside the operator. The tenant gpu-operator HelmRelease defaults enable it and point `driver.kernelModuleConfig.name` at it. Both defaults remain overridable: a user setting `addons.gpuOperator.valuesOverride` still wins via the existing `mergeOverwrite(defaults, overrides)` in the HelmRelease template. ### Screenshots Not applicable — no UI changes. ### Release note ```release-note feat(kubernetes): enable GPU passthrough out-of-the-box for tenant clusters. GPU node-groups are now labeled `gpu=on` so HAMi's device plugin schedules and advertises `nvidia.com/gpu`, and the tenant gpu-operator loads the nvidia driver with `NVreg_NvLinkDisable=1` so single-SXM-GPU passthrough (no NVSwitch in the VM) no longer hangs on `Fabric State: In Progress` / `system not yet initialized`. Both defaults are overridable via `addons.gpuOperator.valuesOverride`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * GPU-enabled node groups now get the expected GPU label during node registration. * GPU operator supports optional kernel-module parameter configuration with a default parameter set. * **Tests** * Added tests verifying GPU node labeling is applied only for GPU-enabled node groups. * Added tests validating kernel-module config creation, naming, and content behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 86c3de2 + a60c2e6 commit f0f1431

7 files changed

Lines changed: 227 additions & 3 deletions

File tree

‎packages/apps/kubernetes/templates/cluster.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -475,6 +475,15 @@ spec:
475475
joinConfiguration:
476476
nodeRegistration:
477477
kubeletExtraArgs:
478+
{{- /*
479+
GPU node-groups: label every node `gpu=on` so HAMi's
480+
hami-device-plugin DaemonSet (nodeSelector gpu=on) schedules
481+
and advertises nvidia.com/gpu. Without the label the plugin
482+
stays at DESIRED=0 and no GPUs are exposed to the tenant.
483+
*/}}
484+
{{- if $group.gpus }}
485+
node-labels: "gpu=on"
486+
{{- end }}
478487
system-reserved: "memory={{ $systemReservedMemory }},cpu={{ $systemReservedCpu }}"
479488
kube-reserved: "memory={{ $kubeReservedMemory }},cpu={{ $kubeReservedCpu }}"
480489
eviction-hard: "memory.available<{{ $evictionHardMemory }},nodefs.available<10%,imagefs.available<15%,nodefs.inodesFree<5%"

‎packages/apps/kubernetes/templates/helmreleases/gpu-operator.yaml‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,28 @@
11
{{- define "cozystack.defaultGpuOperatorValues" -}}
2-
{{- if .Values.addons.hami.enabled }}
2+
{{- /*
3+
GPU passthrough default: cozystack passes individual SXM GPUs (e.g.
4+
H200) into tenant worker VMs WITHOUT the NVSwitches. The nvidia driver
5+
then waits forever for an NVLink fabric that can never come up (no
6+
NVSwitch / fabricmanager), leaving Fabric State: In Progress — every
7+
CUDA call fails with "system not yet initialized" and the
8+
nvidia-operator-validator crashloops. Loading the kernel module with
9+
NVreg_NvLinkDisable=1 flips Fabric State to N/A so CUDA initializes.
10+
This is correct for per-GPU passthrough since there is no in-VM
11+
NVLink/NVSwitch peering anyway.
12+
13+
The upstream chart only references an existing ConfigMap by name; the
14+
wrapper chart (packages/system/gpu-operator) renders that ConfigMap
15+
when kernelModuleConfig.create is true. Overridable via
16+
addons.gpuOperator.valuesOverride (mergeOverwrite below keeps user
17+
values winning).
18+
*/}}
19+
kernelModuleConfig:
20+
create: true
321
gpu-operator:
22+
driver:
23+
kernelModuleConfig:
24+
name: nvidia-kernel-module-params
25+
{{- if .Values.addons.hami.enabled }}
426
devicePlugin:
527
enabled: false
628
{{- end }}
Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
suite: GPU node-group kubelet node-labels tests
2+
3+
templates:
4+
- templates/cluster.yaml
5+
6+
# The KubeadmConfigTemplate for the first nodeGroup is rendered at
7+
# document index 4 within cluster.yaml. The kubelet args live at:
8+
# spec.template.spec.joinConfiguration.nodeRegistration.kubeletExtraArgs
9+
# GPU node-groups must register their nodes with gpu=on so HAMi's
10+
# hami-device-plugin DaemonSet (nodeSelector gpu=on) schedules.
11+
12+
tests:
13+
- it: labels GPU node-groups with gpu=on
14+
release:
15+
name: test-k8s
16+
namespace: tenant-test
17+
set:
18+
_namespace:
19+
etcd: etcd
20+
ingress: nginx
21+
host: example.com
22+
_cluster:
23+
cluster-domain: cozy.local
24+
nodeGroups:
25+
md0:
26+
minReplicas: 0
27+
maxReplicas: 10
28+
instanceType: ""
29+
diskSize: 20Gi
30+
roles:
31+
- worker
32+
resources:
33+
cpu: 8
34+
memory: 16Gi
35+
gpus:
36+
- name: nvidia.com/GH200
37+
kubelet: {}
38+
asserts:
39+
- isKind:
40+
of: KubeadmConfigTemplate
41+
documentIndex: 4
42+
- equal:
43+
path: spec.template.spec.joinConfiguration.nodeRegistration.kubeletExtraArgs.node-labels
44+
value: "gpu=on"
45+
documentIndex: 4
46+
47+
- it: does not label non-GPU node-groups with gpu=on
48+
release:
49+
name: test-k8s
50+
namespace: tenant-test
51+
set:
52+
_namespace:
53+
etcd: etcd
54+
ingress: nginx
55+
host: example.com
56+
_cluster:
57+
cluster-domain: cozy.local
58+
nodeGroups:
59+
md0:
60+
minReplicas: 0
61+
maxReplicas: 10
62+
instanceType: ""
63+
diskSize: 20Gi
64+
roles:
65+
- worker
66+
resources:
67+
cpu: 2
68+
memory: 4Gi
69+
gpus: []
70+
kubelet: {}
71+
asserts:
72+
- isKind:
73+
of: KubeadmConfigTemplate
74+
documentIndex: 4
75+
- notExists:
76+
path: spec.template.spec.joinConfiguration.nodeRegistration.kubeletExtraArgs.node-labels
77+
documentIndex: 4

‎packages/apps/kubernetes/tests/gpu_operator_hami_test.yaml‎

Lines changed: 29 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ tests:
1818
path: spec.values.gpu-operator.devicePlugin.enabled
1919
value: false
2020

21-
- it: should not have values when hami is disabled and no overrides
21+
- it: should default the NvLinkDisable kernel module config even when hami is disabled
2222
set:
2323
addons:
2424
gpuOperator:
@@ -28,8 +28,35 @@ tests:
2828
enabled: false
2929
valuesOverride: {}
3030
asserts:
31+
- equal:
32+
path: spec.values.kernelModuleConfig.create
33+
value: true
34+
- equal:
35+
path: spec.values.gpu-operator.driver.kernelModuleConfig.name
36+
value: nvidia-kernel-module-params
3137
- notExists:
32-
path: spec.values
38+
path: spec.values.gpu-operator.devicePlugin
39+
40+
- it: should let user override the kernel module config name
41+
set:
42+
addons:
43+
gpuOperator:
44+
enabled: true
45+
valuesOverride:
46+
gpu-operator:
47+
driver:
48+
kernelModuleConfig:
49+
name: my-custom-kmc
50+
hami:
51+
enabled: false
52+
valuesOverride: {}
53+
asserts:
54+
- equal:
55+
path: spec.values.gpu-operator.driver.kernelModuleConfig.name
56+
value: my-custom-kmc
57+
- equal:
58+
path: spec.values.kernelModuleConfig.create
59+
value: true
3360

3461
- it: should apply hami defaults when valuesOverride key is omitted
3562
set:
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{{- if .Values.kernelModuleConfig.create }}
2+
{{- /*
3+
Name the ConfigMap after the driver's effective kernelModuleConfig
4+
reference (gpu-operator.driver.kernelModuleConfig.name) — that is the
5+
name the operator mounts into the driver pod. Falling back to the
6+
wrapper's own kernelModuleConfig.name keeps the two from ever diverging:
7+
overriding only the driver reference would otherwise leave the driver
8+
pointing at a ConfigMap that is never rendered.
9+
*/}}
10+
{{- $gpuOperator := index .Values "gpu-operator" | default dict -}}
11+
{{- $driver := index $gpuOperator "driver" | default dict -}}
12+
{{- $driverKmc := index $driver "kernelModuleConfig" | default dict -}}
13+
{{- $name := $driverKmc.name | default .Values.kernelModuleConfig.name -}}
14+
apiVersion: v1
15+
kind: ConfigMap
16+
metadata:
17+
name: {{ $name }}
18+
namespace: {{ .Release.Namespace }}
19+
labels:
20+
app.kubernetes.io/managed-by: {{ .Release.Service }}
21+
app.kubernetes.io/name: gpu-operator
22+
data:
23+
nvidia.conf: |
24+
{{- .Values.kernelModuleConfig.content | nindent 4 }}
25+
{{- end }}
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
suite: GPU operator kernel module ConfigMap tests
2+
3+
templates:
4+
- templates/kernel-module-config.yaml
5+
6+
tests:
7+
- it: does not render the ConfigMap when create is false (default)
8+
asserts:
9+
- hasDocuments:
10+
count: 0
11+
12+
- it: renders the ConfigMap into the release namespace with the nvidia.conf key when create is true
13+
release:
14+
name: gpu-operator
15+
namespace: cozy-gpu-operator
16+
set:
17+
kernelModuleConfig:
18+
create: true
19+
asserts:
20+
- hasDocuments:
21+
count: 1
22+
- isKind:
23+
of: ConfigMap
24+
- equal:
25+
path: metadata.name
26+
value: nvidia-kernel-module-params
27+
- equal:
28+
path: metadata.namespace
29+
value: cozy-gpu-operator
30+
- matchRegex:
31+
path: data["nvidia.conf"]
32+
pattern: NVreg_NvLinkDisable=1
33+
34+
- it: names the ConfigMap after the driver kernelModuleConfig reference so the two cannot diverge
35+
release:
36+
name: gpu-operator
37+
namespace: cozy-gpu-operator
38+
set:
39+
kernelModuleConfig:
40+
create: true
41+
gpu-operator:
42+
driver:
43+
kernelModuleConfig:
44+
name: my-custom-kmc
45+
asserts:
46+
- equal:
47+
path: metadata.name
48+
value: my-custom-kmc

‎packages/system/gpu-operator/values.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,19 @@ gpu-operator:
2525
# values-vgpu.yaml.
2626
vgpuDeviceManager:
2727
enabled: false
28+
29+
# Optional ConfigMap holding extra nvidia kernel module parameters.
30+
# The upstream chart only references an existing ConfigMap by name via
31+
# driver.kernelModuleConfig.name — it never creates it. When create is
32+
# true this wrapper chart renders the ConfigMap into the release
33+
# namespace so the params actually land. The driver entrypoint reads
34+
# ${base}/nvidia.conf and appends each line to the nvidia modprobe
35+
# options. Used by tenant Kubernetes clusters to set
36+
# NVreg_NvLinkDisable=1 (see packages/apps/kubernetes gpu-operator
37+
# HelmRelease defaults); off by default for host-level passthrough/vgpu
38+
# variants where the in-host driver is disabled.
39+
kernelModuleConfig:
40+
create: false
41+
name: nvidia-kernel-module-params
42+
content: |
43+
NVreg_NvLinkDisable=1

0 commit comments

Comments
 (0)