Skip to content

Commit b610803

Browse files
authored
feat(gpu-operator): add container variant for preinstalled host driver (cozystack#2766)
## What this PR does Add a third architectural variant to `cozystack.gpu-operator` for hosts where the NVIDIA driver and `nvidia-container-toolkit` are already installed by the distro package manager (the typical Ubuntu/Debian GPU node shape: apt-installed driver + apt-installed CT). The existing variants do not produce a working setup in that shape. `default` (passthrough) runs `vfioManager`, which unbinds the host driver to bind `vfio-pci` — with the host driver active, `k8s-driver-manager` early-exits and the bind never happens, so the GPU never reaches a pod. `vgpu` requires the proprietary NVIDIA vGPU host driver and a license server; the user-installed open or proprietary driver does not satisfy it. The new `container` variant maps onto upstream gpu-operator's third mode (`sandboxWorkloads.enabled=false` + `devicePlugin.enabled=true`) with every component that would conflict with the host install switched off: - `driver.enabled=false` — use the host driver. - `toolkit.enabled=false` — host already has `nvidia-container-toolkit`. Enabling the operator's toolkit DaemonSet would overwrite the `/etc/containerd/config.toml` the host package configured (via `nvidia-ctk runtime configure`), breaking the host runtime wiring. Matches NVIDIA's documented guidance and the existing `examples/values-native-talos.yaml` reference, which pins the same flag for the same reason. - `devicePlugin.enabled=true` — publish `nvidia.com/gpu` to the kubelet. - `vfioManager.enabled=false` — do not unbind the host driver. Upstream renders this `true` unconditionally; it only stays at zero pods because no node carries a `vm-passthrough` sandbox assignment when `sandboxWorkloads` is off — pinned off so the rendered policy matches intent. - `cdi.enabled=false` — upstream defaults this `true`, but CDI spec generation is serviced by the operator-managed toolkit DaemonSet this variant disables. An apt host doesn't auto-generate CDI specs (`nvidia-ctk cdi generate` is a manual step) and uses the classic `nvidia-container-runtime` hook, so leaving CDI on would point the device plugin at CDI injection with no specs on the host and silently break allocation. - `sandboxWorkloads.enabled=false` — regular pod workloads, not VMs. With this off, upstream `defaultWorkload` is already `container`, so no override is needed. Host prerequisite: because the operator's toolkit component is disabled, the node's containerd must already have the `nvidia` runtime registered (`nvidia-ctk runtime configure --runtime=containerd`); `apt install nvidia-container-toolkit` installs binaries only and does not wire containerd. With `driver.enabled=false` the operator uses the pre-installed host driver at its standard location, so a stock apt install needs no `hostPaths.driverInstallDir` override (Talos lands the driver under a non-standard prefix — see `examples/values-native-talos.yaml`). The bundle wiring (`packages/core/platform/templates/bundles/iaas.yaml`) already enables `cozystack.gpu-operator` as an optional package — no bundle changes required. Users opt into this variant by setting `variant: container` in their `Package` CR. Verified locally with `helm template` against `values.yaml` + `values-container.yaml`: ClusterPolicy renders with `driver.enabled=false`, `toolkit.enabled=false`, `devicePlugin.enabled=true`, `vfioManager.enabled=false`, `cdi.enabled=false`, `sandboxWorkloads.enabled=false`. Platform `helm unittest` and the `gpu-operator` variant bats suite are green. Not yet smoke-tested on a real Ubuntu GPU node — the config matches the documented apt-host wiring (legacy `nvidia-container-runtime` hook, CDI off), but an end-to-end CUDA-pod run on hardware is still owed. Closes cozystack#2764 ### Release note ```release-note feat(gpu-operator): add `container` variant for hosts with a pre-installed NVIDIA driver and nvidia-container-toolkit, exposing containerized GPU workloads via the standard NVIDIA device plugin without VFIO passthrough or vGPU. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a "container" GPU operator variant to support containerized GPU workloads on hosts with the NVIDIA driver and nvidia-container-toolkit. * **Documentation** * Updated variants documentation and OS support table to include the container variant and per-host requirements (including Ubuntu and Talos notes). * Clarified Talos native pod example wording. * **Tests** * Added regression tests to verify container-variant configuration and component enablement/disablement. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents fb070ff + 9352d3e commit b610803

6 files changed

Lines changed: 135 additions & 30 deletions

File tree

‎docs/gpu-vgpu.md‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -23,10 +23,11 @@ This guide focuses on the **SR-IOV path**, which is the only model NVIDIA suppor
2323

2424
## Variants
2525

26-
The `gpu-operator` package exposes two variants:
26+
The `gpu-operator` package exposes three variants. This document is vGPU-focused; the variant inventory is shared.
2727

2828
- **`default`** — passthrough mode (`vfio-pci`). Whole GPU goes to a single VM. Talos is supported here; the kernel module is the open-source `vfio-pci`, no proprietary driver is needed on the host.
2929
- **`vgpu`** — SR-IOV vGPU mode. One physical GPU is sliced into multiple VFs, each VF bound to a vGPU profile that the guest sees as its own GPU.
30+
- **`container`** — containerized GPU workloads (CUDA pods, ML training) via the standard NVIDIA device plugin on hosts that already provide both the NVIDIA driver and `nvidia-container-toolkit` (the typical apt-installed Ubuntu/Debian shape). Sandbox workloads are off, `devicePlugin` is on, `driver` / `toolkit` / `vfioManager` / `cdi` are off so that the operator does not fight the host install. Orthogonal to the two VM variants — it does not pass GPUs to KubeVirt VMs. Note that `apt install nvidia-container-toolkit` installs binaries only — it does not configure containerd. Because this variant disables the operator's toolkit component (which would normally do that wiring), the host must additionally have run `nvidia-ctk runtime configure --runtime=containerd` (followed by a containerd restart) and exposed the `nvidia` runtime as the default or via a RuntimeClass before the device plugin can serve GPUs.
3031

3132
## Building the vGPU Manager image
3233

@@ -231,10 +232,12 @@ Other GPU families have analogous tables in the [NVIDIA Virtual GPU Software Doc
231232

232233
## OS support summary
233234

234-
| Host OS | passthrough (`default`) | vGPU (`vgpu`) |
235-
| --- | --- | --- |
236-
| Ubuntu 24.04 | ✅ supported upstream | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) |
237-
| Ubuntu 22.04 | ✅ | ✅ |
238-
| Ubuntu 20.04 | ✅ | ✅ |
239-
| Ubuntu 26.04 | ⚠️ patch needed in `nvidia-driver` for usr-merge | ⚠️ same patch + own Dockerfile fork |
240-
| Talos Linux | ✅ (open `vfio-pci`) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run`; we tried and the path is blocked |
235+
The `container` variant column assumes the host already ships the NVIDIA driver and `nvidia-container-toolkit` via the distro package manager, with the `nvidia` runtime registered in containerd (`nvidia-ctk runtime configure --runtime=containerd`). With `driver.enabled=false` the operator uses the pre-installed host driver at its standard location, so a stock apt install needs no `hostPaths.driverInstallDir` override. Talos installs the driver under a non-standard prefix, so the operator does not find it at the default location — see `packages/system/gpu-operator/examples/` for the Talos-specific path with a compat DaemonSet and an explicit `hostPaths.driverInstallDir` override.
236+
237+
| Host OS | passthrough (`default`) | vGPU (`vgpu`) | container (`container`) |
238+
| --- | --- | --- | --- |
239+
| Ubuntu 24.04 | ✅ supported upstream | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) | ✅ apt-installed driver + nvidia-container-toolkit |
240+
| Ubuntu 22.04 | ✅ | ✅ | ✅ |
241+
| Ubuntu 20.04 | ✅ | ✅ | ✅ |
242+
| Ubuntu 26.04 | ⚠️ patch needed in `nvidia-driver` for usr-merge | ⚠️ same patch + own Dockerfile fork | ✅ |
243+
| Talos Linux | ✅ (open `vfio-pci`) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run`; we tried and the path is blocked | ⚠️ host driver lands in a non-standard prefix — use `examples/values-native-talos.yaml` (compat DaemonSet + `hostPaths.driverInstallDir` override) as a starting point instead |

‎hack/check-gpu-operator-variants.bats‎

Lines changed: 51 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,36 @@
11
#!/usr/bin/env bats
22
# -----------------------------------------------------------------------------
3-
# Catches silent regressions in the gpu-operator package's two variants:
3+
# Catches silent regressions in the gpu-operator package's three variants:
44
#
55
# 1. The base values.yaml pins ccManager.enabled=false and
66
# vgpuDeviceManager.enabled=false. Both upstream defaults are now true
77
# (chart v26.x). If the next chart bump silently un-pins either one,
88
# the `default` (passthrough) variant would gain confidential-computing
99
# auto-enable on Hopper hardware (ccManager) or render an mdev DaemonSet
10-
# that crashloops on Ada+/Blackwell (vgpuDeviceManager).
10+
# that crashloops on Ada+/Blackwell (vgpuDeviceManager). The same union
11+
# must hold for the `container` variant; the `vgpu` variant re-affirms
12+
# vgpuDeviceManager=false in its own overlay for the same reason.
1113
#
12-
# 2. Each variant's overlay must set sandboxWorkloads.defaultWorkload to its
13-
# own enum value (vm-passthrough / vm-vgpu) — without it, GPU nodes that
14-
# lack the per-node label nvidia.com/gpu.workload.config fall back to
15-
# the upstream default 'container' workload, no DaemonSet renders, and
16-
# the variant is silently a no-op.
14+
# 2. The sandbox variants must set sandboxWorkloads.defaultWorkload to
15+
# their own enum value (vm-passthrough / vm-vgpu) — without it, GPU
16+
# nodes that lack the per-node label nvidia.com/gpu.workload.config
17+
# fall back to the upstream default 'container' workload, no DaemonSet
18+
# renders, and the variant is silently a no-op. The container variant
19+
# relies on the same upstream default but in the opposite direction:
20+
# with sandboxWorkloads.enabled=false the chart picks defaultWorkload
21+
# 'container', which is what that variant wants — pinned here so an
22+
# upstream rename does not silently break it.
1723
#
1824
# 3. The vgpu variant must enable the vGPU manager DaemonSet
19-
# (vgpuManager.enabled=true) and the passthrough variant must keep
20-
# driver.enabled / devicePlugin.enabled at false.
25+
# (vgpuManager.enabled=true), the passthrough variant must keep
26+
# driver.enabled / devicePlugin.enabled at false, and the container
27+
# variant must keep driver / toolkit / vfioManager at false (host
28+
# already provides them) while keeping devicePlugin enabled (publishes
29+
# nvidia.com/gpu to the kubelet). The container variant must also pin
30+
# cdi.enabled=false: upstream defaults it true, but CDI specs are
31+
# serviced by the toolkit DaemonSet this variant disables, so leaving
32+
# it on points the device plugin at CDI injection with no specs on an
33+
# apt host and silently breaks allocation.
2134
#
2235
# Implementation: render the chart with the merged values for each variant
2336
# and grep for the relevant ClusterPolicy keys. We strip the package's
@@ -96,3 +109,32 @@ render_variant() {
96109
grep -A 1 '^ driver:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
97110
grep -A 1 '^ devicePlugin:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
98111
}
112+
113+
@test "container variant: ccManager and vgpuDeviceManager pinned off" {
114+
TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT
115+
render_variant container
116+
grep -A 1 '^ ccManager:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
117+
grep -A 1 '^ vgpuDeviceManager:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
118+
}
119+
120+
@test "container variant: sandboxWorkloads off, defaultWorkload stays upstream 'container'" {
121+
TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT
122+
render_variant container
123+
grep -A 2 '^ sandboxWorkloads:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
124+
grep -A 2 '^ sandboxWorkloads:' "$TMP/rendered.yaml" | grep -q 'defaultWorkload: container'
125+
}
126+
127+
@test "container variant: driver, toolkit, vfioManager off; devicePlugin on" {
128+
TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT
129+
render_variant container
130+
grep -A 1 '^ driver:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
131+
grep -A 1 '^ toolkit:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
132+
grep -A 1 '^ vfioManager:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
133+
grep -A 1 '^ devicePlugin:' "$TMP/rendered.yaml" | grep -q 'enabled: true'
134+
}
135+
136+
@test "container variant: cdi pinned off (toolkit disabled cannot service CDI specs)" {
137+
TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT
138+
render_variant container
139+
grep -A 1 '^ cdi:' "$TMP/rendered.yaml" | grep -q 'enabled: false'
140+
}

‎packages/core/platform/sources/gpu-operator.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,16 @@ spec:
3636
privileged: true
3737
namespace: cozy-gpu-operator
3838
releaseName: gpu-operator
39+
- name: container
40+
dependsOn:
41+
- cozystack.networking
42+
components:
43+
- name: gpu-operator
44+
path: system/gpu-operator
45+
valuesFiles:
46+
- values.yaml
47+
- values-container.yaml
48+
install:
49+
privileged: true
50+
namespace: cozy-gpu-operator
51+
releaseName: gpu-operator

‎packages/system/gpu-operator/examples/README.md‎

Lines changed: 5 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,10 @@
11
# GPU operator — native pod workload on Talos (reference)
22

3-
The files in this directory are **not** templates. They are reference
4-
artifacts that document one working configuration for running GPU
5-
workloads directly in pods on a Talos-based Cozystack cluster, together
6-
with the DCGM metrics needed by the `gpu/gpu-performance` Grafana
7-
dashboard.
8-
9-
The out-of-the-box `values-passthrough.yaml` for this package targets the
10-
sandbox (VFIO passthrough to KubeVirt VMs) scenario. The files here
11-
illustrate an alternative — running CUDA workloads in regular pods with
12-
the NVIDIA device plugin — and the workarounds it currently requires on
13-
Talos.
3+
The files in this directory are **not** templates. They are reference artifacts that document one working configuration for running GPU workloads directly in pods on a Talos-based Cozystack cluster, together with the DCGM metrics needed by the `gpu/gpu-performance` Grafana dashboard.
4+
5+
For non-Talos Linux hosts that already provide the NVIDIA driver and `nvidia-container-toolkit` via the distro package manager (the typical apt-installed Ubuntu/Debian shape), pick the first-class `container` variant instead — `variant: container` in your Package CR is the starting point. Two host prerequisites still apply: containerd must have the `nvidia` runtime registered (`nvidia-ctk runtime configure --runtime=containerd` + restart; `apt install` lays down binaries only, it does not wire containerd), and because the variant runs with `driver.enabled=false` the operator uses the pre-installed host driver at its standard location — so a stock apt install needs no `hostPaths.driverInstallDir` override. The files here remain the right starting point on Talos, where the driver lands under a non-standard prefix and the operator does not find it at the default location without the override below.
6+
7+
The out-of-the-box `values-passthrough.yaml` for this package targets the sandbox (VFIO passthrough to KubeVirt VMs) scenario. The files here illustrate an alternative — running CUDA workloads in regular pods with the NVIDIA device plugin — and the workarounds it currently requires on Talos.
148

159
## Files
1610

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# Container variant: GPU workloads run in regular pods on hosts that
2+
# already have the NVIDIA driver and nvidia-container-toolkit installed
3+
# (typical Ubuntu/Debian shape: apt-installed driver + apt-installed
4+
# nvidia-container-toolkit). The third architectural mode that upstream
5+
# NVIDIA gpu-operator supports — sandboxWorkloads disabled,
6+
# devicePlugin enabled — surfaced here as a Cozystack variant.
7+
gpu-operator:
8+
sandboxWorkloads:
9+
# Default upstream workload on a node without the per-node
10+
# 'nvidia.com/gpu.workload.config' label is 'container' when
11+
# sandboxWorkloads is off — which is exactly what this variant
12+
# wants. No defaultWorkload override needed; the counterpart
13+
# overrides in values-passthrough.yaml / values-vgpu.yaml exist
14+
# only because those variants flip sandboxWorkloads to true.
15+
enabled: false
16+
driver:
17+
# Host already runs the NVIDIA driver (apt-installed). The
18+
# operator's driver DaemonSet would attempt to load its own
19+
# kernel module and clash with the host one.
20+
enabled: false
21+
devicePlugin:
22+
# Publish nvidia.com/gpu to the kubelet so pods can request GPUs.
23+
enabled: true
24+
vfioManager:
25+
# vfioManager unbinds the host driver to bind vfio-pci, which must
26+
# not happen when the host driver is in use and devicePlugin is on.
27+
# Upstream renders vfioManager.enabled=true into the ClusterPolicy
28+
# unconditionally; it only stays at zero pods because the DaemonSet
29+
# schedules solely onto nodes labeled
30+
# nvidia.com/gpu.workload.config=vm-passthrough, and no node carries
31+
# a sandbox assignment when sandboxWorkloads is off. Pin it off so
32+
# the rendered policy matches intent regardless of upstream
33+
# scheduling changes.
34+
enabled: false
35+
toolkit:
36+
# Host already has nvidia-container-toolkit installed (the typical
37+
# apt-installed shape). Running the operator's toolkit DaemonSet
38+
# would overwrite /etc/containerd/config.toml that the host package
39+
# configured (via `nvidia-ctk runtime configure`), breaking the
40+
# host runtime wiring. NVIDIA's documentation recommends disabling
41+
# this when the toolkit is preinstalled on the node.
42+
enabled: false
43+
cdi:
44+
# Upstream defaults cdi.enabled=true, but CDI spec generation and
45+
# runtime wiring are serviced by the operator-managed toolkit, which
46+
# this variant disables. An apt-installed nvidia-container-toolkit
47+
# does not generate CDI specs on its own (`nvidia-ctk cdi generate`
48+
# is a manual step) and uses the classic nvidia-container-runtime
49+
# hook path. Leaving CDI on would point the device plugin at a
50+
# CDI-injection mode with no specs on the host and silently break
51+
# GPU allocation. Pin off so device allocation matches the host
52+
# runtime wiring.
53+
enabled: false

‎packages/system/gpu-operator/values.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Defaults applied to every variant of the gpu-operator package.
2-
# The variant-specific files (values-passthrough.yaml, values-vgpu.yaml)
3-
# layer on top.
2+
# The variant-specific files (values-passthrough.yaml, values-vgpu.yaml,
3+
# values-container.yaml) layer on top.
44
gpu-operator:
55
# Upstream chart v26.x flips ccManager.enabled to true and
66
# ccManager.defaultMode to "on". On Hopper-class hardware

0 commit comments

Comments
 (0)