Skip to content

Commit c6f6fee

Browse files
authored
docs(gpu-operator): document passthrough variant host-driver prerequisite (#2767)
## What this PR does Documents an undocumented prerequisite of the `cozystack.gpu-operator` `default` (passthrough) variant: it assumes the GPU is not already claimed by a host-installed NVIDIA driver. The chart's `vfio-manager` DaemonSet runs the upstream NVIDIA `k8s-driver-manager` init container, which is hard-coded to exit non-zero and label the node `nvidia.com/gpu.deploy.driver=pre-installed` the moment `chroot /host nvidia-smi` returns a driver version (a `PATH` lookup, not a fixed file path). There is no env-var override in `k8s-driver-manager` v0.10.0, so today the only working path is a clean host. The new "Passthrough variant: host preparation" section in `docs/gpu-vgpu.md` covers: - The exact symptom (init log + sandbox-validator crashloop + `lspci` evidence) so operators can recognise it without spelunking. - A clean-host workaround: a build-from-installed purge list (`dpkg-query`, so a zero-match pattern can't silently abort the whole `apt` transaction the way a hardcoded `'cuda-*'` pattern does on a host without the CUDA repo), `/etc/modprobe.d/blacklist-nvidia.conf` (incl. `nvidia_peermem`), `update-initramfs -k all`, reboot. Warns that `autoremove` is dangerous on converged GPU + NVIDIA DOCA / Mellanox hosts because of the shared `nvidia-` prefix. - Post-reboot verification (no `nvidia` modules loaded, no `nvidia-smi` on `PATH`, no leftover NVIDIA DKMS module) and a label-cleanup step — the DaemonSet pods crash-loop and retry on their own, but the operator's success path never resets the `pre-installed` label, so it must be cleared manually. - A "Known limitation" subsection that names the upstream constraint and the planned mitigation path (upstream env-var override + a future `container` variant of the package). Linked to #2763. ### Release note ```release-note docs(gpu-operator): document that the passthrough (`default`) variant requires a clean host and explain how to purge a pre-installed NVIDIA host driver before enabling the variant. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Renamed GPU guide to cover passthrough and vGPU scenarios; updated verification/last-verified info. * Added passthrough host-preparation and troubleshooting, including a clean-host workaround with driver removal, kernel-module blacklist, initramfs update, reboot, and steps to force a fresh device bind. * Clarified FORCE_REINSTALL does not bypass early host-driver detection and noted pending mitigation paths. * Updated OS support table: passthrough requires clean-host on Ubuntu 20.04/22.04/24.04; Ubuntu 26.04 includes usr-merge notes and vGPU marked as caution (⚠️). <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 1d747b3 + cd36b0d commit c6f6fee

1 file changed

Lines changed: 129 additions & 6 deletions

File tree

‎docs/gpu-vgpu.md‎

Lines changed: 129 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# GPU Operator: vGPU Support
1+
# GPU Operator: passthrough and vGPU
22

33
This document describes how to configure the GPU Operator package with NVIDIA vGPU support so that a single physical GPU can be sliced and shared across multiple virtual machines.
44

@@ -29,6 +29,129 @@ The `gpu-operator` package exposes three variants. This document is vGPU-focused
2929
- **`vgpu`** — SR-IOV vGPU mode. One physical GPU is sliced into multiple VFs, each VF bound to a vGPU profile that the guest sees as its own GPU.
3030
- **`container`** — containerized GPU workloads (CUDA pods, ML training) via the standard NVIDIA device plugin on hosts that already provide both the NVIDIA driver and `nvidia-container-toolkit` (the typical apt-installed Ubuntu/Debian shape). Sandbox workloads are off, `devicePlugin` is on, `driver` / `toolkit` / `vfioManager` / `cdi` are off so that the operator does not fight the host install. Orthogonal to the two VM variants — it does not pass GPUs to KubeVirt VMs. Note that `apt install nvidia-container-toolkit` installs binaries only — it does not configure containerd. Because this variant disables the operator's toolkit component (which would normally do that wiring), the host must additionally have run `nvidia-ctk runtime configure --runtime=containerd` (followed by a containerd restart) and exposed the `nvidia` runtime as the default or via a RuntimeClass before the device plugin can serve GPUs.
3131

32+
## Passthrough variant: host preparation
33+
34+
**Last verified:** 2026-05-28 against `cozystack.gpu-operator` chart `gpu-operator` v26.3.1 + image `nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0` (distroless image with a Go ELF binary `/usr/bin/driver-manager`, entrypoint `["driver-manager", "preflight_check"]`; verified via `crane export … | tar -x usr/bin/driver-manager` and `strings` over the resulting binary) + Ubuntu 24.04 host with `nvidia-driver-580-open` 580.82.07 pre-installed. Tracks [#2763](https://github.com/cozystack/cozystack/issues/2763).
35+
36+
The `default` (passthrough) variant assumes the GPU is **owned by the host kernel's `vfio-pci` driver and nothing else**.
37+
38+
**How the operator detects a pre-installed host driver.** The chart's `vfio-manager` DaemonSet runs the upstream NVIDIA `k8s-driver-manager` init container with the `uninstall_driver` subcommand; that path calls the Go method `(*DriverManager).isHostDriver`, which runs `chroot /host nvidia-smi --query-gpu=driver_version --format=csv,noheader` and treats any non-empty stdout as "host driver present" (file-existence is not pre-tested — if `nvidia-smi` is missing, the chroot exec errors and `isHostDriver` returns false, which is the intended path on a clean host: the operator then proceeds with the uninstall flow and `vfio-manager` binds `vfio-pci` as designed). On a positive detection the binary logs `Host driver detected: <ver>`, labels the node `nvidia.com/gpu.deploy.driver=pre-installed`, exits, and `nvidia-sandbox-validator` then crashloops with `device not bound to 'vfio-pci'`.
39+
40+
**`FORCE_REINSTALL` does not bypass this.** `k8s-driver-manager` v0.10.0 exposes a `FORCE_REINSTALL` / `--force-reinstall` env+flag pair (visible in the binary's strings table), but it gates a later "same-config already loaded" branch inside `uninstallDriver`, not the `isHostDriver` short-circuit at the top — operators who set it and expect a bypass will report a false bug. There is currently no opt-out for the `isHostDriver` guard itself.
41+
42+
**Current mitigation.** The only mitigation available on `main` today is to remove the host NVIDIA stack before enabling the variant (the "Clean-host workaround" section below).
43+
44+
**Future alternatives and scope.** An upcoming `container` variant of `cozystack.gpu-operator` is tracked in [#2766](https://github.com/cozystack/cozystack/pull/2766) — once merged it will give operators a no-purge path that keeps the host driver and exposes GPUs to pods rather than VMs — but it is not part of the current package and is **not** a usable workaround until it lands. Talos is unaffected because the Talos image ships only the `vfio-pci` extension; this section applies to Linux distributions where you installed the host driver yourself (typically Ubuntu / Debian / RHEL with `apt install nvidia-driver-*` or equivalent). See also [`packages/system/gpu-operator/examples/README.md`](../packages/system/gpu-operator/examples/README.md) for the native-pod-workload reference flow that predates the `container` variant.
45+
46+
### Symptom
47+
48+
The first thing you see is `kubectl get pods -n cozy-gpu-operator` showing `nvidia-vfio-manager-*` stuck in `Init:Error` / `Init:CrashLoopBackOff` — the init container exits non-zero after detecting the host driver, so kubelet keeps restarting it.
49+
50+
Its log shows it skipped the bind step:
51+
52+
```text
53+
Host driver detected: 580.82.07
54+
NVIDIA GPU driver is already pre-installed on the node,
55+
disabling the containerized driver
56+
Labeling node <NODE> with nvidia.com/gpu.deploy.driver=pre-installed
57+
```
58+
59+
`nvidia-sandbox-validator` then crashloops:
60+
61+
```text
62+
Error: error validating vfio-pci driver installation:
63+
device not bound to 'vfio-pci'; device: 0000:18:00.0 driver: 'nvidia'
64+
```
65+
66+
`lspci -nnk -d 10de:` still shows `Kernel driver in use: nvidia` on every target GPU, the node carries `nvidia.com/gpu.deploy.driver=pre-installed`, and `kubectl get node <NODE> -o json | jq '.status.allocatable | with_entries(select(.key | startswith("nvidia.com/")))'` reports `{}` — no GPU resource was registered.
67+
68+
### Clean-host workaround
69+
70+
Purge the NVIDIA host stack and blacklist the kernel modules so the host never re-claims the GPU. Two pitfalls to avoid: `apt autoremove` is dangerous here because the `nvidia-` prefix is shared with NVIDIA DOCA / Mellanox / InfiniBand userspace, so a blanket autoremove can take RDMA out on a converged GPU + RDMA host; and a hardcoded `apt purge 'nvidia-*' 'cuda-*'` pattern list is fragile — `apt` treats `*` as a cache-wide regex and **aborts the entire transaction, purging nothing**, if any pattern matches nothing in the cache (e.g. `cuda-*` on a host without NVIDIA's CUDA repo). Build the list from what is actually installed instead:
71+
72+
```bash
73+
# dpkg-query patterns are true globs over INSTALLED packages: no
74+
# zero-match abort (unlike apt's cache-wide regex) and no accidental
75+
# substring over-match. List first, then review before purging.
76+
dpkg-query -W -f '${Package}\n' 'nvidia-*' 'libnvidia-*' 'cuda-*' 2>/dev/null
77+
```
78+
79+
Review the list before purging:
80+
81+
- `nvidia-dkms-*` and `nvidia-kernel-*` are the load-bearing kernel pieces — without removing them DKMS rebuilds `nvidia.ko` on the next reboot and the blacklist below is bypassed by any explicit `modprobe`.
82+
- On a **converged GPU + RDMA host**, drop any `libnvidia-*` that belong to NVIDIA DOCA / Mellanox OFED — purging those breaks RDMA.
83+
- If the list is **empty**, the driver was installed with NVIDIA's `.run` installer rather than apt — run `sudo nvidia-uninstall` instead of the purge below.
84+
85+
Then purge the reviewed list, blacklist the modules, and rebuild the initramfs:
86+
87+
```bash
88+
# Replace with the packages you kept from the list above.
89+
sudo apt purge nvidia-driver-580-open nvidia-dkms-580-open <...>
90+
91+
sudo tee /etc/modprobe.d/blacklist-nvidia.conf > /dev/null <<'EOF'
92+
blacklist nouveau
93+
blacklist nvidia
94+
blacklist nvidia_drm
95+
blacklist nvidia_modeset
96+
blacklist nvidia_uvm
97+
blacklist nvidia_peermem
98+
EOF
99+
100+
# -k all rebuilds EVERY installed kernel's initramfs (plain -u touches
101+
# only the running kernel), so a just-upgraded kernel also boots with
102+
# the blacklist and cannot re-claim the GPU.
103+
sudo update-initramfs -u -k all
104+
sudo reboot
105+
```
106+
107+
After the reboot, confirm the host is clean. The init container exits non-zero after detecting a host driver, so both DaemonSet pods sit in `Init:CrashLoopBackOff` and **will retry on their own** — deleting them just skips the up-to-5-minute backoff window:
108+
109+
```bash
110+
# Should print nothing.
111+
lsmod | grep -E '^(nvidia|nouveau)'
112+
113+
# command -v matches what isHostDriver does (PATH lookup inside the
114+
# chroot), so it also catches /usr/local/bin/nvidia-smi left by a .run
115+
# or CUDA-toolkit install. Prints "ok: gone" on success.
116+
command -v nvidia-smi >/dev/null && echo "STILL PRESENT — purge incomplete" || echo "ok: gone"
117+
118+
# A leftover DKMS module rebuilds nvidia.ko on the next kernel update
119+
# and an explicit modprobe bypasses the blacklist — should print nothing.
120+
dkms status | grep -i nvidia
121+
122+
# Skip the CrashLoopBackOff backoff window by deleting the stuck pods.
123+
# The DaemonSet labels are operator-managed and not stable across
124+
# gpu-operator versions, so delete by name pattern — anchored so the
125+
# match is the two DaemonSets and nothing else.
126+
kubectl -n cozy-gpu-operator get pods -o name \
127+
| grep -E '^pod/(nvidia-vfio-manager|nvidia-sandbox-validator)-' \
128+
| xargs -r kubectl -n cozy-gpu-operator delete
129+
```
130+
131+
Within a couple of minutes `vfio-manager` should bind every target GPU to `vfio-pci` and the node's `allocatable` will gain the registered resource:
132+
133+
```bash
134+
lspci -nnk -d 10de: | grep 'Kernel driver in use'
135+
# Kernel driver in use: vfio-pci
136+
137+
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable}{"\n"}{end}' | grep nvidia.com
138+
```
139+
140+
One label is **not** cleaned up automatically. The init container set `nvidia.com/gpu.deploy.driver=pre-installed`, and the operator's success path restores only the operand labels — it never resets this one (`rescheduleGPUOperatorComponents` in `k8s-driver-manager` v0.10.0 touches the validator / toolkit / device-plugin / etc. labels, not `deploy.driver`). So the node keeps `pre-installed` indefinitely: it is the same label the Symptom section uses as evidence, and it would disable the containerized-driver DaemonSet if the node later switches to the `container` workload. Clear it once recovery is confirmed:
141+
142+
```bash
143+
# Nothing on the success path resets this label; clear it so it does not
144+
# mislead future debugging or block a later container-workload switch.
145+
kubectl label node <NODE> nvidia.com/gpu.deploy.driver-
146+
```
147+
148+
### Known limitation
149+
150+
The skip-on-pre-installed behavior lives in the upstream [`NVIDIA/k8s-driver-manager`](https://github.com/NVIDIA/k8s-driver-manager) Go binary at `cmd/driver-manager/main.go`: the method `(*DriverManager).isHostDriver` is called from `(*DriverManager).uninstallDriver` and has no in-band opt-out in `:v0.10.0`. `FORCE_REINSTALL` exists as an env / CLI flag in the same binary but gates the later "same-config already loaded" branch inside `uninstallDriver`, not the `isHostDriver` short-circuit. Hosts that need to keep the NVIDIA host driver installed for non-Kubernetes workloads cannot currently share the same GPU with the passthrough variant. Two future mitigation paths are in flight; **neither is available today**:
151+
152+
- **`container` variant — not yet merged** — [#2766](https://github.com/cozystack/cozystack/pull/2766) adds a third variant of `cozystack.gpu-operator` that targets the apt-installed-driver host shape and exposes GPUs to pods (not VMs) without unbinding the host driver. Once it lands, operators on hosts where the purge workaround is unacceptable will have a path that does not require touching the host driver.
153+
- **Upstream override** — an env-var override of `isHostDriver` is the only structural fix that would let the passthrough variant coexist with a host driver. Requested in [NVIDIA/k8s-driver-manager#191](https://github.com/NVIDIA/k8s-driver-manager/issues/191).
154+
32155
## Building the vGPU Manager image
33156

34157
The proprietary vGPU Manager driver must be obtained from NVIDIA and packaged into a container image that the gpu-operator chart pulls — it is not installed from a raw `.run` at runtime. NVIDIA owns this build path; their [`gpu-driver-container`](https://github.com/NVIDIA/gpu-driver-container) repository ships per-OS Dockerfiles under `vgpu-manager/<os>/` and is the source of truth for build args, base images, and supported OS releases. Follow the README in that repository.
@@ -266,8 +389,8 @@ The `container` variant column assumes the host already ships the NVIDIA driver
266389

267390
| Host OS | passthrough (`default`) | vGPU (`vgpu`) | container (`container`) |
268391
| --- | --- | --- | --- |
269-
| Ubuntu 24.04 | ✅ supported upstream | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) | ✅ apt-installed driver + nvidia-container-toolkit |
270-
| Ubuntu 22.04 | ✅ | ✅ | ✅ |
271-
| Ubuntu 20.04 | ✅ | ✅ | ✅ |
272-
| Ubuntu 26.04 | ⚠️ patch needed in `nvidia-driver` for usr-merge | ⚠️ same patch + own Dockerfile fork | ✅ |
273-
| Talos Linux | ✅ (open `vfio-pci`) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run`; we tried and the path is blocked | ⚠️ host driver lands in a non-standard prefix — use `examples/values-native-talos.yaml` (compat DaemonSet + `hostPaths.driverInstallDir` override) as a starting point instead |
392+
| Ubuntu 24.04 | ⚠️ supported upstream, but the host must be clean of any apt-installed NVIDIA driver — see ["Passthrough variant: host preparation"](#passthrough-variant-host-preparation) | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) | ✅ apt-installed driver + nvidia-container-toolkit |
393+
| Ubuntu 22.04 | ⚠️ same clean-host requirement as 24.04 | ✅ | ✅ |
394+
| Ubuntu 20.04 | ⚠️ same clean-host requirement as 24.04 | ✅ | ✅ |
395+
| Ubuntu 26.04 | ⚠️ same clean-host requirement as 24.04, plus a `nvidia-driver` patch for usr-merge (details pending) | ⚠️ same patch + own Dockerfile fork | ✅ |
396+
| Talos Linux | ✅ (open `vfio-pci`; the Talos image ships no host NVIDIA stack, so the clean-host check passes trivially) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run`; we tried and the path is blocked | ⚠️ host driver lands in a non-standard prefix — use `examples/values-native-talos.yaml` (compat DaemonSet + `hostPaths.driverInstallDir` override) as a starting point instead |

0 commit comments

Comments
 (0)