Skip to content

Commit f30f047

Browse files
committed
docs(gpu-operator): document passthrough variant host-driver prerequisite
The vfio-manager DaemonSet's k8s-driver-manager init container exits early and labels the node pre-installed whenever it finds a host NVIDIA driver, leaving vfio-pci unbound and the sandbox validator in CrashLoopBackOff. Upstream offers no override knob, so the only working path today is a clean host. Document the symptom, the purge plus module-blacklist workaround (with the autoremove-vs-DOCA warning), and the post-reboot pod-restart step. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 67ba252 commit f30f047

1 file changed

Lines changed: 98 additions & 6 deletions

File tree

‎docs/gpu-vgpu.md‎

Lines changed: 98 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# GPU Operator: vGPU Support
1+
# GPU Operator: passthrough and vGPU
22

33
This document describes how to configure the GPU Operator package with NVIDIA vGPU support so that a single physical GPU can be sliced and shared across multiple virtual machines.
44

@@ -28,6 +28,98 @@ The `gpu-operator` package exposes two variants:
2828
- **`default`** — passthrough mode (`vfio-pci`). Whole GPU goes to a single VM. Talos is supported here; the kernel module is the open-source `vfio-pci`, no proprietary driver is needed on the host.
2929
- **`vgpu`** — SR-IOV vGPU mode. One physical GPU is sliced into multiple VFs, each VF bound to a vGPU profile that the guest sees as its own GPU.
3030

31+
## Passthrough variant: host preparation
32+
33+
**Last verified:** 2026-05-28 against `cozystack.gpu-operator` chart `gpu-operator` v26.3.1 + image `nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0` (distroless image with a Go ELF binary `/usr/bin/driver-manager`, entrypoint `["driver-manager", "preflight_check"]`; verified via `crane export … | tar -x usr/bin/driver-manager` and `strings` over the resulting binary) + Ubuntu 24.04 host with `nvidia-driver-580-open` 580.82.07 pre-installed. Tracks [#2763](https://github.com/cozystack/cozystack/issues/2763).
34+
35+
The `default` (passthrough) variant assumes the GPU is **owned by the host kernel's `vfio-pci` driver and nothing else**. The chart's `vfio-manager` DaemonSet runs the upstream NVIDIA `k8s-driver-manager` init container with the `uninstall_driver` subcommand; that path calls the Go method `(*DriverManager).isHostDriver`, which runs `chroot /host nvidia-smi --query-gpu=driver_version --format=csv,noheader` and treats any non-empty stdout as "host driver present" (file-existence is not pre-tested — if `nvidia-smi` is missing, the chroot exec errors and `isHostDriver` returns false, which is the intended path on a clean host: the operator then proceeds with the uninstall flow and `vfio-manager` binds `vfio-pci` as designed). On a positive detection the binary logs `Host driver detected: <ver>`, labels the node `nvidia.com/gpu.deploy.driver=pre-installed`, exits, and `nvidia-sandbox-validator` then crashloops with `device not bound to 'vfio-pci'`. `k8s-driver-manager` v0.10.0 exposes a `FORCE_REINSTALL` / `--force-reinstall` env+flag pair (visible in the binary's strings table), but it gates a later "same-config already loaded" branch inside `uninstallDriver`, not the `isHostDriver` short-circuit at the top — operators who set it and expect a bypass will report a false bug. There is currently no opt-out for the `isHostDriver` guard itself. The only mitigation available on `main` today is to remove the host NVIDIA stack before enabling the variant (the "Clean-host workaround" section below). An upcoming `container` variant of `cozystack.gpu-operator` is tracked in [#2766](https://github.com/cozystack/cozystack/pull/2766) — once merged it will give operators a no-purge path that keeps the host driver and exposes GPUs to pods rather than VMs — but it is not part of the current package and is **not** a usable workaround until it lands. Talos is unaffected because the Talos image ships only the `vfio-pci` extension; this section applies to Linux distributions where you installed the host driver yourself (typically Ubuntu / Debian / RHEL with `apt install nvidia-driver-*` or equivalent). See also [`packages/system/gpu-operator/examples/README.md`](../packages/system/gpu-operator/examples/README.md) for the native-pod-workload reference flow that predates the `container` variant.
36+
37+
### Symptom
38+
39+
`vfio-manager` reports it skipped the bind step:
40+
41+
```text
42+
Host driver detected: 580.82.07
43+
NVIDIA GPU driver is already pre-installed on the node,
44+
disabling the containerized driver
45+
Labeling node <NODE> with nvidia.com/gpu.deploy.driver=pre-installed
46+
```
47+
48+
`nvidia-sandbox-validator` then crashloops:
49+
50+
```text
51+
Error: error validating vfio-pci driver installation:
52+
device not bound to 'vfio-pci'; device: 0000:18:00.0 driver: 'nvidia'
53+
```
54+
55+
`lspci -nnk -d 10de:` still shows `Kernel driver in use: nvidia` on every target GPU, the node carries `nvidia.com/gpu.deploy.driver=pre-installed`, and `kubectl get node <NODE> -o json | jq '.status.allocatable | with_entries(select(.key | startswith("nvidia.com/")))'` reports `{}` — no GPU resource was registered.
56+
57+
### Clean-host workaround
58+
59+
Purge the NVIDIA host stack and blacklist the kernel modules so the host never re-claims the GPU. Use an **explicit package list** — `apt autoremove` is dangerous here because the `nvidia-` prefix is shared with NVIDIA DOCA / Mellanox / InfiniBand userspace, and a blanket autoremove can take those out at the same time on a converged GPU + RDMA host:
60+
61+
```bash
62+
# nvidia-dkms-* and nvidia-kernel-* are the load-bearing kernel
63+
# pieces — without them DKMS rebuilds nvidia.ko on the next reboot
64+
# and the blacklist below is bypassed by any explicit modprobe.
65+
# libnvidia-* is intentionally split into the userspace driver
66+
# pieces only — a bare 'libnvidia-*' glob also matches DOCA-OFED
67+
# shipped libnvidia packages on converged GPU + RDMA hosts.
68+
sudo apt purge \
69+
'nvidia-driver-*' 'nvidia-dkms-*' 'nvidia-kernel-*' \
70+
'nvidia-firmware-*' 'nvidia-utils-*' \
71+
'nvidia-modprobe' 'nvidia-settings' 'nvidia-prime' \
72+
'libnvidia-cfg1-*' 'libnvidia-compute-*' 'libnvidia-decode-*' \
73+
'libnvidia-encode-*' 'libnvidia-extra-*' 'libnvidia-fbc1-*' \
74+
'libnvidia-gl-*' \
75+
'cuda-toolkit-*' 'cuda-drivers' 'cuda-drivers-*'
76+
77+
sudo tee /etc/modprobe.d/blacklist-nvidia.conf > /dev/null <<'EOF'
78+
blacklist nouveau
79+
blacklist nvidia
80+
blacklist nvidia_drm
81+
blacklist nvidia_modeset
82+
blacklist nvidia_uvm
83+
EOF
84+
85+
sudo update-initramfs -u
86+
sudo reboot
87+
```
88+
89+
After the reboot, confirm the host is clean and kick the stuck DaemonSet pods so they re-create against the cleaned state — they have already labeled the node and will not retry on their own:
90+
91+
```bash
92+
# Should print nothing.
93+
lsmod | grep -E '^(nvidia|nouveau)'
94+
95+
# Self-evident verification: prints "ok: gone" on success.
96+
test -e /usr/bin/nvidia-smi && echo "STILL PRESENT — purge incomplete" || echo "ok: gone"
97+
98+
# Force a fresh pass of vfio-manager and the sandbox validator.
99+
# The DaemonSet labels are operator-managed and not stable across
100+
# gpu-operator versions, so delete by name pattern — anchored so the
101+
# match is the two DaemonSets and nothing else.
102+
kubectl -n cozy-gpu-operator get pods -o name \
103+
| grep -E '^pod/(nvidia-vfio-manager|nvidia-sandbox-validator)-' \
104+
| xargs -r kubectl -n cozy-gpu-operator delete
105+
```
106+
107+
Within a couple of minutes `vfio-manager` should bind every target GPU to `vfio-pci` and the node's `allocatable` will gain the registered resource:
108+
109+
```bash
110+
lspci -nnk -d 10de: | grep 'Kernel driver in use'
111+
# Kernel driver in use: vfio-pci
112+
113+
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable}{"\n"}{end}' | grep nvidia.com
114+
```
115+
116+
### Known limitation
117+
118+
The skip-on-pre-installed behavior lives in the upstream [`NVIDIA/k8s-driver-manager`](https://github.com/NVIDIA/k8s-driver-manager) Go binary at `cmd/driver-manager/main.go`: the method `(*DriverManager).isHostDriver` is called from `(*DriverManager).uninstallDriver` and has no in-band opt-out in `:v0.10.0`. `FORCE_REINSTALL` exists as an env / CLI flag in the same binary but gates the later "same-config already loaded" branch inside `uninstallDriver`, not the `isHostDriver` short-circuit. Hosts that need to keep the NVIDIA host driver installed for non-Kubernetes workloads cannot currently share the same GPU with the passthrough variant. Two future mitigation paths are in flight; **neither is available today**:
119+
120+
- **`container` variant — not yet merged** — [#2766](https://github.com/cozystack/cozystack/pull/2766) adds a third variant of `cozystack.gpu-operator` that targets the apt-installed-driver host shape and exposes GPUs to pods (not VMs) without unbinding the host driver. Once it lands, operators on hosts where the purge workaround is unacceptable will have a path that does not require touching the host driver.
121+
- **Upstream override** — an env-var override of `isHostDriver` is the only structural fix that would let the passthrough variant coexist with a host driver. Requested in [NVIDIA/k8s-driver-manager#191](https://github.com/NVIDIA/k8s-driver-manager/issues/191).
122+
31123
## Building the vGPU Manager image
32124

33125
The proprietary vGPU Manager driver must be obtained from NVIDIA and packaged into a container image that the gpu-operator chart pulls — it is not installed from a raw `.run` at runtime. NVIDIA owns this build path; their [`gpu-driver-container`](https://github.com/NVIDIA/gpu-driver-container) repository ships per-OS Dockerfiles under `vgpu-manager/<os>/` and is the source of truth for build args, base images, and supported OS releases. Follow the README in that repository.
@@ -233,8 +325,8 @@ Other GPU families have analogous tables in the [NVIDIA Virtual GPU Software Doc
233325

234326
| Host OS | passthrough (`default`) | vGPU (`vgpu`) |
235327
| --- | --- | --- |
236-
| Ubuntu 24.04 | ✅ supported upstream | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) |
237-
| Ubuntu 22.04 | ✅ | ✅ |
238-
| Ubuntu 20.04 | ✅ | ✅ |
239-
| Ubuntu 26.04 | ⚠️ patch needed in `nvidia-driver` for usr-merge | ⚠️ same patch + own Dockerfile fork |
240-
| Talos Linux | ✅ (open `vfio-pci`) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run`; we tried and the path is blocked |
328+
| Ubuntu 24.04 | ⚠️ supported upstream, but the host must be clean of any apt-installed NVIDIA driver — see ["Passthrough variant: host preparation"](#passthrough-variant-host-preparation) | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) |
329+
| Ubuntu 22.04 | ⚠️ same clean-host requirement as 24.04 | ✅ |
330+
| Ubuntu 20.04 | ⚠️ same clean-host requirement as 24.04 | ✅ |
331+
| Ubuntu 26.04 | ⚠️ same clean-host requirement as 24.04, plus a `nvidia-driver` patch for usr-merge | ⚠️ same patch + own Dockerfile fork |
332+
| Talos Linux | ✅ (open `vfio-pci`; the Talos image ships no host NVIDIA stack, so the clean-host check passes trivially) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run`; we tried and the path is blocked |

0 commit comments

Comments
 (0)