|
1 | | -# GPU Operator: vGPU Support |
| 1 | +# GPU Operator: passthrough and vGPU |
2 | 2 |
|
3 | 3 | This document describes how to configure the GPU Operator package with NVIDIA vGPU support so that a single physical GPU can be sliced and shared across multiple virtual machines. |
4 | 4 |
|
@@ -28,6 +28,98 @@ The `gpu-operator` package exposes two variants: |
28 | 28 | - **`default`** — passthrough mode (`vfio-pci`). Whole GPU goes to a single VM. Talos is supported here; the kernel module is the open-source `vfio-pci`, no proprietary driver is needed on the host. |
29 | 29 | - **`vgpu`** — SR-IOV vGPU mode. One physical GPU is sliced into multiple VFs, each VF bound to a vGPU profile that the guest sees as its own GPU. |
30 | 30 |
|
| 31 | +## Passthrough variant: host preparation |
| 32 | + |
| 33 | +**Last verified:** 2026-05-28 against `cozystack.gpu-operator` chart `gpu-operator` v26.3.1 + image `nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0` (distroless image with a Go ELF binary `/usr/bin/driver-manager`, entrypoint `["driver-manager", "preflight_check"]`; verified via `crane export … | tar -x usr/bin/driver-manager` and `strings` over the resulting binary) + Ubuntu 24.04 host with `nvidia-driver-580-open` 580.82.07 pre-installed. Tracks [#2763](https://github.com/cozystack/cozystack/issues/2763). |
| 34 | + |
| 35 | +The `default` (passthrough) variant assumes the GPU is **owned by the host kernel's `vfio-pci` driver and nothing else**. The chart's `vfio-manager` DaemonSet runs the upstream NVIDIA `k8s-driver-manager` init container with the `uninstall_driver` subcommand; that path calls the Go method `(*DriverManager).isHostDriver`, which runs `chroot /host nvidia-smi --query-gpu=driver_version --format=csv,noheader` and treats any non-empty stdout as "host driver present" (file-existence is not pre-tested — if `nvidia-smi` is missing, the chroot exec errors and `isHostDriver` returns false, which is the intended path on a clean host: the operator then proceeds with the uninstall flow and `vfio-manager` binds `vfio-pci` as designed). On a positive detection the binary logs `Host driver detected: <ver>`, labels the node `nvidia.com/gpu.deploy.driver=pre-installed`, exits, and `nvidia-sandbox-validator` then crashloops with `device not bound to 'vfio-pci'`. `k8s-driver-manager` v0.10.0 exposes a `FORCE_REINSTALL` / `--force-reinstall` env+flag pair (visible in the binary's strings table), but it gates a later "same-config already loaded" branch inside `uninstallDriver`, not the `isHostDriver` short-circuit at the top — operators who set it and expect a bypass will report a false bug. There is currently no opt-out for the `isHostDriver` guard itself. The only mitigation available on `main` today is to remove the host NVIDIA stack before enabling the variant (the "Clean-host workaround" section below). An upcoming `container` variant of `cozystack.gpu-operator` is tracked in [#2766](https://github.com/cozystack/cozystack/pull/2766) — once merged it will give operators a no-purge path that keeps the host driver and exposes GPUs to pods rather than VMs — but it is not part of the current package and is **not** a usable workaround until it lands. Talos is unaffected because the Talos image ships only the `vfio-pci` extension; this section applies to Linux distributions where you installed the host driver yourself (typically Ubuntu / Debian / RHEL with `apt install nvidia-driver-*` or equivalent). See also [`packages/system/gpu-operator/examples/README.md`](../packages/system/gpu-operator/examples/README.md) for the native-pod-workload reference flow that predates the `container` variant. |
| 36 | + |
| 37 | +### Symptom |
| 38 | + |
| 39 | +`vfio-manager` reports it skipped the bind step: |
| 40 | + |
| 41 | +```text |
| 42 | +Host driver detected: 580.82.07 |
| 43 | +NVIDIA GPU driver is already pre-installed on the node, |
| 44 | + disabling the containerized driver |
| 45 | +Labeling node <NODE> with nvidia.com/gpu.deploy.driver=pre-installed |
| 46 | +``` |
| 47 | + |
| 48 | +`nvidia-sandbox-validator` then crashloops: |
| 49 | + |
| 50 | +```text |
| 51 | +Error: error validating vfio-pci driver installation: |
| 52 | + device not bound to 'vfio-pci'; device: 0000:18:00.0 driver: 'nvidia' |
| 53 | +``` |
| 54 | + |
| 55 | +`lspci -nnk -d 10de:` still shows `Kernel driver in use: nvidia` on every target GPU, the node carries `nvidia.com/gpu.deploy.driver=pre-installed`, and `kubectl get node <NODE> -o json | jq '.status.allocatable | with_entries(select(.key | startswith("nvidia.com/")))'` reports `{}` — no GPU resource was registered. |
| 56 | + |
| 57 | +### Clean-host workaround |
| 58 | + |
| 59 | +Purge the NVIDIA host stack and blacklist the kernel modules so the host never re-claims the GPU. Use an **explicit package list** — `apt autoremove` is dangerous here because the `nvidia-` prefix is shared with NVIDIA DOCA / Mellanox / InfiniBand userspace, and a blanket autoremove can take those out at the same time on a converged GPU + RDMA host: |
| 60 | + |
| 61 | +```bash |
| 62 | +# nvidia-dkms-* and nvidia-kernel-* are the load-bearing kernel |
| 63 | +# pieces — without them DKMS rebuilds nvidia.ko on the next reboot |
| 64 | +# and the blacklist below is bypassed by any explicit modprobe. |
| 65 | +# libnvidia-* is intentionally split into the userspace driver |
| 66 | +# pieces only — a bare 'libnvidia-*' glob also matches DOCA-OFED |
| 67 | +# shipped libnvidia packages on converged GPU + RDMA hosts. |
| 68 | +sudo apt purge \ |
| 69 | + 'nvidia-driver-*' 'nvidia-dkms-*' 'nvidia-kernel-*' \ |
| 70 | + 'nvidia-firmware-*' 'nvidia-utils-*' \ |
| 71 | + 'nvidia-modprobe' 'nvidia-settings' 'nvidia-prime' \ |
| 72 | + 'libnvidia-cfg1-*' 'libnvidia-compute-*' 'libnvidia-decode-*' \ |
| 73 | + 'libnvidia-encode-*' 'libnvidia-extra-*' 'libnvidia-fbc1-*' \ |
| 74 | + 'libnvidia-gl-*' \ |
| 75 | + 'cuda-toolkit-*' 'cuda-drivers' 'cuda-drivers-*' |
| 76 | + |
| 77 | +sudo tee /etc/modprobe.d/blacklist-nvidia.conf > /dev/null <<'EOF' |
| 78 | +blacklist nouveau |
| 79 | +blacklist nvidia |
| 80 | +blacklist nvidia_drm |
| 81 | +blacklist nvidia_modeset |
| 82 | +blacklist nvidia_uvm |
| 83 | +EOF |
| 84 | + |
| 85 | +sudo update-initramfs -u |
| 86 | +sudo reboot |
| 87 | +``` |
| 88 | + |
| 89 | +After the reboot, confirm the host is clean and kick the stuck DaemonSet pods so they re-create against the cleaned state — they have already labeled the node and will not retry on their own: |
| 90 | + |
| 91 | +```bash |
| 92 | +# Should print nothing. |
| 93 | +lsmod | grep -E '^(nvidia|nouveau)' |
| 94 | + |
| 95 | +# Self-evident verification: prints "ok: gone" on success. |
| 96 | +test -e /usr/bin/nvidia-smi && echo "STILL PRESENT — purge incomplete" || echo "ok: gone" |
| 97 | + |
| 98 | +# Force a fresh pass of vfio-manager and the sandbox validator. |
| 99 | +# The DaemonSet labels are operator-managed and not stable across |
| 100 | +# gpu-operator versions, so delete by name pattern — anchored so the |
| 101 | +# match is the two DaemonSets and nothing else. |
| 102 | +kubectl -n cozy-gpu-operator get pods -o name \ |
| 103 | + | grep -E '^pod/(nvidia-vfio-manager|nvidia-sandbox-validator)-' \ |
| 104 | + | xargs -r kubectl -n cozy-gpu-operator delete |
| 105 | +``` |
| 106 | + |
| 107 | +Within a couple of minutes `vfio-manager` should bind every target GPU to `vfio-pci` and the node's `allocatable` will gain the registered resource: |
| 108 | + |
| 109 | +```bash |
| 110 | +lspci -nnk -d 10de: | grep 'Kernel driver in use' |
| 111 | +# Kernel driver in use: vfio-pci |
| 112 | + |
| 113 | +kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable}{"\n"}{end}' | grep nvidia.com |
| 114 | +``` |
| 115 | + |
| 116 | +### Known limitation |
| 117 | + |
| 118 | +The skip-on-pre-installed behavior lives in the upstream [`NVIDIA/k8s-driver-manager`](https://github.com/NVIDIA/k8s-driver-manager) Go binary at `cmd/driver-manager/main.go`: the method `(*DriverManager).isHostDriver` is called from `(*DriverManager).uninstallDriver` and has no in-band opt-out in `:v0.10.0`. `FORCE_REINSTALL` exists as an env / CLI flag in the same binary but gates the later "same-config already loaded" branch inside `uninstallDriver`, not the `isHostDriver` short-circuit. Hosts that need to keep the NVIDIA host driver installed for non-Kubernetes workloads cannot currently share the same GPU with the passthrough variant. Two future mitigation paths are in flight; **neither is available today**: |
| 119 | + |
| 120 | +- **`container` variant — not yet merged** — [#2766](https://github.com/cozystack/cozystack/pull/2766) adds a third variant of `cozystack.gpu-operator` that targets the apt-installed-driver host shape and exposes GPUs to pods (not VMs) without unbinding the host driver. Once it lands, operators on hosts where the purge workaround is unacceptable will have a path that does not require touching the host driver. |
| 121 | +- **Upstream override** — an env-var override of `isHostDriver` is the only structural fix that would let the passthrough variant coexist with a host driver. Requested in [NVIDIA/k8s-driver-manager#191](https://github.com/NVIDIA/k8s-driver-manager/issues/191). |
| 122 | + |
31 | 123 | ## Building the vGPU Manager image |
32 | 124 |
|
33 | 125 | The proprietary vGPU Manager driver must be obtained from NVIDIA and packaged into a container image that the gpu-operator chart pulls — it is not installed from a raw `.run` at runtime. NVIDIA owns this build path; their [`gpu-driver-container`](https://github.com/NVIDIA/gpu-driver-container) repository ships per-OS Dockerfiles under `vgpu-manager/<os>/` and is the source of truth for build args, base images, and supported OS releases. Follow the README in that repository. |
@@ -233,8 +325,8 @@ Other GPU families have analogous tables in the [NVIDIA Virtual GPU Software Doc |
233 | 325 |
|
234 | 326 | | Host OS | passthrough (`default`) | vGPU (`vgpu`) | |
235 | 327 | | --- | --- | --- | |
236 | | -| Ubuntu 24.04 | ✅ supported upstream | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) | |
237 | | -| Ubuntu 22.04 | ✅ | ✅ | |
238 | | -| Ubuntu 20.04 | ✅ | ✅ | |
239 | | -| Ubuntu 26.04 | ⚠️ patch needed in `nvidia-driver` for usr-merge | ⚠️ same patch + own Dockerfile fork | |
240 | | -| Talos Linux | ✅ (open `vfio-pci`) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run`; we tried and the path is blocked | |
| 328 | +| Ubuntu 24.04 | ⚠️ supported upstream, but the host must be clean of any apt-installed NVIDIA driver — see ["Passthrough variant: host preparation"](#passthrough-variant-host-preparation) | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) | |
| 329 | +| Ubuntu 22.04 | ⚠️ same clean-host requirement as 24.04 | ✅ | |
| 330 | +| Ubuntu 20.04 | ⚠️ same clean-host requirement as 24.04 | ✅ | |
| 331 | +| Ubuntu 26.04 | ⚠️ same clean-host requirement as 24.04, plus a `nvidia-driver` patch for usr-merge | ⚠️ same patch + own Dockerfile fork | |
| 332 | +| Talos Linux | ✅ (open `vfio-pci`; the Talos image ships no host NVIDIA stack, so the clean-host check passes trivially) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run`; we tried and the path is blocked | |
0 commit comments