|
1 | | -# GPU Operator: vGPU Support |
| 1 | +# GPU Operator: passthrough and vGPU |
2 | 2 |
|
3 | 3 | This document describes how to configure the GPU Operator package with NVIDIA vGPU support so that a single physical GPU can be sliced and shared across multiple virtual machines. |
4 | 4 |
|
@@ -29,6 +29,129 @@ The `gpu-operator` package exposes three variants. This document is vGPU-focused |
29 | 29 | - **`vgpu`** — SR-IOV vGPU mode. One physical GPU is sliced into multiple VFs, each VF bound to a vGPU profile that the guest sees as its own GPU. |
30 | 30 | - **`container`** — containerized GPU workloads (CUDA pods, ML training) via the standard NVIDIA device plugin on hosts that already provide both the NVIDIA driver and `nvidia-container-toolkit` (the typical apt-installed Ubuntu/Debian shape). Sandbox workloads are off, `devicePlugin` is on, `driver` / `toolkit` / `vfioManager` / `cdi` are off so that the operator does not fight the host install. Orthogonal to the two VM variants — it does not pass GPUs to KubeVirt VMs. Note that `apt install nvidia-container-toolkit` installs binaries only — it does not configure containerd. Because this variant disables the operator's toolkit component (which would normally do that wiring), the host must additionally have run `nvidia-ctk runtime configure --runtime=containerd` (followed by a containerd restart) and exposed the `nvidia` runtime as the default or via a RuntimeClass before the device plugin can serve GPUs. |
31 | 31 |
|
| 32 | +## Passthrough variant: host preparation |
| 33 | + |
| 34 | +**Last verified:** 2026-05-28 against `cozystack.gpu-operator` chart `gpu-operator` v26.3.1 + image `nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0` (distroless image with a Go ELF binary `/usr/bin/driver-manager`, entrypoint `["driver-manager", "preflight_check"]`; verified via `crane export … | tar -x usr/bin/driver-manager` and `strings` over the resulting binary) + Ubuntu 24.04 host with `nvidia-driver-580-open` 580.82.07 pre-installed. Tracks [#2763](https://github.com/cozystack/cozystack/issues/2763). |
| 35 | + |
| 36 | +The `default` (passthrough) variant assumes the GPU is **owned by the host kernel's `vfio-pci` driver and nothing else**. |
| 37 | + |
| 38 | +**How the operator detects a pre-installed host driver.** The chart's `vfio-manager` DaemonSet runs the upstream NVIDIA `k8s-driver-manager` init container with the `uninstall_driver` subcommand; that path calls the Go method `(*DriverManager).isHostDriver`, which runs `chroot /host nvidia-smi --query-gpu=driver_version --format=csv,noheader` and treats any non-empty stdout as "host driver present" (file-existence is not pre-tested — if `nvidia-smi` is missing, the chroot exec errors and `isHostDriver` returns false, which is the intended path on a clean host: the operator then proceeds with the uninstall flow and `vfio-manager` binds `vfio-pci` as designed). On a positive detection the binary logs `Host driver detected: <ver>`, labels the node `nvidia.com/gpu.deploy.driver=pre-installed`, exits, and `nvidia-sandbox-validator` then crashloops with `device not bound to 'vfio-pci'`. |
| 39 | + |
| 40 | +**`FORCE_REINSTALL` does not bypass this.** `k8s-driver-manager` v0.10.0 exposes a `FORCE_REINSTALL` / `--force-reinstall` env+flag pair (visible in the binary's strings table), but it gates a later "same-config already loaded" branch inside `uninstallDriver`, not the `isHostDriver` short-circuit at the top — operators who set it and expect a bypass will report a false bug. There is currently no opt-out for the `isHostDriver` guard itself. |
| 41 | + |
| 42 | +**Current mitigation.** The only mitigation available on `main` today is to remove the host NVIDIA stack before enabling the variant (the "Clean-host workaround" section below). |
| 43 | + |
| 44 | +**Future alternatives and scope.** An upcoming `container` variant of `cozystack.gpu-operator` is tracked in [#2766](https://github.com/cozystack/cozystack/pull/2766) — once merged it will give operators a no-purge path that keeps the host driver and exposes GPUs to pods rather than VMs — but it is not part of the current package and is **not** a usable workaround until it lands. Talos is unaffected because the Talos image ships only the `vfio-pci` extension; this section applies to Linux distributions where you installed the host driver yourself (typically Ubuntu / Debian / RHEL with `apt install nvidia-driver-*` or equivalent). See also [`packages/system/gpu-operator/examples/README.md`](../packages/system/gpu-operator/examples/README.md) for the native-pod-workload reference flow that predates the `container` variant. |
| 45 | + |
| 46 | +### Symptom |
| 47 | + |
| 48 | +The first thing you see is `kubectl get pods -n cozy-gpu-operator` showing `nvidia-vfio-manager-*` stuck in `Init:Error` / `Init:CrashLoopBackOff` — the init container exits non-zero after detecting the host driver, so kubelet keeps restarting it. |
| 49 | + |
| 50 | +Its log shows it skipped the bind step: |
| 51 | + |
| 52 | +```text |
| 53 | +Host driver detected: 580.82.07 |
| 54 | +NVIDIA GPU driver is already pre-installed on the node, |
| 55 | + disabling the containerized driver |
| 56 | +Labeling node <NODE> with nvidia.com/gpu.deploy.driver=pre-installed |
| 57 | +``` |
| 58 | + |
| 59 | +`nvidia-sandbox-validator` then crashloops: |
| 60 | + |
| 61 | +```text |
| 62 | +Error: error validating vfio-pci driver installation: |
| 63 | + device not bound to 'vfio-pci'; device: 0000:18:00.0 driver: 'nvidia' |
| 64 | +``` |
| 65 | + |
| 66 | +`lspci -nnk -d 10de:` still shows `Kernel driver in use: nvidia` on every target GPU, the node carries `nvidia.com/gpu.deploy.driver=pre-installed`, and `kubectl get node <NODE> -o json | jq '.status.allocatable | with_entries(select(.key | startswith("nvidia.com/")))'` reports `{}` — no GPU resource was registered. |
| 67 | + |
| 68 | +### Clean-host workaround |
| 69 | + |
| 70 | +Purge the NVIDIA host stack and blacklist the kernel modules so the host never re-claims the GPU. Two pitfalls to avoid: `apt autoremove` is dangerous here because the `nvidia-` prefix is shared with NVIDIA DOCA / Mellanox / InfiniBand userspace, so a blanket autoremove can take RDMA out on a converged GPU + RDMA host; and a hardcoded `apt purge 'nvidia-*' 'cuda-*'` pattern list is fragile — `apt` treats `*` as a cache-wide regex and **aborts the entire transaction, purging nothing**, if any pattern matches nothing in the cache (e.g. `cuda-*` on a host without NVIDIA's CUDA repo). Build the list from what is actually installed instead: |
| 71 | + |
| 72 | +```bash |
| 73 | +# dpkg-query patterns are true globs over INSTALLED packages: no |
| 74 | +# zero-match abort (unlike apt's cache-wide regex) and no accidental |
| 75 | +# substring over-match. List first, then review before purging. |
| 76 | +dpkg-query -W -f '${Package}\n' 'nvidia-*' 'libnvidia-*' 'cuda-*' 2>/dev/null |
| 77 | +``` |
| 78 | + |
| 79 | +Review the list before purging: |
| 80 | + |
| 81 | +- `nvidia-dkms-*` and `nvidia-kernel-*` are the load-bearing kernel pieces — without removing them DKMS rebuilds `nvidia.ko` on the next reboot and the blacklist below is bypassed by any explicit `modprobe`. |
| 82 | +- On a **converged GPU + RDMA host**, drop any `libnvidia-*` that belong to NVIDIA DOCA / Mellanox OFED — purging those breaks RDMA. |
| 83 | +- If the list is **empty**, the driver was installed with NVIDIA's `.run` installer rather than apt — run `sudo nvidia-uninstall` instead of the purge below. |
| 84 | + |
| 85 | +Then purge the reviewed list, blacklist the modules, and rebuild the initramfs: |
| 86 | + |
| 87 | +```bash |
| 88 | +# Replace with the packages you kept from the list above. |
| 89 | +sudo apt purge nvidia-driver-580-open nvidia-dkms-580-open <...> |
| 90 | + |
| 91 | +sudo tee /etc/modprobe.d/blacklist-nvidia.conf > /dev/null <<'EOF' |
| 92 | +blacklist nouveau |
| 93 | +blacklist nvidia |
| 94 | +blacklist nvidia_drm |
| 95 | +blacklist nvidia_modeset |
| 96 | +blacklist nvidia_uvm |
| 97 | +blacklist nvidia_peermem |
| 98 | +EOF |
| 99 | + |
| 100 | +# -k all rebuilds EVERY installed kernel's initramfs (plain -u touches |
| 101 | +# only the running kernel), so a just-upgraded kernel also boots with |
| 102 | +# the blacklist and cannot re-claim the GPU. |
| 103 | +sudo update-initramfs -u -k all |
| 104 | +sudo reboot |
| 105 | +``` |
| 106 | + |
| 107 | +After the reboot, confirm the host is clean. The init container exits non-zero after detecting a host driver, so both DaemonSet pods sit in `Init:CrashLoopBackOff` and **will retry on their own** — deleting them just skips the up-to-5-minute backoff window: |
| 108 | + |
| 109 | +```bash |
| 110 | +# Should print nothing. |
| 111 | +lsmod | grep -E '^(nvidia|nouveau)' |
| 112 | + |
| 113 | +# command -v matches what isHostDriver does (PATH lookup inside the |
| 114 | +# chroot), so it also catches /usr/local/bin/nvidia-smi left by a .run |
| 115 | +# or CUDA-toolkit install. Prints "ok: gone" on success. |
| 116 | +command -v nvidia-smi >/dev/null && echo "STILL PRESENT — purge incomplete" || echo "ok: gone" |
| 117 | + |
| 118 | +# A leftover DKMS module rebuilds nvidia.ko on the next kernel update |
| 119 | +# and an explicit modprobe bypasses the blacklist — should print nothing. |
| 120 | +dkms status | grep -i nvidia |
| 121 | + |
| 122 | +# Skip the CrashLoopBackOff backoff window by deleting the stuck pods. |
| 123 | +# The DaemonSet labels are operator-managed and not stable across |
| 124 | +# gpu-operator versions, so delete by name pattern — anchored so the |
| 125 | +# match is the two DaemonSets and nothing else. |
| 126 | +kubectl -n cozy-gpu-operator get pods -o name \ |
| 127 | + | grep -E '^pod/(nvidia-vfio-manager|nvidia-sandbox-validator)-' \ |
| 128 | + | xargs -r kubectl -n cozy-gpu-operator delete |
| 129 | +``` |
| 130 | + |
| 131 | +Within a couple of minutes `vfio-manager` should bind every target GPU to `vfio-pci` and the node's `allocatable` will gain the registered resource: |
| 132 | + |
| 133 | +```bash |
| 134 | +lspci -nnk -d 10de: | grep 'Kernel driver in use' |
| 135 | +# Kernel driver in use: vfio-pci |
| 136 | + |
| 137 | +kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable}{"\n"}{end}' | grep nvidia.com |
| 138 | +``` |
| 139 | + |
| 140 | +One label is **not** cleaned up automatically. The init container set `nvidia.com/gpu.deploy.driver=pre-installed`, and the operator's success path restores only the operand labels — it never resets this one (`rescheduleGPUOperatorComponents` in `k8s-driver-manager` v0.10.0 touches the validator / toolkit / device-plugin / etc. labels, not `deploy.driver`). So the node keeps `pre-installed` indefinitely: it is the same label the Symptom section uses as evidence, and it would disable the containerized-driver DaemonSet if the node later switches to the `container` workload. Clear it once recovery is confirmed: |
| 141 | + |
| 142 | +```bash |
| 143 | +# Nothing on the success path resets this label; clear it so it does not |
| 144 | +# mislead future debugging or block a later container-workload switch. |
| 145 | +kubectl label node <NODE> nvidia.com/gpu.deploy.driver- |
| 146 | +``` |
| 147 | + |
| 148 | +### Known limitation |
| 149 | + |
| 150 | +The skip-on-pre-installed behavior lives in the upstream [`NVIDIA/k8s-driver-manager`](https://github.com/NVIDIA/k8s-driver-manager) Go binary at `cmd/driver-manager/main.go`: the method `(*DriverManager).isHostDriver` is called from `(*DriverManager).uninstallDriver` and has no in-band opt-out in `:v0.10.0`. `FORCE_REINSTALL` exists as an env / CLI flag in the same binary but gates the later "same-config already loaded" branch inside `uninstallDriver`, not the `isHostDriver` short-circuit. Hosts that need to keep the NVIDIA host driver installed for non-Kubernetes workloads cannot currently share the same GPU with the passthrough variant. Two future mitigation paths are in flight; **neither is available today**: |
| 151 | + |
| 152 | +- **`container` variant — not yet merged** — [#2766](https://github.com/cozystack/cozystack/pull/2766) adds a third variant of `cozystack.gpu-operator` that targets the apt-installed-driver host shape and exposes GPUs to pods (not VMs) without unbinding the host driver. Once it lands, operators on hosts where the purge workaround is unacceptable will have a path that does not require touching the host driver. |
| 153 | +- **Upstream override** — an env-var override of `isHostDriver` is the only structural fix that would let the passthrough variant coexist with a host driver. Requested in [NVIDIA/k8s-driver-manager#191](https://github.com/NVIDIA/k8s-driver-manager/issues/191). |
| 154 | + |
32 | 155 | ## Building the vGPU Manager image |
33 | 156 |
|
34 | 157 | The proprietary vGPU Manager driver must be obtained from NVIDIA and packaged into a container image that the gpu-operator chart pulls — it is not installed from a raw `.run` at runtime. NVIDIA owns this build path; their [`gpu-driver-container`](https://github.com/NVIDIA/gpu-driver-container) repository ships per-OS Dockerfiles under `vgpu-manager/<os>/` and is the source of truth for build args, base images, and supported OS releases. Follow the README in that repository. |
@@ -266,8 +389,8 @@ The `container` variant column assumes the host already ships the NVIDIA driver |
266 | 389 |
|
267 | 390 | | Host OS | passthrough (`default`) | vGPU (`vgpu`) | container (`container`) | |
268 | 391 | | --- | --- | --- | --- | |
269 | | -| Ubuntu 24.04 | ✅ supported upstream | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) | ✅ apt-installed driver + nvidia-container-toolkit | |
270 | | -| Ubuntu 22.04 | ✅ | ✅ | ✅ | |
271 | | -| Ubuntu 20.04 | ✅ | ✅ | ✅ | |
272 | | -| Ubuntu 26.04 | ⚠️ patch needed in `nvidia-driver` for usr-merge | ⚠️ same patch + own Dockerfile fork | ✅ | |
273 | | -| Talos Linux | ✅ (open `vfio-pci`) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run`; we tried and the path is blocked | ⚠️ host driver lands in a non-standard prefix — use `examples/values-native-talos.yaml` (compat DaemonSet + `hostPaths.driverInstallDir` override) as a starting point instead | |
| 392 | +| Ubuntu 24.04 | ⚠️ supported upstream, but the host must be clean of any apt-installed NVIDIA driver — see ["Passthrough variant: host preparation"](#passthrough-variant-host-preparation) | ✅ supported upstream (`vgpu-manager/ubuntu24.04`) | ✅ apt-installed driver + nvidia-container-toolkit | |
| 393 | +| Ubuntu 22.04 | ⚠️ same clean-host requirement as 24.04 | ✅ | ✅ | |
| 394 | +| Ubuntu 20.04 | ⚠️ same clean-host requirement as 24.04 | ✅ | ✅ | |
| 395 | +| Ubuntu 26.04 | ⚠️ same clean-host requirement as 24.04, plus a `nvidia-driver` patch for usr-merge (details pending) | ⚠️ same patch + own Dockerfile fork | ✅ | |
| 396 | +| Talos Linux | ✅ (open `vfio-pci`; the Talos image ships no host NVIDIA stack, so the clean-host check passes trivially) | ❌ NVIDIA does not grant redistribution rights for the proprietary `.run`; we tried and the path is blocked | ⚠️ host driver lands in a non-standard prefix — use `examples/values-native-talos.yaml` (compat DaemonSet + `hostPaths.driverInstallDir` override) as a starting point instead | |
0 commit comments