Skip to content

Commit bf8cda4

Browse files
committed
docs(gpu-vgpu): drop the manual KubeVirt patch step
The platform now wires permittedHostDevices and the HostDevices feature gate into the KubeVirt CR automatically whenever cozystack.gpu-operator is in bundles.enabledPackages. The KubeVirt configuration section now describes the auto-wiring contract, the per-variant defaults, the .gpu.permittedHostDevices / .gpu.replaceDefaults escape hatches, and calls out the sandbox-device-plugin resourceName slug convention so operators know where to look when a node's allocatable name does not match the platform default. Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 8c4dd1e commit bf8cda4

1 file changed

Lines changed: 65 additions & 30 deletions

File tree

‎docs/gpu-vgpu.md‎

Lines changed: 65 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -38,30 +38,30 @@ The proprietary `.run` is the **Linux KVM** variant (not the Ubuntu KVM `.deb`,
3838
3939
## Deploying with the vgpu variant
4040

41-
Create a `Package` CR pointing at your image coordinates:
41+
The platform's `iaas` bundle deploys the gpu-operator Package CR when `cozystack.gpu-operator` is in `bundles.enabledPackages` and `bundles.iaas.gpuOperatorVariant: vgpu` is set. The vGPU Manager image is proprietary and not redistributable, so the bundle does not ship a default tag — the operator builds the container per the upstream [`gpu-driver-container`](https://github.com/NVIDIA/gpu-driver-container) recipe and supplies the private-registry coordinates through platform values:
4242

4343
```yaml
44-
apiVersion: cozystack.io/v1alpha1
45-
kind: Package
46-
metadata:
47-
name: cozystack.gpu-operator
48-
spec:
49-
variant: vgpu
50-
components:
51-
gpu-operator:
52-
values:
53-
gpu-operator:
54-
vgpuManager:
55-
repository: registry.example.com/nvidia
56-
image: vgpu-manager
57-
version: "595.58.02-ubuntu24.04"
58-
# imagePullSecrets lives per-component (vgpuManager,
59-
# driver, validator, dcgmExporter, …). The value is a
60-
# list of strings, not [{name: ...}].
61-
imagePullSecrets:
62-
- nvidia-registry-secret
44+
bundles:
45+
iaas:
46+
enabled: true
47+
gpuOperatorVariant: vgpu
48+
enabledPackages:
49+
- cozystack.gpu-operator
50+
51+
gpu:
52+
vgpuManager:
53+
repository: registry.example.com/nvidia
54+
image: vgpu-manager
55+
version: "595.58.02-ubuntu24.04"
56+
# imagePullSecrets lives per-component (vgpuManager, driver,
57+
# validator, dcgmExporter, …). The value is a list of strings,
58+
# not [{name: ...}].
59+
imagePullSecrets:
60+
- nvidia-registry-secret
6361
```
6462
63+
The platform forwards `gpu.vgpuManager` into the emitted gpu-operator Package CR's `components.gpu-operator.values.gpu-operator.vgpuManager`, so the bundle handles the variant + image coordinates in one place. If you need to override anything else on the gpu-operator chart (driver, validator, dcgmExporter, custom node selectors), hand-craft a `Package` CR named `cozystack.gpu-operator` with the full `components.gpu-operator.values` block — that takes precedence over the bundle render.
64+
6565
The `nvidia-registry-secret` should be a docker-registry Secret created beforehand in `cozy-gpu-operator`.
6666

6767
Verify the DaemonSet is running and `nvidia.ko` loads on every GPU node:
@@ -94,24 +94,59 @@ For Pascal–Ampere GPUs (V100, T4, A100, A30) the mdev model still applies. Fli
9494

9595
## KubeVirt configuration
9696

97-
After [kubevirt#16890](https://github.com/kubevirt/kubevirt/pull/16890), `virt-handler` recognises SR-IOV VFs bound to the `nvidia` driver as candidates whenever a vGPU profile is configured (`current_vgpu_type` ≠ 0). PFs are skipped automatically.
97+
When `cozystack.gpu-operator` is in `bundles.enabledPackages` (and not also in `bundles.disabledPackages`), the platform mirrors the chosen GPU variant into the `KubeVirt` CR automatically. There is no manual `kubectl patch` step.
98+
99+
If you opt out of bundle management and hand-craft a `cozystack.gpu-operator` Package CR directly — typically to apply overrides the bundle does not expose (driver settings, custom node selectors, validator / dcgmExporter tweaks, etc.) — the platform does NOT auto-wire `HostDevices` or `permittedHostDevices` into the KubeVirt CR. In that flow you also hand-craft a `cozystack.kubevirt` Package CR with `components.kubevirt.values.extraFeatureGates: [HostDevices]` and the appropriate `permittedHostDevices` block. The escape-hatch values shape under `.gpu` (below) is intentionally documented in the bundle-managed flow only; the manual Package-CR override path takes precedence over the bundle render whenever both exist.
100+
101+
- `developerConfiguration.featureGates` gets `HostDevices` appended (current KubeVirt splits this from the `GPU` gate; the admission webhook rejects `spec.template.spec.domain.devices.hostDevices` without it).
102+
- `permittedHostDevices.pciHostDevices` is filled from `packages/core/platform/files/gpu-passthrough-defaults.yaml` when `bundles.iaas.gpuOperatorVariant: default` (the package default). The table covers Hopper (H100/H200), Ada Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4), Volta (V100/V100S). All entries carry `externalResourceProvider: true` because the resource names come from `nvidia-sandbox-device-plugin`, not from KubeVirt's in-tree device plugin.
103+
- `permittedHostDevices.mediatedDevices` and `mediatedDevicesConfiguration` are filled from `packages/core/platform/files/gpu-vgpu-defaults.yaml` when `bundles.iaas.gpuOperatorVariant: vgpu`. The starter set covers Pascal–Ampere mdev profiles (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q, V100D-32C, T4-16Q) — the same family range as the upstream `vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace / Blackwell SR-IOV vGPU is out of scope for the chart's default list; advertise those VFs via the user-override hook below.
104+
105+
### Extending or replacing the default table
98106

99-
Patch the `KubeVirt` CR to permit the resource:
107+
The platform exposes three knobs under `.gpu`:
100108

101109
```yaml
102-
spec:
103-
configuration:
104-
permittedHostDevices:
105-
pciHostDevices:
106-
- pciVendorSelector: "10DE:26B9" # L40S — same device ID for PF and VF
107-
resourceName: nvidia.com/L40S-24Q
110+
gpu:
111+
# Extend the platform defaults with cluster-specific entries. Both list
112+
# keys are read in both variants: pciHostDevices feeds the passthrough
113+
# (vfio-pci) path AND the post-kubevirt#16890 SR-IOV vGPU VF path on
114+
# Ada Lovelace / Blackwell; mediatedDevices feeds the pre-#16890 mdev
115+
# path on Pascal–Ampere. Both render into the same KubeVirt CR.
116+
permittedHostDevices:
117+
pciHostDevices:
118+
- pciVendorSelector: "10DE:26B9" # L40S, advertised as a VF for SR-IOV vGPU
119+
resourceName: nvidia.com/L40S-24Q
120+
# externalResourceProvider is intentionally omitted here: after
121+
# kubevirt/kubevirt#16890, virt-handler's in-tree device plugin
122+
# advertises the resource directly, no sandbox plugin in the loop.
123+
mediatedDevices: []
124+
# mediatedDevicesConfiguration drives the per-node vGPU profile
125+
# auto-creation loop in vgpu mode. Helm's mergeOverwrite applies — a
126+
# user-supplied top-level key (mediatedDeviceTypes,
127+
# nodeMediatedDeviceTypes) REPLACES the platform default for that key
128+
# rather than appending. Combine with replaceDefaults: true when both
129+
# this and permittedHostDevices.mediatedDevices should be wiped together.
130+
mediatedDevicesConfiguration: {}
131+
# Wipe the platform defaults entirely and ship only the cluster's
132+
# curated lists. Useful for non-NVIDIA-only clusters and strict
133+
# allowlist requirements.
134+
replaceDefaults: false
108135
```
109136

137+
`replaceDefaults: false` (the default) appends user entries to the NVIDIA defaults. `replaceDefaults: true` drops the NVIDIA table entirely — if you don't then supply your own pciHostDevices / mediatedDevices list, the rendered KubeVirt CR has no `permittedHostDevices` block and the admission webhook will reject every GPU VM.
138+
139+
### Notes on `nvidia-sandbox-device-plugin` resource names
140+
141+
The `resourceName` strings in `gpu-passthrough-defaults.yaml` follow what `nvidia-sandbox-device-plugin` v25.x generates: a slug of `<arch>_<model>_<form>_<mem>` (e.g. `nvidia.com/GH100_H200_SXM_141GB`). If the plugin version drifts that convention, your node will publish a different name — check with `kubectl describe node <node> | grep nvidia.com/` and override via `.gpu.permittedHostDevices.pciHostDevices` (or wipe the table with `replaceDefaults: true` and curate it yourself). PCI vendor:device IDs themselves are stable across driver versions.
142+
143+
### SR-IOV PF vs VF (Ada Lovelace and newer)
144+
110145
On L40S (and other Ada-Lovelace cards) the SR-IOV VFs report the same PCI device ID as the PF — `lspci -nn -d 10de:` on the host shows both as `[10de:26b9]`. `virt-handler` distinguishes them by `is-VF + has-vGPU-profile`, so a single `pciVendorSelector` matches the right set. Verify on your specific GPU before assuming this — some other generations split PF/VF IDs.
111146

112-
`externalResourceProvider: true` is **not** required here (and should be omitted) — the device plugin lives inside `virt-handler` itself, no external sandbox-device-plugin advertises this resource.
147+
`externalResourceProvider: true` is **not** required when the resource is advertised by `virt-handler`'s in-tree device plugin (the SR-IOV path post kubevirt#16890). The platform passthrough defaults include the flag because that path is driven by the external sandbox plugin.
113148

114-
Verify allocatable capacity:
149+
### Verifying allocatable capacity
115150

116151
```bash
117152
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable.nvidia\.com/L40S-24Q}{"\n"}{end}'

0 commit comments

Comments
 (0)