You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ae699f9
Browse filesBrowse the repository at this point in the historyBrowse files
docs(gpu-vgpu): drop the manual KubeVirt patch step
The platform now wires permittedHostDevices and the HostDevices feature
gate into the KubeVirt CR automatically whenever cozystack.gpu-operator
is in bundles.enabledPackages. The KubeVirt configuration section now
describes the auto-wiring contract, the per-variant defaults, the
.gpu.permittedHostDevices / .gpu.replaceDefaults escape hatches, and
calls out the sandbox-device-plugin resourceName slug convention so
operators know where to look when a node's allocatable name does not
match the platform default.
Signed-off-by: Aleksei Sviridkin <[email protected]>
Copy file name to clipboardExpand all lines: docs/gpu-vgpu.md
+65-30Lines changed: 65 additions & 30 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -38,30 +38,30 @@ The proprietary `.run` is the **Linux KVM** variant (not the Ubuntu KVM `.deb`,
38
38
39
39
## Deploying with the vgpu variant
40
40
41
-
Create a `Package`CR pointing at your image coordinates:
41
+
The platform's `iaas` bundle deploys the gpu-operator Package CR when `cozystack.gpu-operator` is in `bundles.enabledPackages` and `bundles.iaas.gpuOperatorVariant: vgpu` is set. The vGPU Manager image is proprietary and not redistributable, so the bundle does not ship a default tag — the operator builds the container per the upstream [`gpu-driver-container`](https://github.com/NVIDIA/gpu-driver-container) recipe and supplies the private-registry coordinates through platform values:
# validator, dcgmExporter, …). The value is a list of strings,
58
+
# not [{name: ...}].
59
+
imagePullSecrets:
60
+
- nvidia-registry-secret
63
61
```
64
62
63
+
The platform forwards `gpu.vgpuManager` into the emitted gpu-operator Package CR's `components.gpu-operator.values.gpu-operator.vgpuManager`, so the bundle handles the variant + image coordinates in one place. If you need to override anything else on the gpu-operator chart (driver, validator, dcgmExporter, custom node selectors), hand-craft a `Package` CR named `cozystack.gpu-operator` with the full `components.gpu-operator.values` block — that takes precedence over the bundle render.
64
+
65
65
The `nvidia-registry-secret` should be a docker-registry Secret created beforehand in `cozy-gpu-operator`.
66
66
67
67
Verify the DaemonSet is running and `nvidia.ko` loads on every GPU node:
@@ -94,24 +94,59 @@ For Pascal–Ampere GPUs (V100, T4, A100, A30) the mdev model still applies. Fli
94
94
95
95
## KubeVirt configuration
96
96
97
-
After [kubevirt#16890](https://github.com/kubevirt/kubevirt/pull/16890), `virt-handler` recognises SR-IOV VFs bound to the `nvidia` driver as candidates whenever a vGPU profile is configured (`current_vgpu_type` ≠ 0). PFs are skipped automatically.
97
+
When `cozystack.gpu-operator` is in `bundles.enabledPackages` (and not also in `bundles.disabledPackages`), the platform mirrors the chosen GPU variant into the `KubeVirt` CR automatically. There is no manual `kubectl patch` step.
98
+
99
+
If you opt out of bundle management and hand-craft a `cozystack.gpu-operator` Package CR directly — typically to apply overrides the bundle does not expose (driver settings, custom node selectors, validator / dcgmExporter tweaks, etc.) — the platform does NOT auto-wire `HostDevices` or `permittedHostDevices` into the KubeVirt CR. In that flow you also hand-craft a `cozystack.kubevirt` Package CR with `components.kubevirt.values.extraFeatureGates: [HostDevices]` and the appropriate `permittedHostDevices` block. The escape-hatch values shape under `.gpu` (below) is intentionally documented in the bundle-managed flow only; the manual Package-CR override path takes precedence over the bundle render whenever both exist.
100
+
101
+
- `developerConfiguration.featureGates`gets `HostDevices` appended (current KubeVirt splits this from the `GPU` gate; the admission webhook rejects `spec.template.spec.domain.devices.hostDevices` without it).
102
+
- `permittedHostDevices.pciHostDevices` is filled from `packages/core/platform/files/gpu-passthrough-defaults.yaml` when `bundles.iaas.gpuOperatorVariant: default` (the package default). The table covers Hopper (H100/H200), Ada Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4), Volta (V100/V100S). All entries carry `externalResourceProvider: true` because the resource names come from `nvidia-sandbox-device-plugin`, not from KubeVirt's in-tree device plugin.
103
+
- `permittedHostDevices.mediatedDevices` and `mediatedDevicesConfiguration` are filled from `packages/core/platform/files/gpu-vgpu-defaults.yaml` when `bundles.iaas.gpuOperatorVariant: vgpu`. The starter set covers Pascal–Ampere mdev profiles (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q, V100D-32C, T4-16Q) — the same family range as the upstream `vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace / Blackwell SR-IOV vGPU is out of scope for the chart's default list; advertise those VFs via the user-override hook below.
104
+
105
+
### Extending or replacing the default table
98
106
99
-
Patch the `KubeVirt` CR to permit the resource:
107
+
The platform exposes three knobs under `.gpu`:
100
108
101
109
```yaml
102
-
spec:
103
-
configuration:
104
-
permittedHostDevices:
105
-
pciHostDevices:
106
-
- pciVendorSelector: "10DE:26B9" # L40S — same device ID for PF and VF
107
-
resourceName: nvidia.com/L40S-24Q
110
+
gpu:
111
+
# Extend the platform defaults with cluster-specific entries. Both list
112
+
# keys are read in both variants: pciHostDevices feeds the passthrough
113
+
# (vfio-pci) path AND the post-kubevirt#16890 SR-IOV vGPU VF path on
114
+
# Ada Lovelace / Blackwell; mediatedDevices feeds the pre-#16890 mdev
115
+
# path on Pascal–Ampere. Both render into the same KubeVirt CR.
116
+
permittedHostDevices:
117
+
pciHostDevices:
118
+
- pciVendorSelector: "10DE:26B9" # L40S, advertised as a VF for SR-IOV vGPU
119
+
resourceName: nvidia.com/L40S-24Q
120
+
# externalResourceProvider is intentionally omitted here: after
# nodeMediatedDeviceTypes) REPLACES the platform default for that key
128
+
# rather than appending. Combine with replaceDefaults: true when both
129
+
# this and permittedHostDevices.mediatedDevices should be wiped together.
130
+
mediatedDevicesConfiguration: {}
131
+
# Wipe the platform defaults entirely and ship only the cluster's
132
+
# curated lists. Useful for non-NVIDIA-only clusters and strict
133
+
# allowlist requirements.
134
+
replaceDefaults: false
108
135
```
109
136
137
+
`replaceDefaults: false` (the default) appends user entries to the NVIDIA defaults. `replaceDefaults: true` drops the NVIDIA table entirely — if you don't then supply your own pciHostDevices / mediatedDevices list, the rendered KubeVirt CR has no `permittedHostDevices` block and the admission webhook will reject every GPU VM.
138
+
139
+
### Notes on `nvidia-sandbox-device-plugin` resource names
140
+
141
+
The `resourceName` strings in `gpu-passthrough-defaults.yaml` follow what `nvidia-sandbox-device-plugin` v25.x generates: a slug of `<arch>_<model>_<form>_<mem>` (e.g. `nvidia.com/GH100_H200_SXM_141GB`). If the plugin version drifts that convention, your node will publish a different name — check with `kubectl describe node <node> | grep nvidia.com/` and override via `.gpu.permittedHostDevices.pciHostDevices` (or wipe the table with `replaceDefaults: true` and curate it yourself). PCI vendor:device IDs themselves are stable across driver versions.
142
+
143
+
### SR-IOV PF vs VF (Ada Lovelace and newer)
144
+
110
145
On L40S (and other Ada-Lovelace cards) the SR-IOV VFs report the same PCI device ID as the PF — `lspci -nn -d 10de:` on the host shows both as `[10de:26b9]`. `virt-handler` distinguishes them by `is-VF + has-vGPU-profile`, so a single `pciVendorSelector` matches the right set. Verify on your specific GPU before assuming this — some other generations split PF/VF IDs.
111
146
112
-
`externalResourceProvider: true` is **not** required here (and should be omitted) — the device plugin lives inside `virt-handler` itself, no external sandbox-device-plugin advertises this resource.
147
+
`externalResourceProvider: true` is **not** required when the resource is advertised by `virt-handler`'s in-tree device plugin (the SR-IOV path post kubevirt#16890). The platform passthrough defaults include the flag because that path is driven by the external sandbox plugin.
113
148
114
-
Verify allocatable capacity:
149
+
### Verifying allocatable capacity
115
150
116
151
```bash
117
152
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{": "}{.status.allocatable.nvidia\.com/L40S-24Q}{"\n"}{end}'
0 commit comments