Skip to content

Commit f5e48b1

Browse files
authored
docs(gpu): drop manual KubeVirt patch step now that the platform auto-wires permittedHostDevices (#556)
Companion to [cozystack/cozystack#2768](cozystack/cozystack#2768). Rewrites step 2 of the GPU Passthrough guide. Until now the page instructed operators to run `kubectl edit kubevirt -n cozy-kubevirt` and hand-paste a `permittedHostDevices.pciHostDevices` block — that is the friction that ticket #2765 asked the platform to remove. With cozystack/cozystack#2768 landed, the bundle mirrors the chosen GPU variant into the KubeVirt CR automatically: `HostDevices` is appended to the feature-gate list and a starter NVIDIA `pciHostDevices` table (Hopper, Ada Lovelace, Ampere, Turing, Volta) is rendered alongside the operator's `.gpu.permittedHostDevices` extensions. The new step 2 documents: - The contract — what the platform auto-injects and why (`HostDevices` gate, NVIDIA default table, `externalResourceProvider: true` semantics). - How to verify (`kubectl -n cozy-kubevirt get kubevirt kubevirt -o yaml | yq ...`). - The escape hatch — `gpu.replaceDefaults`, `gpu.permittedHostDevices.pciHostDevices`, plus the consequence of `replaceDefaults: true` with an empty list (no admittable GPU VMs). - The manual Package-CR override path — when an operator hand-crafts `cozystack.gpu-operator` outside the bundle for advanced overrides, they also hand-craft `cozystack.kubevirt` with the matching `extraFeatureGates` / `permittedHostDevices`. The manual override takes precedence over the bundle render. Only `next/virtualization/gpu.md` is touched. The released doc versions (`v1.4` and earlier) describe earlier Cozystack releases that still require the manual `kubectl edit`, and stay as-is. ### Release note ```release-note docs(gpu): the GPU Passthrough guide no longer instructs operators to manually patch the KubeVirt CR — Cozystack now auto-wires the HostDevices feature gate and a starter NVIDIA permittedHostDevices table whenever cozystack.gpu-operator is enabled in bundles.enabledPackages. Operators extend or replace the defaults via .gpu.permittedHostDevices and .gpu.replaceDefaults. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated GPU virtualization guide to remove a manual edit step and explain automated configuration when the GPU package is enabled. * Added verification commands to inspect rendered GPU settings and detailed how to extend or fully replace NVIDIA defaults via package values. * Included upgrade guidance for prior manual setups and documented a manual override path for non-bundled deployments. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 417069b + 77ff959 commit f5e48b1

1 file changed

Lines changed: 84 additions & 18 deletions

File tree

  • content/en/docs/next/virtualization

‎content/en/docs/next/virtualization/gpu.md‎

Lines changed: 84 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -100,32 +100,98 @@ Allocatable:
100100
For example, the database entry for A10 reads `2236 GA102GL [A10]`, which results in a resource name `nvidia.com/GA102GL_A10`.
101101
{{% /alert %}}
102102

103-
## 2. Update the KubeVirt Custom Resource
103+
## 2. KubeVirt is wired automatically
104104

105-
Next, we will update the KubeVirt Custom Resource, as documented in the
106-
[KubeVirt user guide](https://kubevirt.io/user-guide/virtual_machines/host-devices/#listing-permitted-devices),
107-
so that the passthrough GPUs are permitted and can be requested by a KubeVirt VM.
105+
When `cozystack.gpu-operator` is in `bundles.enabledPackages`, Cozystack mirrors the chosen GPU variant into the `KubeVirt` Custom Resource for you. There is no `kubectl edit kubevirt` step.
108106

109-
Adjust the `pciVendorSelector` and `resourceName` values to match your specific GPU model.
110-
Setting `externalResourceProvider=true` indicates that this resource is provided by an external device plugin,
111-
in this case the `sandbox-device-plugin` which is deployed by the Operator.
107+
Specifically, the platform injects:
108+
109+
- `HostDevices` into `spec.configuration.developerConfiguration.featureGates` (current KubeVirt splits this from the `GPU` gate; the admission webhook rejects `domain.devices.hostDevices` without it).
110+
- A starter `spec.configuration.permittedHostDevices.pciHostDevices` table (rendered in the default `gpuOperatorVariant: default` — vfio-pci passthrough) covering common NVIDIA datacenter GPUs — Hopper (H100, H200), Ada Lovelace (L4, L40, L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4), Volta (V100, V100S). PCI vendor:device pairs are stable; each `resourceName` slug is whatever `nvidia-sandbox-device-plugin` derives mechanically from the card's PCI-IDs database name — uppercase the name, turn `/`, `.` and whitespace into `_`, then strip the surrounding `[` / `]`. The slug therefore carries every token the PCI-IDs string holds (the `GL` die suffix, the `Tesla` brand on Turing/Volta, the form factor, the memory size), not a tidy `<arch>_<model>`: `TU104GL [Tesla T4]` becomes `nvidia.com/TU104GL_TESLA_T4`, `GA100GL [A30 PCIe]` becomes `nvidia.com/GA100GL_A30_PCIE`, and the H200 SXM becomes `nvidia.com/GH100_H200_SXM_141GB`. Confirm the exact strings your nodes advertise with `kubectl describe node <node> | grep nvidia.com/`. `externalResourceProvider: true` is set on every entry because the resources are advertised by the sandbox plugin, not by KubeVirt's in-tree device plugin.
111+
112+
Verify the resulting CR:
112113

113114
```bash
114-
kubectl edit kubevirt -n cozy-kubevirt
115+
kubectl -n cozy-kubevirt get kubevirt kubevirt -o json \
116+
| jq '.spec.configuration | {featureGates: .developerConfiguration.featureGates, permittedHostDevices: .permittedHostDevices}'
115117
```
116-
example config:
118+
119+
{{% alert color="info" %}}
120+
121+
**My GPU isn't in the default table — where's the old `kubectl edit kubevirt` step?** It is gone on purpose. `permittedHostDevices` is now owned by the chart template and reconciled from platform values, so any hand edit to the live CR is reverted on the next Flux/Helm reconcile. Add your card through `.gpu.permittedHostDevices` instead — see [Extending or replacing the NVIDIA defaults](#extending-or-replacing-the-nvidia-defaults) below. If you are upgrading from a release where you hand-edited the CR, follow [Upgrading from a hand-edited KubeVirt CR](#upgrading-from-a-hand-edited-kubevirt-cr) first.
122+
123+
{{% /alert %}}
124+
125+
### Extending or replacing the NVIDIA defaults
126+
127+
If your cluster ships a GPU not in the default table, or your `nvidia-sandbox-device-plugin` version emits a different `resourceName` (check with `kubectl describe node <node> | grep nvidia.com/`), extend the defaults via platform values:
128+
117129
```yaml
118-
...
119-
spec:
120-
configuration:
121-
permittedHostDevices:
122-
pciHostDevices:
123-
- externalResourceProvider: true
124-
pciVendorSelector: 10DE:2236
125-
resourceName: nvidia.com/GA102GL_A10
126-
...
130+
# Platform Package values
131+
gpu:
132+
# Append (default) — your entries land alongside the NVIDIA table.
133+
# Set to true to drop the NVIDIA table entirely (useful for non-NVIDIA-only
134+
# clusters or strict allowlists). With replaceDefaults: true and an empty
135+
# list below, the rendered CR carries no permittedHostDevices block at all
136+
# and the admission webhook rejects every GPU VM — supply your own list.
137+
replaceDefaults: false
138+
permittedHostDevices:
139+
pciHostDevices:
140+
- pciVendorSelector: "10DE:2236"
141+
resourceName: nvidia.com/GA102GL_A10
142+
externalResourceProvider: true
143+
```
144+
145+
To **re-point** a card already in the NVIDIA table (for example to give `10DE:1EB8` a different `resourceName`), do not append a second entry for the same `pciVendorSelector` — both entries are rendered and KubeVirt resolves the duplicated selector non-deterministically. Set `replaceDefaults: true` and supply the full list you want instead.
146+
147+
### Upgrading from a hand-edited KubeVirt CR
148+
149+
Earlier Cozystack releases left `spec.configuration.permittedHostDevices` for operators to hand-edit (`kubectl edit kubevirt`). The bundle now **owns** that field: the first reconcile after the upgrade replaces your manual entries with the rendered NVIDIA default table.
150+
151+
Before upgrading:
152+
153+
1. Dump your current entries:
154+
155+
```bash
156+
kubectl -n cozy-kubevirt get kubevirt kubevirt -o json \
157+
| jq '.spec.configuration.permittedHostDevices'
158+
```
159+
160+
2. Move any custom entries into the Platform Package values under `.gpu.permittedHostDevices` (set `.gpu.replaceDefaults: true` if you want only your own list instead of appending to the NVIDIA defaults).
161+
162+
3. Verify every `resourceName` against what your nodes actually advertise. The default table carries the slug `nvidia-sandbox-device-plugin` generates from each card's PCI-IDs name (uppercased, e.g. `nvidia.com/TU104GL_TESLA_T4` for a Tesla T4), but a different plugin build or PCI-IDs snapshot can emit a different string:
163+
164+
```bash
165+
kubectl describe node <node> | grep nvidia.com/
166+
```
167+
168+
A `resourceName` mismatch is silent until a GPU VM restarts or migrates, at which point the admission webhook rejects it.
169+
170+
### Manual Package-CR override path
171+
172+
If you opt out of bundle management and hand-craft a `cozystack.gpu-operator` Package CR directly (to apply overrides the bundle does not expose — driver settings, custom node selectors, validator / dcgmExporter tweaks), the platform does NOT auto-wire `HostDevices` or `permittedHostDevices` into the KubeVirt CR. In that flow, mirror the bundle behaviour by also creating a `cozystack.kubevirt` Package CR that carries `extraFeatureGates` and the matching `permittedHostDevices` block under `spec.components.kubevirt.values` (a cozystack `Package` always nests component values under `spec.components.<name>.values`, never a top-level `spec.values`):
173+
174+
```yaml
175+
apiVersion: cozystack.io/v1alpha1
176+
kind: Package
177+
metadata:
178+
name: cozystack.kubevirt
179+
spec:
180+
variant: default
181+
components:
182+
kubevirt:
183+
values:
184+
extraFeatureGates:
185+
- HostDevices
186+
permittedHostDevices:
187+
pciHostDevices:
188+
- pciVendorSelector: "10DE:2236"
189+
resourceName: nvidia.com/GA102GL_A10
190+
externalResourceProvider: true
127191
```
128192
193+
The manual Package-CR override path takes precedence over the bundle render whenever both exist.
194+
129195
## 3. Create a Virtual Machine
130196
131197
We are now ready to create a VM.

0 commit comments

Comments
 (0)