You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f5e48b1
Browse filesBrowse the repository at this point in the historyBrowse files
docs(gpu): drop manual KubeVirt patch step now that the platform auto-wires permittedHostDevices (#556)
Companion to
[cozystack/cozystack#2768](cozystack/cozystack#2768).
Rewrites step 2 of the GPU Passthrough guide. Until now the page
instructed operators to run `kubectl edit kubevirt -n cozy-kubevirt` and
hand-paste a `permittedHostDevices.pciHostDevices` block — that is the
friction that ticket #2765 asked the platform to remove. With
cozystack/cozystack#2768 landed, the bundle mirrors the chosen GPU
variant into the KubeVirt CR automatically: `HostDevices` is appended to
the feature-gate list and a starter NVIDIA `pciHostDevices` table
(Hopper, Ada Lovelace, Ampere, Turing, Volta) is rendered alongside the
operator's `.gpu.permittedHostDevices` extensions.
The new step 2 documents:
- The contract — what the platform auto-injects and why (`HostDevices`
gate, NVIDIA default table, `externalResourceProvider: true` semantics).
- How to verify (`kubectl -n cozy-kubevirt get kubevirt kubevirt -o yaml
| yq ...`).
- The escape hatch — `gpu.replaceDefaults`,
`gpu.permittedHostDevices.pciHostDevices`, plus the consequence of
`replaceDefaults: true` with an empty list (no admittable GPU VMs).
- The manual Package-CR override path — when an operator hand-crafts
`cozystack.gpu-operator` outside the bundle for advanced overrides, they
also hand-craft `cozystack.kubevirt` with the matching
`extraFeatureGates` / `permittedHostDevices`. The manual override takes
precedence over the bundle render.
Only `next/virtualization/gpu.md` is touched. The released doc versions
(`v1.4` and earlier) describe earlier Cozystack releases that still
require the manual `kubectl edit`, and stay as-is.
### Release note
```release-note
docs(gpu): the GPU Passthrough guide no longer instructs operators to manually patch the KubeVirt CR — Cozystack now auto-wires the HostDevices feature gate and a starter NVIDIA permittedHostDevices table whenever cozystack.gpu-operator is enabled in bundles.enabledPackages. Operators extend or replace the defaults via .gpu.permittedHostDevices and .gpu.replaceDefaults.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Updated GPU virtualization guide to remove a manual edit step and
explain automated configuration when the GPU package is enabled.
* Added verification commands to inspect rendered GPU settings and
detailed how to extend or fully replace NVIDIA defaults via package
values.
* Included upgrade guidance for prior manual setups and documented a
manual override path for non-bundled deployments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Copy file name to clipboardExpand all lines: content/en/docs/next/virtualization/gpu.md
+84-18Lines changed: 84 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -100,32 +100,98 @@ Allocatable:
100
100
For example, the database entry forA10 reads `2236 GA102GL [A10]`, which resultsin a resource name `nvidia.com/GA102GL_A10`.
101
101
{{% /alert %}}
102
102
103
-
## 2. Update the KubeVirt Custom Resource
103
+
## 2. KubeVirt is wired automatically
104
104
105
-
Next, we will update the KubeVirt Custom Resource, as documented in the
106
-
[KubeVirt user guide](https://kubevirt.io/user-guide/virtual_machines/host-devices/#listing-permitted-devices),
107
-
so that the passthrough GPUs are permitted and can be requested by a KubeVirt VM.
105
+
When `cozystack.gpu-operator` is in`bundles.enabledPackages`, Cozystack mirrors the chosen GPU variant into the `KubeVirt` Custom Resource for you. There is no `kubectl edit kubevirt` step.
108
106
109
-
Adjust the `pciVendorSelector` and `resourceName` values to match your specific GPU model.
110
-
Setting `externalResourceProvider=true` indicates that this resource is provided by an external device plugin,
111
-
in this case the `sandbox-device-plugin` which is deployed by the Operator.
107
+
Specifically, the platform injects:
108
+
109
+
- `HostDevices` into `spec.configuration.developerConfiguration.featureGates` (current KubeVirt splits this from the `GPU` gate; the admission webhook rejects `domain.devices.hostDevices` without it).
110
+
- A starter `spec.configuration.permittedHostDevices.pciHostDevices` table (rendered in the default `gpuOperatorVariant: default` — vfio-pci passthrough) covering common NVIDIA datacenter GPUs — Hopper (H100, H200), Ada Lovelace (L4, L40, L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4), Volta (V100, V100S). PCI vendor:device pairs are stable; each `resourceName` slug is whatever `nvidia-sandbox-device-plugin` derives mechanically from the card's PCI-IDs database name — uppercase the name, turn `/`, `.` and whitespace into `_`, then strip the surrounding `[` / `]`. The slug therefore carries every token the PCI-IDs string holds (the `GL` die suffix, the `Tesla` brand on Turing/Volta, the form factor, the memory size), not a tidy `<arch>_<model>`: `TU104GL [Tesla T4]` becomes `nvidia.com/TU104GL_TESLA_T4`, `GA100GL [A30 PCIe]` becomes `nvidia.com/GA100GL_A30_PCIE`, and the H200 SXM becomes `nvidia.com/GH100_H200_SXM_141GB`. Confirm the exact strings your nodes advertise with `kubectl describe node <node> | grep nvidia.com/`. `externalResourceProvider: true` is set on every entry because the resources are advertised by the sandbox plugin, not by KubeVirt's in-tree device plugin.
111
+
112
+
Verify the resulting CR:
112
113
113
114
```bash
114
-
kubectl edit kubevirt -n cozy-kubevirt
115
+
kubectl -n cozy-kubevirt get kubevirt kubevirt -o json \
**My GPU isn't in the default table — where's the old `kubectl edit kubevirt` step?** It is gone on purpose. `permittedHostDevices` is now owned by the chart template and reconciled from platform values, so any hand edit to the live CR is reverted on the next Flux/Helm reconcile. Add your card through `.gpu.permittedHostDevices` instead — see [Extending or replacing the NVIDIA defaults](#extending-or-replacing-the-nvidia-defaults) below. If you are upgrading from a release where you hand-edited the CR, follow [Upgrading from a hand-edited KubeVirt CR](#upgrading-from-a-hand-edited-kubevirt-cr) first.
122
+
123
+
{{% /alert %}}
124
+
125
+
### Extending or replacing the NVIDIA defaults
126
+
127
+
If your cluster ships a GPU not in the default table, or your `nvidia-sandbox-device-plugin` version emits a different `resourceName` (check with `kubectl describe node <node>| grep nvidia.com/`), extend the defaults via platform values:
128
+
117
129
```yaml
118
-
...
119
-
spec:
120
-
configuration:
121
-
permittedHostDevices:
122
-
pciHostDevices:
123
-
- externalResourceProvider: true
124
-
pciVendorSelector: 10DE:2236
125
-
resourceName: nvidia.com/GA102GL_A10
126
-
...
130
+
# Platform Package values
131
+
gpu:
132
+
# Append (default) — your entries land alongside the NVIDIA table.
133
+
# Set to true to drop the NVIDIA table entirely (useful for non-NVIDIA-only
134
+
# clusters or strict allowlists). With replaceDefaults: true and an empty
135
+
# list below, the rendered CR carries no permittedHostDevices block at all
136
+
# and the admission webhook rejects every GPU VM — supply your own list.
137
+
replaceDefaults: false
138
+
permittedHostDevices:
139
+
pciHostDevices:
140
+
- pciVendorSelector: "10DE:2236"
141
+
resourceName: nvidia.com/GA102GL_A10
142
+
externalResourceProvider: true
143
+
```
144
+
145
+
To **re-point** a card already in the NVIDIA table (for example to give `10DE:1EB8` a different `resourceName`), do not append a second entry for the same `pciVendorSelector` — both entries are rendered and KubeVirt resolves the duplicated selector non-deterministically. Set `replaceDefaults: true` and supply the full list you want instead.
146
+
147
+
### Upgrading from a hand-edited KubeVirt CR
148
+
149
+
Earlier Cozystack releases left `spec.configuration.permittedHostDevices`for operators to hand-edit (`kubectl edit kubevirt`). The bundle now **owns** that field: the first reconcile after the upgrade replaces your manual entries with the rendered NVIDIA default table.
150
+
151
+
Before upgrading:
152
+
153
+
1. Dump your current entries:
154
+
155
+
```bash
156
+
kubectl -n cozy-kubevirt get kubevirt kubevirt -o json \
157
+
| jq '.spec.configuration.permittedHostDevices'
158
+
```
159
+
160
+
2. Move any custom entries into the Platform Package values under `.gpu.permittedHostDevices` (set `.gpu.replaceDefaults: true`if you want only your own list instead of appending to the NVIDIA defaults).
161
+
162
+
3. Verify every `resourceName` against what your nodes actually advertise. The default table carries the slug `nvidia-sandbox-device-plugin` generates from each card's PCI-IDs name (uppercased, e.g. `nvidia.com/TU104GL_TESLA_T4` for a Tesla T4), but a different plugin build or PCI-IDs snapshot can emit a different string:
163
+
164
+
```bash
165
+
kubectl describe node <node> | grep nvidia.com/
166
+
```
167
+
168
+
A `resourceName` mismatch is silent until a GPU VM restarts or migrates, at which point the admission webhook rejects it.
169
+
170
+
### Manual Package-CR override path
171
+
172
+
If you opt out of bundle management and hand-craft a `cozystack.gpu-operator` Package CR directly (to apply overrides the bundle does not expose — driver settings, custom node selectors, validator / dcgmExporter tweaks), the platform does NOT auto-wire `HostDevices` or `permittedHostDevices` into the KubeVirt CR. In that flow, mirror the bundle behaviour by also creating a `cozystack.kubevirt` Package CR that carries `extraFeatureGates` and the matching `permittedHostDevices` block under `spec.components.kubevirt.values` (a cozystack `Package` always nests component values under `spec.components.<name>.values`, never a top-level `spec.values`):
173
+
174
+
```yaml
175
+
apiVersion: cozystack.io/v1alpha1
176
+
kind: Package
177
+
metadata:
178
+
name: cozystack.kubevirt
179
+
spec:
180
+
variant: default
181
+
components:
182
+
kubevirt:
183
+
values:
184
+
extraFeatureGates:
185
+
- HostDevices
186
+
permittedHostDevices:
187
+
pciHostDevices:
188
+
- pciVendorSelector: "10DE:2236"
189
+
resourceName: nvidia.com/GA102GL_A10
190
+
externalResourceProvider: true
127
191
```
128
192
193
+
The manual Package-CR override path takes precedence over the bundle render whenever both exist.
0 commit comments