Skip to content

Commit 117d317

Browse files
committed
fix(platform): drop fragile numeric vGPU mediatedDeviceTypes defaults
The vgpu defaults shipped a mediatedDevicesConfiguration block whose mediatedDeviceTypes were hardcoded nvidia-NNN ids. Those ids are not stable identifiers — they are the per-(SKU, driver) indices under which a host enumerates mdev types in /sys/.../mdev_supported_types/, so the same profile gets a different number on a different card or driver branch. Auto-injecting them made KubeVirt try to create mdev types that usually do not exist on the operator's nodes, and duplicated work the GPU Operator's vGPU Device Manager already does. Delegate mdev creation to the vGPU Device Manager, which is name-based and stable, and keep only the name-based permittedHostDevices.mediatedDevices exposure list (mdevNameSelector) in the defaults. KubeVirt-driven creation stays available through the .gpu.mediatedDevicesConfiguration override for operators who supply host-verified type ids; document the lookup recipe. Update the bundle wiring, the gpu/vgpu values comments, the vGPU guide and the tests accordingly (the default vgpu path now emits no mediatedDevicesConfiguration; the override path is covered explicitly). Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 9190736 commit 117d317

5 files changed

Lines changed: 108 additions & 60 deletions

File tree

‎docs/gpu-vgpu.md‎

Lines changed: 3 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ If you opt out of bundle management and hand-craft a `cozystack.gpu-operator` Pa
100100

101101
- `developerConfiguration.featureGates` gets `HostDevices` appended (current KubeVirt splits this from the `GPU` gate; the admission webhook rejects `spec.template.spec.domain.devices.hostDevices` without it).
102102
- `permittedHostDevices.pciHostDevices` is filled from `packages/core/platform/files/gpu-passthrough-defaults.yaml` when `bundles.iaas.gpuOperatorVariant: default` (the package default). The table covers Hopper (H100/H200), Ada Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4), Volta (V100/V100S). All entries carry `externalResourceProvider: true` because the resource names come from `nvidia-sandbox-device-plugin`, not from KubeVirt's in-tree device plugin.
103-
- `permittedHostDevices.mediatedDevices` and `mediatedDevicesConfiguration` are filled from `packages/core/platform/files/gpu-vgpu-defaults.yaml` when `bundles.iaas.gpuOperatorVariant: vgpu`. The starter set covers Pascal–Ampere mdev profiles (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q, V100D-32C, T4-16Q) — the same family range as the upstream `vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace / Blackwell SR-IOV vGPU is out of scope for the chart's default list; advertise those VFs via the user-override hook below.
103+
- `permittedHostDevices.mediatedDevices` is filled from `packages/core/platform/files/gpu-vgpu-defaults.yaml` when `bundles.iaas.gpuOperatorVariant: vgpu`. This list only EXPOSES, by profile name (`mdevNameSelector`), mdevs that the GPU Operator's vGPU Device Manager CREATES on the node; the platform does not ship a numeric `mediatedDevicesConfiguration` default (those `nvidia-NNN` type ids are per-SKU/driver sysfs indices with no portable value — set `.gpu.mediatedDevicesConfiguration` yourself, with host-verified ids, only if you want KubeVirt rather than the Device Manager to create mdevs). The starter set covers Pascal–Ampere mdev profiles (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q, V100D-32C, T4-16Q) — the same family range as the upstream `vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace / Blackwell SR-IOV vGPU is out of scope for the chart's default list; advertise those VFs via the user-override hook below.
104104

105105
### Extending or replacing the default table
106106

@@ -121,12 +121,7 @@ gpu:
121121
# kubevirt/kubevirt#16890, virt-handler's in-tree device plugin
122122
# advertises the resource directly, no sandbox plugin in the loop.
123123
mediatedDevices: []
124-
# mediatedDevicesConfiguration drives the per-node vGPU profile
125-
# auto-creation loop in vgpu mode. Helm's mergeOverwrite applies — a
126-
# user-supplied top-level key (mediatedDeviceTypes,
127-
# nodeMediatedDeviceTypes) REPLACES the platform default for that key
128-
# rather than appending. Combine with replaceDefaults: true when both
129-
# this and permittedHostDevices.mediatedDevices should be wiped together.
124+
# mediatedDevicesConfiguration makes KubeVirt itself create mdevs (vgpu mode). No platform default: mdev creation is normally delegated to the vGPU Device Manager (name-based), and these mediatedDeviceTypes are host/driver-specific nvidia-NNN sysfs indices (look yours up via /sys/bus/pci/devices/<BDF>/mdev_supported_types/*/name). Set this only to opt into KubeVirt-driven creation; mergeOverwrite REPLACES a supplied top-level key wholesale.
130125
mediatedDevicesConfiguration: {}
131126
# Wipe the platform defaults entirely and ship only the cluster's
132127
# curated lists. Useful for non-NVIDIA-only clusters and strict
@@ -138,7 +133,7 @@ gpu:
138133

139134
### Notes on `nvidia-sandbox-device-plugin` resource names
140135

141-
The `resourceName` strings in `gpu-passthrough-defaults.yaml` follow what `nvidia-sandbox-device-plugin` v25.x generates: a slug of `<arch>_<model>_<form>_<mem>` (e.g. `nvidia.com/GH100_H200_SXM_141GB`). If the plugin version drifts that convention, your node will publish a different name — check with `kubectl describe node <node> | grep nvidia.com/` and override via `.gpu.permittedHostDevices.pciHostDevices` (or wipe the table with `replaceDefaults: true` and curate it yourself). PCI vendor:device IDs themselves are stable across driver versions.
136+
The `resourceName` strings in `gpu-passthrough-defaults.yaml` are what `nvidia-sandbox-device-plugin` (`nvcr.io/nvidia/kubevirt-gpu-device-plugin`) advertises: it derives each slug mechanically from the device's PCI-IDs database name by uppercasing it, turning `/`, `.` and whitespace into `_`, and stripping the remaining non-alphanumerics (the `[` / `]`). So `TU104GL [Tesla T4]` becomes `nvidia.com/TU104GL_TESLA_T4` and `GA100GL [A30 PCIe]` becomes `nvidia.com/GA100GL_A30_PCIE` — the slug carries every token the PCI-IDs string holds (the `GL` die suffix, the `Tesla` brand on Turing/Volta, form factor, memory), not a tidy `<arch>_<model>`. The names track the pci.ids snapshot bundled in the plugin image, so a different plugin build can publish a different string — check with `kubectl describe node <node> | grep nvidia.com/` and override via `.gpu.permittedHostDevices.pciHostDevices` (or wipe the table with `replaceDefaults: true` and curate it yourself). PCI vendor:device IDs themselves are stable across driver versions.
142137

143138
### SR-IOV PF vs VF (Ada Lovelace and newer)
144139

‎packages/core/platform/files/gpu-vgpu-defaults.yaml‎

Lines changed: 28 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,32 @@
1-
# NVIDIA vGPU profiles commonly exposed by nvidia-vgpu-manager in vGPU
2-
# mode. Used by bundles/iaas.yaml when bundles.iaas.gpuOperatorVariant=vgpu
3-
# and cozystack.gpu-operator is in bundles.enabledPackages. Auto-injected
4-
# into the KubeVirt CR at spec.configuration.permittedHostDevices.mediatedDevices
5-
# and spec.configuration.mediatedDevicesConfiguration.
1+
# NVIDIA vGPU profiles to EXPOSE to VMs in vGPU mode. Used by
2+
# bundles/iaas.yaml when bundles.iaas.gpuOperatorVariant=vgpu and
3+
# cozystack.gpu-operator is in bundles.enabledPackages. Auto-injected
4+
# into the KubeVirt CR at spec.configuration.permittedHostDevices.mediatedDevices.
65
#
7-
# The profiles below cover the 24Q (3D-optimised) and 48Q (datacenter
8-
# whole-card) splits on the GPU families cozystack supports for vGPU
9-
# (Ampere and earlier — Ada Lovelace and Blackwell use per-VF NVIDIA
10-
# sysfs and are out of scope for the chart's vGPU defaults; see
11-
# system/gpu-operator/values-vgpu.yaml).
6+
# Division of labour: the GPU Operator's vGPU Device Manager CREATES the
7+
# mdev devices on each node from a name-based profile config; KubeVirt
8+
# only EXPOSES the already-created mdevs to VMs, matched by profile name
9+
# via mdevNameSelector. This file therefore carries only the name-based
10+
# exposure list — it does NOT ship a mediatedDevicesConfiguration block.
11+
# That block's mediatedDeviceTypes are numeric nvidia-NNN ids that make
12+
# KubeVirt itself create mdevs, but those ids are per-(SKU, driver) sysfs
13+
# indices with no portable value, and would duplicate the Device
14+
# Manager. Operators who deliberately want KubeVirt to drive creation set
15+
# host-verified ids via .Values.gpu.mediatedDevicesConfiguration instead.
16+
#
17+
# mdevNameSelector must match the on-host profile name the driver exposes
18+
# under /sys/bus/pci/devices/<BDF>/mdev_supported_types/*/name — which is
19+
# driver-branch-dependent ("NVIDIA A100-40C" on the vGPU 13+/AIE branch,
20+
# "GRID A100-40C" on older GRID drivers). The profiles below cover the
21+
# 24Q/48Q and whole-card C splits on the families cozystack supports for
22+
# vGPU (Ampere and earlier — Ada Lovelace and Blackwell use per-VF NVIDIA
23+
# sysfs and are out of scope; see system/gpu-operator/values-vgpu.yaml).
24+
# Verify against the live node and override via
25+
# .Values.gpu.permittedHostDevices / .Values.gpu.replaceDefaults if your
26+
# driver exposes different names:
27+
#
28+
# for t in /sys/bus/pci/devices/<BDF>/mdev_supported_types/*; do \
29+
# echo "$(basename "$t") -> $(cat "$t"/name)"; done
1230
#
1331
# externalResourceProvider: true is mandatory — the mediated devices are
1432
# published by the vGPU Manager DaemonSet, not by KubeVirt itself.
@@ -38,18 +56,3 @@ permittedHostDevices:
3856
- mdevNameSelector: NVIDIA T4-16Q
3957
resourceName: nvidia.com/T4-16Q
4058
externalResourceProvider: true
41-
mediatedDevicesConfiguration:
42-
# mediatedDeviceTypes lists the vGPU profile IDs the operator may
43-
# auto-create per-node from the underlying physical GPUs. Numeric IDs
44-
# are the NVIDIA-published vGPU type numbers (see GRID/vGPU release
45-
# notes for the full mapping); listing one whole-card type per GPU
46-
# family keeps the default deterministic.
47-
mediatedDeviceTypes:
48-
- nvidia-559 # A100-40C
49-
- nvidia-560 # A100-80C
50-
- nvidia-558 # A40-48Q
51-
- nvidia-555 # A40-24Q
52-
- nvidia-453 # A30-24C
53-
- nvidia-471 # A10-24Q
54-
- nvidia-413 # V100D-32C
55-
- nvidia-232 # T4-16Q

‎packages/core/platform/templates/bundles/iaas.yaml‎

Lines changed: 22 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -55,11 +55,21 @@
5555
pciHostDevices applies in both variants — passthrough (vfio-pci, whole
5656
GPU) and post-kubevirt#16890 SR-IOV vGPU VFs (Ada Lovelace / Blackwell
5757
with the vGPU 17/20 driver branch) both advertise via pciHostDevices.
58-
mediatedDevices and mediatedDevicesConfiguration are vgpu-only: they
59-
drive the pre-#16890 mdev path (Pascal–Ampere). Emitting them in the
60-
passthrough variant would carry vGPU-only configuration into a CR that
61-
has no vGPU manager deployed — a footgun for operators who keep a
62-
shared gpu values block and flip the variant back to default.
58+
mediatedDevices is vgpu-only: it EXPOSES, by stable profile name
59+
(mdevNameSelector), the mdev devices the GPU Operator's vGPU Device
60+
Manager creates on the node. Emitting it in the passthrough variant
61+
would carry vGPU-only configuration into a CR that has no vGPU manager
62+
deployed — a footgun for operators who keep a shared gpu values block
63+
and flip the variant back to default.
64+
65+
mediatedDevicesConfiguration is intentionally NOT defaulted. Its
66+
mediatedDeviceTypes entries are numeric nvidia-NNN ids that make
67+
KubeVirt itself create mdevs — but those ids are per-(SKU, driver)
68+
sysfs indices with no portable value, and mdev creation is already
69+
handled by the vGPU Device Manager (name-based). It is therefore
70+
emitted only from the .gpu.mediatedDevicesConfiguration override, for
71+
operators who deliberately want KubeVirt to drive creation with
72+
host-verified type ids.
6373
*/ -}}
6474
{{- $pci := list -}}
6575
{{- $mdev := list -}}
@@ -72,18 +82,18 @@
7282
{{- $pci = concat $pci $userPci -}}
7383
{{- else -}}
7484
{{- /*
75-
vgpu defaults table is only useful when nvidia-vgpu-device-manager is
76-
actually deployed — without it, /sys/class/mdev_bus/ stays empty and
77-
the rendered mediated resources can never become allocatable.
78-
Suppress the platform mdev defaults when vgpuDeviceManager.enabled is
79-
false (the case on Ada Lovelace / Blackwell SR-IOV); the operator can
80-
still set their own mediated entries via .gpu.permittedHostDevices.
85+
The mediatedDevices defaults name the mdev profiles the vGPU Device
86+
Manager creates; they are only useful when that DaemonSet is actually
87+
deployed — without it, /sys/class/mdev_bus/ stays empty and the
88+
rendered mediated resources can never become allocatable. Suppress the
89+
platform mdev defaults when vgpuDeviceManager.enabled is false (the
90+
case on Ada Lovelace / Blackwell SR-IOV); the operator can still set
91+
their own mediated entries via .gpu.permittedHostDevices.
8192
*/ -}}
8293
{{- $deviceManagerOn := default false (((.Values.gpu).vgpuDeviceManager).enabled) -}}
8394
{{- if and (not $replace) $deviceManagerOn -}}
8495
{{- $vgpuDefaults := $.Files.Get "files/gpu-vgpu-defaults.yaml" | fromYaml -}}
8596
{{- $mdev = concat $mdev $vgpuDefaults.permittedHostDevices.mediatedDevices -}}
86-
{{- $mdevConfig = $vgpuDefaults.mediatedDevicesConfiguration -}}
8797
{{- end -}}
8898
{{- $pci = concat $pci $userPci -}}
8999
{{- $mdev = concat $mdev $userMdev -}}

‎packages/core/platform/tests/bundles_gpu_kubevirt_wiring_test.yaml‎

Lines changed: 39 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ tests:
9595
path: spec.variant
9696
value: vgpu
9797

98-
- it: wires HostDevices gate plus NVIDIA mediatedDevices and mediatedDevicesConfiguration when gpu-operator runs in vgpu variant and gpu.vgpuDeviceManager.enabled is true (Pascal-Ampere mdev path)
98+
- it: exposes NVIDIA mediatedDevices by name in vgpu variant when gpu.vgpuDeviceManager.enabled is true, and does NOT default a numeric mediatedDevicesConfiguration (mdev creation is delegated to the vGPU Device Manager)
9999
set:
100100
bundles:
101101
system:
@@ -120,8 +120,11 @@ tests:
120120
path: spec.components.kubevirt.values.permittedHostDevices.mediatedDevices
121121
- notExists:
122122
path: spec.components.kubevirt.values.permittedHostDevices.pciHostDevices
123-
- exists:
124-
path: spec.components.kubevirt.values.mediatedDevicesConfiguration.mediatedDeviceTypes
123+
# No numeric mediatedDeviceTypes is shipped by default — those ids are
124+
# per-(SKU, driver) sysfs indices, and the vGPU Device Manager already
125+
# creates the mdevs by name.
126+
- notExists:
127+
path: spec.components.kubevirt.values.mediatedDevicesConfiguration
125128

126129
- it: suppresses the NVIDIA mdev defaults in vgpu variant when gpu.vgpuDeviceManager.enabled is false (default — Ada Lovelace SR-IOV path), so KubeVirt does not advertise resources that no daemon can ever create
127130
set:
@@ -665,3 +668,36 @@ tests:
665668
pciVendorSelector: "10DE:1DB1"
666669
resourceName: nvidia.com/GV100_V100_SXM2_32GB
667670
externalResourceProvider: true
671+
672+
- it: still forwards a user-supplied gpu.mediatedDevicesConfiguration into the CR in vgpu mode (the override escape hatch survives the name-based default), alongside the name-based mediatedDevices defaults
673+
set:
674+
bundles:
675+
system:
676+
enabled: true
677+
variant: isp-full
678+
iaas:
679+
enabled: true
680+
gpuOperatorVariant: vgpu
681+
enabledPackages:
682+
- cozystack.gpu-operator
683+
gpu:
684+
vgpuDeviceManager:
685+
enabled: true
686+
mediatedDevicesConfiguration:
687+
mediatedDeviceTypes:
688+
- nvidia-473
689+
documentSelector:
690+
path: metadata.name
691+
value: cozystack.kubevirt
692+
asserts:
693+
# name-based exposure defaults still flow in...
694+
- contains:
695+
path: spec.components.kubevirt.values.permittedHostDevices.mediatedDevices
696+
content:
697+
mdevNameSelector: NVIDIA A100-40C
698+
resourceName: nvidia.com/A100-40C
699+
externalResourceProvider: true
700+
# ...and the operator's explicit numeric config is honoured.
701+
- contains:
702+
path: spec.components.kubevirt.values.mediatedDevicesConfiguration.mediatedDeviceTypes
703+
content: nvidia-473

‎packages/core/platform/values.yaml‎

Lines changed: 16 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -348,13 +348,17 @@ gpu:
348348
# resourceName: nvidia.com/A100-40C
349349
# externalResourceProvider: true
350350
mediatedDevices: []
351-
# mediatedDevicesConfiguration drives the vGPU profile auto-creation loop
352-
# (vgpu variant only). Helm's mergeOverwrite semantics apply: a top-level
353-
# key the user supplies REPLACES the platform-default value for that key
354-
# rather than appending — so user mediatedDeviceTypes: [...] overrides the
355-
# platform's eight defaults entirely. Combine with replaceDefaults: true
356-
# when both this and permittedHostDevices.mediatedDevices should be wiped
357-
# together. Shape matches KubeVirt's MediatedDevicesConfiguration:
351+
# mediatedDevicesConfiguration makes KubeVirt itself create mdevs on the
352+
# node (vgpu variant only). The platform ships NO default here: mdev
353+
# creation is delegated to the GPU Operator's vGPU Device Manager
354+
# (name-based), and the mediatedDeviceTypes ids below are numeric
355+
# nvidia-NNN sysfs indices that vary per GPU SKU and driver branch — no
356+
# value is portable, so look yours up on the host
357+
# (/sys/bus/pci/devices/<BDF>/mdev_supported_types/*/name) before setting
358+
# it. Supplying it opts into KubeVirt-driven creation on top of the
359+
# Device Manager. Helm's mergeOverwrite semantics apply: a top-level key
360+
# you supply REPLACES that key wholesale rather than appending. Shape
361+
# matches KubeVirt's MediatedDevicesConfiguration:
358362
# mediatedDeviceTypes:
359363
# - nvidia-740
360364
# nodeMediatedDeviceTypes:
@@ -389,10 +393,10 @@ gpu:
389393
# DaemonSet crash-loops with "no parent devices found". Flip to true
390394
# on Pascal–Ampere clusters; leave false on Ada/Blackwell and supply
391395
# SR-IOV VFs via .gpu.permittedHostDevices.pciHostDevices instead.
392-
# When true the platform also lets the default mediatedDevices /
393-
# mediatedDevicesConfiguration table from
394-
# files/gpu-vgpu-defaults.yaml flow into the KubeVirt CR; when false
395-
# those defaults are suppressed (advertising mediated resources that
396-
# nothing can ever allocate is a worse footgun than missing entries).
396+
# When true the platform also lets the default name-based mediatedDevices
397+
# exposure list from files/gpu-vgpu-defaults.yaml flow into the KubeVirt
398+
# CR; when false those defaults are suppressed (advertising mediated
399+
# resources that nothing can ever allocate is a worse footgun than
400+
# missing entries).
397401
vgpuDeviceManager:
398402
enabled: false

0 commit comments

Comments
 (0)