Repository navigation
vGPU: SRIOV support - #16890
vGPU: SRIOV support#16890
Conversation
There was a problem hiding this comment.
Hey - I've found 3 issues, and left some high level feedback:
- In discoverPermittedHostPCIDevices, you call IsPhysicalFunction multiple times per device (inside the nvidia branch and again afterwards); consider calling it once, storing the result, and reusing it to avoid redundant sysfs reads and simplify the control flow.
- HasVGPUProfile currently treats any non-empty, non-"0" value as a valid profile; parsing current_vgpu_type as an integer and explicitly handling parse errors would make this more robust against unexpected file contents.
- The sysfs path components for vGPU detection ("nvidia" and "current_vgpu_type") are duplicated between the implementation and tests; consider defining shared constants to avoid divergence if the layout needs to change later.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In discoverPermittedHostPCIDevices, you call IsPhysicalFunction multiple times per device (inside the nvidia branch and again afterwards); consider calling it once, storing the result, and reusing it to avoid redundant sysfs reads and simplify the control flow.
- HasVGPUProfile currently treats any non-empty, non-"0" value as a valid profile; parsing current_vgpu_type as an integer and explicitly handling parse errors would make this more robust against unexpected file contents.
- The sysfs path components for vGPU detection ("nvidia" and "current_vgpu_type") are duplicated between the implementation and tests; consider defining shared constants to avoid divergence if the layout needs to change later.
## Individual Comments
### Comment 1
<location> `pkg/virt-handler/device-manager/common.go:216-217` </location>
<code_context>
+// IsPhysicalFunction checks if a PCI device is an SR-IOV Physical Function by
+// looking for virtfn* subdirectories in its sysfs path. Only Physical Functions
+// have these subdirectories representing their associated Virtual Functions.
+func (h *DeviceUtilsHandler) IsPhysicalFunction(basepath, pciAddress string) bool {
+ entries, err := os.ReadDir(filepath.Join(basepath, pciAddress))
+ if err != nil {
+ return false
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Consider distinguishing between "not a PF" and "failed to inspect" instead of returning false on any error.
Currently any `os.ReadDir` error is treated the same as “not a Physical Function”, so permission issues, transient I/O errors, or path bugs become indistinguishable from a genuine “no virtfn* entries” case. That can hide misconfiguration and make failures hard to diagnose. Consider returning a `(bool, error)` or at least logging non-`os.IsNotExist` errors so unexpected failures are visible while still treating the normal “directory missing” case as non-PF.
Suggested implementation:
```golang
// IsPhysicalFunction checks if a PCI device is an SR-IOV Physical Function by
// looking for virtfn* subdirectories in its sysfs path. Only Physical Functions
// have these subdirectories representing their associated Virtual Functions.
func (h *DeviceUtilsHandler) IsPhysicalFunction(basepath, pciAddress string) (bool, error) {
dirPath := filepath.Join(basepath, pciAddress)
entries, err := os.ReadDir(dirPath)
if err != nil {
// Treat the normal "directory does not exist" case as "not a PF".
if os.IsNotExist(err) {
return false, nil
}
// Propagate unexpected errors so callers can distinguish inspection
// failures from "not a PF".
return false, fmt.Errorf("failed to read PCI device directory %q: %w", dirPath, err)
}
for _, entry := range entries {
if entry.IsDir() && strings.HasPrefix(entry.Name(), "virtfn") {
return true, nil
}
}
return false, nil
}
```
1. Ensure the file imports `fmt` (and `os`, `filepath`, `strings` if they are not already imported), e.g.:
- `import "fmt"`
2. Update all call sites of `IsPhysicalFunction` to handle the new `(bool, error)` signature, for example:
- `isPF, err := h.IsPhysicalFunction(basepath, pciAddress)`
- Handle `err` appropriately (log/propagate) and use `isPF` only when `err == nil`.
</issue_to_address>
### Comment 2
<location> `pkg/virt-launcher/virtwrap/device/hostdevice/gpu/hostdev.go:38` </location>
<code_context>
DefaultDisplayOn = true
)
+// TODO: Pass in also medata so metada can be constructed also in tests, include if display is possible
func CreateHostDevices(vmiGPUs []v1.GPU) ([]api.HostDevice, error) {
return CreateHostDevicesFromPools(vmiGPUs, NewPCIAddressPool(vmiGPUs), NewMDEVAddressPool(vmiGPUs))
</code_context>
<issue_to_address>
**nitpick (typo):** Fix minor typos in the TODO comment for clarity.
"medata" and "metada" should both be "metadata" to keep the TODO clear for future readers.
```suggestion
// TODO: Pass in also metadata so metadata can be constructed also in tests, include if display is possible
```
</issue_to_address>
### Comment 3
<location> `pkg/virt-handler/device-manager/pci_device.go:268` </location>
<code_context>
if resourceName, supported := supportedPCIDeviceMap[pciID]; supported {
// check device driver
driver, err := handler.GetDeviceDriver(pciBasePath, info.Name())
- if err != nil || driver != "vfio-pci" {
</code_context>
<issue_to_address>
**issue (complexity):** Consider simplifying this logic by caching the physical-function check and moving the driver/vGPU decision into a dedicated helper to keep the walk callback flat and readable.
You can reduce the added complexity by (1) caching the PF check and (2) extracting the driver/vGPU policy into a small helper. That keeps the walk callback linear while preserving behavior.
### 1. Cache `IsPhysicalFunction` result
Avoid calling `IsPhysicalFunction` twice and scattering PF logic:
```go
isPF := handler.IsPhysicalFunction(pciBasePath, info.Name())
```
Use `isPF` in all checks instead of calling `IsPhysicalFunction` again.
### 2. Extract driver/vGPU decision into a helper
Move the branching policy into a focused function that returns a decision + reason, and handle logging once in the walk:
```go
type pciDecision int
const (
pciUse pciDecision = iota
pciSkip
)
func shouldUsePCIDevice(
handler HandlerType, // use the actual type
pciBasePath, devName, pciID, driver string,
isPF bool,
) (pciDecision, string) {
if driver == "vfio-pci" {
if isPF {
return pciSkip, fmt.Sprintf("Skipping SR-IOV Physical Function %s (%s)", devName, pciID)
}
return pciUse, ""
}
if driver == "nvidia" {
if isPF {
return pciSkip, fmt.Sprintf("Skipping non-vfio-pci Physical Function %s (%s)", devName, pciID)
}
if !handler.HasVGPUProfile(pciBasePath, devName) {
return pciSkip, fmt.Sprintf(
"Skipping non-vfio-pci device without vGPU profile %s (%s, driver: %s)",
devName, pciID, driver,
)
}
return pciUse, fmt.Sprintf("Found vGPU-capable VF %s (%s, driver: %s)", devName, pciID, driver)
}
return pciSkip, fmt.Sprintf("Not supported driver %s, pci %s", driver, pciID)
}
```
Then the walk callback becomes flatter:
```go
pciID, err := handler.GetDevicePCIID(pciBasePath, info.Name())
if err != nil {
log.DefaultLogger().Reason(err).Errorf("failed get vendor:device ID for device: %s", info.Name())
return nil
}
if resourceName, supported := supportedPCIDeviceMap[pciID]; supported {
driver, err := handler.GetDeviceDriver(pciBasePath, info.Name())
if err != nil {
log.DefaultLogger().Reason(err).Errorf("failed to get driver for device: %s", info.Name())
return nil
}
isPF := handler.IsPhysicalFunction(pciBasePath, info.Name())
decision, msg := shouldUsePCIDevice(handler, pciBasePath, info.Name(), pciID, driver, isPF)
if decision == pciSkip {
if msg != "" {
log.DefaultLogger().Infof("%s", msg)
}
return nil
}
if msg != "" {
log.DefaultLogger().Infof("%s", msg)
}
pcidev := &PCIDevice{
pciID: pciID,
pciAddress: info.Name(),
}
iommuGroup, err := handler.GetDeviceIOMMUGroup(pciBasePath, info.Name())
if err != nil {
return nil
}
pcidev.iommuGroup = iommuGroup
pcidev.driver = driver
pcidev.numaNode = handler.GetDeviceNumaNode(pciBasePath, info.Name())
pciDevicesMap[resourceName] = append(pciDevicesMap[resourceName], pcidev)
} else {
log.DefaultLogger().V(9).Infof("Not supported %s", pciID)
}
```
This keeps all existing behavior (including all logging messages and vGPU rules) but removes nested driver branches, deduplicates PF checks, and centralizes policy in a small, testable helper.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| DefaultDisplayOn = true | ||
| ) | ||
|
|
||
| // TODO: Pass in also medata so metada can be constructed also in tests, include if display is possible |
There was a problem hiding this comment.
nitpick (typo): Fix minor typos in the TODO comment for clarity.
"medata" and "metada" should both be "metadata" to keep the TODO clear for future readers.
| // TODO: Pass in also medata so metada can be constructed also in tests, include if display is possible | |
| // TODO: Pass in also metadata so metadata can be constructed also in tests, include if display is possible |
There was a problem hiding this comment.
Yes, exactly — that TODO comment.
If you want to fix it inline, it would look like this:
// TODO: Pass in also metadata so metadata can be constructed also in tests, include if display is possible|
@dhiller Are you tracking this problem https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/16890/pull-kubevirt-e2e-k8s-1.35-sig-compute-serial/2024514878953230336 |
| pcidev.numaNode = handler.GetDeviceNumaNode(pciBasePath, info.Name()) | ||
| pciDevicesMap[resourceName] = append(pciDevicesMap[resourceName], pcidev) | ||
| } else { | ||
| log.DefaultLogger().V(9).Infof("Not supported %s", pciID) |
There was a problem hiding this comment.
+1
However, V(9) sounds a bit high for this log.
There was a problem hiding this comment.
Any suggestion? This log fires a lot of times that's why I thing 9 is good here.
|
|
||
| // For devices without vfio-pci driver, check if they're NVIDIA vGPU VFs | ||
| if driver != "vfio-pci" { | ||
| if driver == "nvidia" { |
There was a problem hiding this comment.
and instead of this:
if driver == "nvidia" {
| // Just force the driver to NOT vfio-pci to ensure they all get ignored. | ||
| mockPCI.EXPECT().GetDeviceIOMMUGroup(pciBasePath, gomock.Any()).AnyTimes() | ||
| mockPCI.EXPECT().GetDeviceDriver(pciBasePath, gomock.Any()).Return("definitely-not-vfio-pci", nil).AnyTimes() | ||
| mockPCI.EXPECT().GetDeviceDriver(pciBasePath, gomock.Any()).Return("vfio-pci", nil).AnyTimes() |
There was a problem hiding this comment.
Can you explain why we need this?
There was a problem hiding this comment.
I dropped this. @jean-edouard Wat is the rational to not return vfio-pci?
| if err != nil { | ||
| return nil |
There was a problem hiding this comment.
Not related to the pr but I wonder if we should add a log here
| entries, err := os.ReadDir(filepath.Join(basepath, pciAddress)) | ||
| if err != nil { | ||
| return false | ||
| } | ||
|
|
||
| for _, entry := range entries { | ||
| if strings.HasPrefix(entry.Name(), "virtfn") { | ||
| return true | ||
| } | ||
| } | ||
|
|
||
| return false |
There was a problem hiding this comment.
maybe we can just check if sriov_totalvfs file exist iiuc it should only exist when this is PF
|
I think we should add unit tests for |
|
|
||
| // For devices without vfio-pci driver, check if they're NVIDIA vGPU VFs | ||
| if driver != "vfio-pci" { | ||
| if driver == "nvidia" { |
There was a problem hiding this comment.
This is one of my major concerns ... I'm really worried that we will start accumulating these vendor-specific implementations...
Can you please isolate all this into a helper function, something like shouldPermitNonVfioPCIDevice?
I think we need to follow up with a plugable checker - I'm not sure right now how to implement this, but we could have some kind of a DeviceEligibilityChecker interface
, which will have a ShouldPermit function. We could then have a vendor dedicated pci/checkers/nvidia.go - each one will register itself...
something like that.
We've been seeing clustered failures with this symptom for a while on the serial lane now - example: https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/directory/pull-kubevirt-e2e-k8s-1.34-sig-compute-serial/2013237621370654720 Tracker #16608 |
|
@vladikr PTAL |
| return pciDevicesMap | ||
| } | ||
|
|
||
| func shouldPermitNvidia(driver, pciBasePath, pciAddress, pciID string) bool { |
There was a problem hiding this comment.
:(
I think we should avoid using vendor names in functions...
There was a problem hiding this comment.
I think we need to somehow separate all of these vendor specifc validations
maybe we can create a folder of these validators for each vendor?
something like pkg/virt-handler/device-manager/pci/validators/nvidia.go
Then on the top of the file (around line 275) we will have a generic isDeviceEligible(pcidev, handler) that will loop over these external validators.
Half baked idea... I'll try to write it.
There was a problem hiding this comment.
Why is vendor name better in file name than function?
There was a problem hiding this comment.
same answer tbh... """because this way it's separate and cleaner - do you really want to spread this all over the codebase? to me this is crazy."""
I would prefer to consolidate this instead of spreading this all over the code base.
This way, there could be different validators for different vendors, and this is clearer and consolidated, rather than multiple functions with vendor names
| // For devices without vfio-pci driver, check if they're NVIDIA vGPU VFs | ||
| if driver != "vfio-pci" { | ||
| if !shouldPermitNvidia(driver, pciBasePath, info.Name(), pciID) { | ||
| log.DefaultLogger().V(9).Infof("Not supported driver %s, pci %s", driver, pciID) | ||
| return nil | ||
| } | ||
| } | ||
|
|
||
| pcidev := &PCIDevice{ | ||
| pciID: pciID, | ||
| pciAddress: info.Name(), | ||
| } |
There was a problem hiding this comment.
@xpivarc
Here is what I can suggest. Create a generic pluggable validator, so each vendor will be separated into its own file and more could be added.
I think we can create these in
pkg/virt-handler/device-manager/pci/validator.go
pkg/virt-handler/device-manager/pci/validators/nvidia.go
Here is what I had in mind:
https://gist.github.com/vladikr/297fd699327e58d8562647f0394c8150
and then here we could do something like:
pcidev := &PCIDevice{
pciID: pciID,
pciAddress: info.Name(),
driver: driver,
}
if !isDeviceEligible(pcidev, handler) {
log.DefaultLogger().V(9).Infof("Not supported driver %s, pci %s", driver, pciID)
return nil
}
There was a problem hiding this comment.
Is it really worth it? what is the difference?
There was a problem hiding this comment.
because this way it's separate and cleaner - do you really want to spread this all over the codebase? to me this is crazy.
There was a problem hiding this comment.
Honestly, it's a very simple change - If you want, I can write the code, but I think that this is 100% worth it - as a long term investmet
The last thing this project needs is spreading vendor specific code around the code base - this was my take for a very long time
There was a problem hiding this comment.
I agree we should not spread it all over the place, can we compromise to move the logic under specific file but not creating unnecessary abstraction?
Replace all instances of `strings.Replace(str, old, new, -1)` with `strings.ReplaceAll(str, old, new)` in mediated_devices_types.go. This is the idiomatic way to express "replace all occurrences" since Go 1.12 and improves code readability. Signed-off-by: Luboslav Pivarc <[email protected]>
Make sure that tests are isolated and the handler variable is always reverted. Signed-off-by: Luboslav Pivarc <[email protected]>
This commit adds following functions to existing `DeviceHandler`: 1. HasVGPUProfile - returns true if VF has a vGPU assigned (nvidia only) 2. IsPhysicalFunction - returns true if PCI device is SRIOV PF Signed-off-by: Luboslav Pivarc <[email protected]>
Some PCI devices can now also enable Display and RamFB, this is left out to implement. Signed-off-by: Luboslav Pivarc <[email protected]>
| } | ||
|
|
||
| func shouldPermitNonVFIOPCI(driver, pciBasePath, pciAddress, pciID string) bool { | ||
| return shouldPermitNvidia(driver, pciBasePath, pciAddress, pciID) |
There was a problem hiding this comment.
Would you consider a follow up to make this generic?
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: vladikr The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/retest |
|
|
||
| func shouldPermitNonVFIOPCI(driver, pciBasePath, pciAddress, pciID string) bool { | ||
| return shouldPermitNvidia(driver, pciBasePath, pciAddress, pciID) | ||
| } |
There was a problem hiding this comment.
This was asked by Vladik to have a generic place where we can then evolve with more hw support, e.g amd :)
|
/lgtm |
|
Required labels detected, running phase 2 presubmits: |
|
/retest-required |
|
Nice Note please that once we discuss MDEV vGPU emulation, it case we like the direction, we can do the same for SR-IOV vGPU, For that will be great if you have a manifest / e2e test to share when time comes, even quick and dirty one that you tested on the real HW, Thanks |
The original draft documented mediated devices (mdev) — that path does not apply to current NVIDIA data-centre GPUs (L4, L40, L40S, B100). On the vGPU 17/20 driver branch these GPUs use SR-IOV with per-VF NVIDIA sysfs; KubeVirt advertises VFs via permittedHostDevices.pciHostDevices, not mediatedDevices. This commit rewrites the vGPU section accordingly: - Add a driver-model note at the top making the mdev/SR-IOV split explicit. Pascal–Ampere readers are sent to upstream NVIDIA docs. - Replace mediatedDeviceTypes / mediatedDevices / gpus: with pciHostDevices / hostDevices throughout. - Document KubeVirt v1.9.0 (ETA July 2026, kubevirt/kubevirt#16890 shipping in that release) as a hard prerequisite — current released tags v1.6.x/1.7.x/1.8.x do not include the patch and backports are not planned. - Replace the legacy NLS block (ServerAddress=, ServerPort=7070, FeatureType=1) with the DLS ClientConfigToken flow. - Switch build instructions to the upstream NVIDIA repo (github.com/NVIDIA/gpu-driver-container) — the gitlab repo is archived. - Note explicitly that Talos is not recommended for vGPU (NVIDIA redistribution restrictions, siderolabs/extensions#461 closed won't-fix); passthrough on Talos is unaffected. - Add a warning that a 2.4 GiB containerDisk overlay is too small to install the GRID guest driver in-place — recommend a CDI DataVolume of 20 GiB+ for non-throwaway tests. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
The original draft documented mediated devices (mdev) — that path does not apply to current NVIDIA data-centre GPUs (L4, L40, L40S, B100). On the vGPU 17/20 driver branch these GPUs use SR-IOV with per-VF NVIDIA sysfs; KubeVirt advertises VFs via permittedHostDevices.pciHostDevices, not mediatedDevices. This commit rewrites the vGPU section accordingly: - Add a driver-model note at the top making the mdev/SR-IOV split explicit. Pascal–Ampere readers are sent to upstream NVIDIA docs. - Replace mediatedDeviceTypes / mediatedDevices / gpus: with pciHostDevices / hostDevices throughout. - Document KubeVirt v1.9.0 (ETA July 2026, kubevirt/kubevirt#16890 shipping in that release) as a hard prerequisite — current released tags v1.6.x/1.7.x/1.8.x do not include the patch and backports are not planned. - Replace the legacy NLS block (ServerAddress=, ServerPort=7070, FeatureType=1) with the DLS ClientConfigToken flow. - Switch build instructions to the upstream NVIDIA repo (github.com/NVIDIA/gpu-driver-container) — the gitlab repo is archived. - Note explicitly that Talos is not recommended for vGPU (NVIDIA redistribution restrictions, siderolabs/extensions#461 closed won't-fix); passthrough on Talos is unaffected. - Add a warning that a 2.4 GiB containerDisk overlay is too small to install the GRID guest driver in-place — recommend a CDI DataVolume of 20 GiB+ for non-throwaway tests. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
…2323) ## What this PR does Updates the NVIDIA GPU Operator from v25.3.0 to v26.3.1 and adds an **experimental** `vgpu` variant for SR-IOV vGPU on Ada Lovelace and newer GPUs. Changes: - Update vendored upstream chart to v26.3.1 (CDI enabled by default, component version bumps). - Pin chart version in Makefile for reproducibility. - Add `values-vgpu.yaml` for the vGPU variant with `defaultWorkload: vm-vgpu` so the chart activates the vGPU Manager DaemonSet without per-node labelling. - Add `vgpu` variant to PackageSource alongside the existing `default` (passthrough). - Rename `values-talos.yaml` → `values-passthrough.yaml` and pin `defaultWorkload: vm-passthrough` for symmetry with the vgpu variant. The overrides apply to any host OS using vfio-pci, not Talos specifically. - Refresh `hack/dcgm-default-counters.csv` to match the new dcgm-exporter image tag (`4.5.1-4.8.0`) — adds `DCGM_FI_DEV_FB_RESERVED` and other fields introduced upstream. `hack/check-gpu-recording-rules.bats` passes against the new fixture. - Add `hack/check-gpu-operator-variants.bats` that renders both variants via `helm template` and asserts the pinned ClusterPolicy values, so the next chart bump cannot silently un-pin them. - Add internal documentation for the vGPU bring-up. The `vgpu` variant assumes the user supplies a proprietary vGPU Manager image via Package CR component values — the .run cannot be redistributed. ### `vgpu` variant is experimental NVIDIA's `vgpu-device-manager` walks `/sys/class/mdev_bus/`, which does not exist on Ada Lovelace and Blackwell — the DaemonSet errors with "no parent devices found" and is disabled by default. Until an SR-IOV-aware controller is shipped, profile assignment is an out-of-band step (`echo <id> > /sys/.../current_vgpu_type` per VF) that must be re-applied after every node reboot. Without it, `permittedHostDevices.pciHostDevices` reports zero allocatable resources. The variant ships so the chart bump and the underlying skeleton (DaemonSet + image plumbing + docs) land together; production deployments need an automated profile-assignment mechanism in addition to this PR. ### KubeVirt requirement KubeVirt advertises VFs via `permittedHostDevices.pciHostDevices` rather than `mediatedDevices`. This requires kubevirt/kubevirt#16890 ("vGPU: SRIOV support", merged 2026-04-10) which is `main`-only — released tags `v1.6.x` / `v1.7.x` / `v1.8.x` do not include it. The patch is targeted at the next minor KubeVirt release (v1.9.0); track the PR for the actual release tag. Backports to released branches are not planned. ### Behavior changes inherited from upstream chart v25 → v26 The upstream chart flipped two defaults that are pinned back to `false` here, so `default` (passthrough) deployments do not silently change footprint on upgrade: - `ccManager.enabled` was flipped to `true` upstream — would auto-enable Confidential Computing on Hopper hardware (H100/H200/B100) after upgrade. - `vgpuDeviceManager.enabled` was flipped to `true` upstream — DaemonSet would render even in passthrough mode. Both stay opt-in via Package CR overrides. The new bats test guards the pin. Other inherited changes that are user-visible and worth pinning down before upgrade: - `driver.licensingConfig.configMapName` is deprecated upstream in favour of `driver.licensingConfig.secretName`. The deprecation marker lives in the CRD's `description` field (visible via `kubectl explain`) — there is no helm-render-time warning, the chart template silently prefers `secretName` and falls back to `configMapName`. Operators with existing licensing overrides should migrate the key on the next chart upgrade. - `driver.version` upstream default jumped from `570.x` to `580.x`. Invisible on the `default` variant (which sets `driver.enabled: false`), user-visible for anyone running the upstream Ubuntu driver path. - `vfioManager.image` switched from `nvcr.io/nvidia/cuda:12.8.1-base-ubi9` to `nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0`. Affects every `default` (passthrough) deployment on upgrade — different image, different layer, different image-pull surface. Worth pre-mirroring for air-gapped clusters. - `dcgmExporter.image` switched to the `-distroless` flavour. Smaller surface, fewer base-image CVEs, but no shell — operators who used to `kubectl exec` into the exporter for ad-hoc debugging will need to switch to `kubectl debug` or container ephemeral images. - `CDI` (Container Device Interface) is enabled by default in v26.3.1; this is upstream's recommended setting and is left as-is. - `node-feature-discovery` subchart source switched from the GitHub Pages Helm repo to `oci://registry.k8s.io/nfd/charts`. Air-gapped installations need to mirror the new OCI URL. ### Talos Talos is **not recommended** for the vGPU variant. NVIDIA does not publicly distribute the vGPU guest driver — it requires NVIDIA Enterprise Portal access — and Sidero closed siderolabs/extensions#461 noting that they cannot support vGPU "unless NVIDIA changes their licensing terms or provides us a way to obtain, test, and distribute the software". Building a Talos system extension that bakes in the vGPU driver is therefore not feasible without a private fork that violates the EULA. Passthrough (`default` variant) on Talos is unaffected. ### Release note ```release-note [gpu-operator] Update NVIDIA GPU Operator to v26.3.1 and add experimental `vgpu` variant for SR-IOV vGPU on Ada Lovelace / Blackwell (L4, L40, L40S, B100; Ubuntu host; requires KubeVirt v1.9.0+ for `pciHostDevices` advertising). The variant ships skeleton support — profile assignment (`current_vgpu_type` per VF) is currently out of band and must be automated by the operator before production use. On the existing `default` variant: `ccManager` and `vgpuDeviceManager` upstream defaults are pinned back to `false` to preserve prior behavior on chart upgrade; `defaultWorkload` is now pinned to `vm-passthrough` (was unset, falling back to `container`); `driver.licensingConfig.configMapName` is deprecated by upstream in favour of `secretName`; `driver.version` upstream default moves 570 → 580; `vfioManager.image` switches from CUDA UBI9 to k8s-driver-manager (relevant for air-gapped mirrors); `dcgmExporter` image switched to the `-distroless` flavour. NFD subchart source switched to `oci://registry.k8s.io/nfd/charts`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Introduced vGPU support variant with end-to-end configuration guide, validation procedures, and troubleshooting documentation * **Updates** * Upgraded GPU Operator to v26.3.1 and Node Feature Discovery to v0.18.3 * Improved component health checks and added PodDisruptionBudget availability controls <!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does Adds a practical guide for running VMs with NVIDIA vGPU on Cozystack to the existing GPU passthrough page. The guide covers the **SR-IOV vGPU path** used by current data-centre GPUs (L4, L40, L40S, B100) on the vGPU 20.x driver branch. The mediated-devices path used by older GPUs (Pascal–Ampere) is explicitly out of scope and the reader is pointed at upstream NVIDIA docs. Steps documented: - Build the proprietary vGPU Manager container image from `github.com/NVIDIA/gpu-driver-container` (the older `gitlab.com/nvidia/container-images/driver` is archived). - Deploy GPU Operator with the `vgpu` variant via Package CR (depends on cozystack/cozystack#2323). - Assign vGPU profiles to SR-IOV VFs (`current_vgpu_type` sysfs), with a periodic profile-loader DaemonSet skeleton and an explicit experimental warning. - Configure DLS licensing via ClientConfigToken (the legacy NLS / `ServerAddress=` / `ServerPort=7070` flow no longer applies). - Patch the KubeVirt CR with `permittedHostDevices.pciHostDevices` (after kubevirt/kubevirt#16890; first stable KubeVirt release with the patch is targeted at v1.9.0). - Sample `VirtualMachine` (raw `kubevirt.io/v1`) using a CDI `DataVolume` so the rootfs has room for in-VM driver install, with a `cloudInitNoCloud` disk that drops the licensing token, `gridd.conf`, an SSH key, and the build dependencies. - vGPU profile reference table for L40S with the Q/A/B suffix taxonomy. - Warning about the 2.4 GiB containerDisk root overflow during in-VM driver install (we observed `SIGBUS` from a write into an mmap of a file the kernel could no longer extend; the new example sidesteps it via `DataVolume`). - Talos is explicitly noted as not recommended for vGPU; passthrough on Talos is unaffected. ### Release note ```release-note NONE ```
|
Field report for anyone assembling the full stack on top of this: we're running it in a pilot on an 8× H200 SXM (NVSwitch) host — KubeVirt v1.9.0-beta.0, SR-IOV vGPU, currently ~40 vGPU devices across dozens of VMs, mixing MIG-backed (1g/3g) and whole-card profiles. The VF passing itself works well. The remaining gaps were all on the NVIDIA tooling side; open PRs if anyone hits the same walls: vendor-VFIO VF discovery with per-profile resource names (NVIDIA/kubevirt-gpu-device-plugin#192), NVLink fabric-partition activation for whole-card vGPU on NVSwitch (NVIDIA/kubevirt-gpu-device-plugin#193; multi-GPU NVIDIA/kubevirt-gpu-device-plugin#195), runtime profile rediscovery (NVIDIA/kubevirt-gpu-device-plugin#194), and vGPU device creation on Ada/Hopper+ (NVIDIA/vgpu-device-manager#195). Thanks for landing this. |
What this PR does
Starting with newer generation of NVIDIA cards (Ampere and newer) the hw is shipped with SRIOV support. Some Kernels maintain mdev support for this cards, while others don't. This PR enables the later Kernels where the SRIOV capability is used to slice the gpu into multiple VF(Virtual Functions) and particular profile(same as mdev) is assigned for each/some (note some vfs might not have any profile. Note2 the PF cannot be passed).
The last quirk here is that the VFs are still bind to nvidia driver rather the vfio-pci.
Now what this PR does is following:
Extend our PCI driver to accept nvidia driver, recognize if PCI device is PF, and recognize if particular VF is assigned with profile. In these conditions the PCI can be advertised and consumed by the virt-launcher.
Now I know I wrote last quirk already so I say "really last quirk" is that these VFs can support display and ramfb options but this PR left this out, intentionally. The support for display and ramfb for PCI will be implemented in later patch.
Special notes for your reviewer
E2E tests are missing as we don't have a capable HW in our CI for now. I have tested these changes against supported HW.
Release note