Skip to content

vGPU: SRIOV support - #16890

Merged
kubevirt-bot merged 5 commits into
kubevirt:mainfrom
xpivarc:sriov-vgpu
Apr 10, 2026
Merged

kubevirt-bot merged 5 commits into
kubevirt:mainfrom
xpivarc:sriov-vgpu

Conversation

@xpivarc

@xpivarc xpivarc commented Feb 19, 2026

Copy link
Copy Markdown
Member

What this PR does

Starting with newer generation of NVIDIA cards (Ampere and newer) the hw is shipped with SRIOV support. Some Kernels maintain mdev support for this cards, while others don't. This PR enables the later Kernels where the SRIOV capability is used to slice the gpu into multiple VF(Virtual Functions) and particular profile(same as mdev) is assigned for each/some (note some vfs might not have any profile. Note2 the PF cannot be passed).
The last quirk here is that the VFs are still bind to nvidia driver rather the vfio-pci.

Now what this PR does is following:
Extend our PCI driver to accept nvidia driver, recognize if PCI device is PF, and recognize if particular VF is assigned with profile. In these conditions the PCI can be advertised and consumed by the virt-launcher.

Now I know I wrote last quirk already so I say "really last quirk" is that these VFs can support display and ramfb options but this PR left this out, intentionally. The support for display and ramfb for PCI will be implemented in later patch.

Special notes for your reviewer

E2E tests are missing as we don't have a capable HW in our CI for now. I have tested these changes against supported HW.

Release note

KubeVirt's PCI device plugin now supports passing of pre-setup VF (vGPU)

@kubevirt-bot kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Feb 19, 2026
@dosubot dosubot Bot added the area/handler label Feb 19, 2026

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 3 issues, and left some high level feedback:

  • In discoverPermittedHostPCIDevices, you call IsPhysicalFunction multiple times per device (inside the nvidia branch and again afterwards); consider calling it once, storing the result, and reusing it to avoid redundant sysfs reads and simplify the control flow.
  • HasVGPUProfile currently treats any non-empty, non-"0" value as a valid profile; parsing current_vgpu_type as an integer and explicitly handling parse errors would make this more robust against unexpected file contents.
  • The sysfs path components for vGPU detection ("nvidia" and "current_vgpu_type") are duplicated between the implementation and tests; consider defining shared constants to avoid divergence if the layout needs to change later.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In discoverPermittedHostPCIDevices, you call IsPhysicalFunction multiple times per device (inside the nvidia branch and again afterwards); consider calling it once, storing the result, and reusing it to avoid redundant sysfs reads and simplify the control flow.
- HasVGPUProfile currently treats any non-empty, non-"0" value as a valid profile; parsing current_vgpu_type as an integer and explicitly handling parse errors would make this more robust against unexpected file contents.
- The sysfs path components for vGPU detection ("nvidia" and "current_vgpu_type") are duplicated between the implementation and tests; consider defining shared constants to avoid divergence if the layout needs to change later.

## Individual Comments

### Comment 1
<location> `pkg/virt-handler/device-manager/common.go:216-217` </location>
<code_context>
+// IsPhysicalFunction checks if a PCI device is an SR-IOV Physical Function by
+// looking for virtfn* subdirectories in its sysfs path. Only Physical Functions
+// have these subdirectories representing their associated Virtual Functions.
+func (h *DeviceUtilsHandler) IsPhysicalFunction(basepath, pciAddress string) bool {
+	entries, err := os.ReadDir(filepath.Join(basepath, pciAddress))
+	if err != nil {
+		return false
</code_context>

<issue_to_address>
**suggestion (bug_risk):** Consider distinguishing between "not a PF" and "failed to inspect" instead of returning false on any error.

Currently any `os.ReadDir` error is treated the same as “not a Physical Function”, so permission issues, transient I/O errors, or path bugs become indistinguishable from a genuine “no virtfn* entries” case. That can hide misconfiguration and make failures hard to diagnose. Consider returning a `(bool, error)` or at least logging non-`os.IsNotExist` errors so unexpected failures are visible while still treating the normal “directory missing” case as non-PF.

Suggested implementation:

```golang
 // IsPhysicalFunction checks if a PCI device is an SR-IOV Physical Function by
 // looking for virtfn* subdirectories in its sysfs path. Only Physical Functions
 // have these subdirectories representing their associated Virtual Functions.
func (h *DeviceUtilsHandler) IsPhysicalFunction(basepath, pciAddress string) (bool, error) {
	dirPath := filepath.Join(basepath, pciAddress)

	entries, err := os.ReadDir(dirPath)
	if err != nil {
		// Treat the normal "directory does not exist" case as "not a PF".
		if os.IsNotExist(err) {
			return false, nil
		}
		// Propagate unexpected errors so callers can distinguish inspection
		// failures from "not a PF".
		return false, fmt.Errorf("failed to read PCI device directory %q: %w", dirPath, err)
	}

	for _, entry := range entries {
		if entry.IsDir() && strings.HasPrefix(entry.Name(), "virtfn") {
			return true, nil
		}
	}

	return false, nil
}

```

1. Ensure the file imports `fmt` (and `os`, `filepath`, `strings` if they are not already imported), e.g.:
   - `import "fmt"`
2. Update all call sites of `IsPhysicalFunction` to handle the new `(bool, error)` signature, for example:
   - `isPF, err := h.IsPhysicalFunction(basepath, pciAddress)`
   - Handle `err` appropriately (log/propagate) and use `isPF` only when `err == nil`.
</issue_to_address>

### Comment 2
<location> `pkg/virt-launcher/virtwrap/device/hostdevice/gpu/hostdev.go:38` </location>
<code_context>
 	DefaultDisplayOn             = true
 )

+// TODO: Pass in also medata so metada can be constructed also in tests, include if display is possible
 func CreateHostDevices(vmiGPUs []v1.GPU) ([]api.HostDevice, error) {
 	return CreateHostDevicesFromPools(vmiGPUs, NewPCIAddressPool(vmiGPUs), NewMDEVAddressPool(vmiGPUs))
</code_context>

<issue_to_address>
**nitpick (typo):** Fix minor typos in the TODO comment for clarity.

"medata" and "metada" should both be "metadata" to keep the TODO clear for future readers.

```suggestion
 // TODO: Pass in also metadata so metadata can be constructed also in tests, include if display is possible
```
</issue_to_address>

### Comment 3
<location> `pkg/virt-handler/device-manager/pci_device.go:268` </location>
<code_context>
 		if resourceName, supported := supportedPCIDeviceMap[pciID]; supported {
 			// check device driver
 			driver, err := handler.GetDeviceDriver(pciBasePath, info.Name())
-			if err != nil || driver != "vfio-pci" {
</code_context>

<issue_to_address>
**issue (complexity):** Consider simplifying this logic by caching the physical-function check and moving the driver/vGPU decision into a dedicated helper to keep the walk callback flat and readable.

You can reduce the added complexity by (1) caching the PF check and (2) extracting the driver/vGPU policy into a small helper. That keeps the walk callback linear while preserving behavior.

### 1. Cache `IsPhysicalFunction` result

Avoid calling `IsPhysicalFunction` twice and scattering PF logic:

```go
isPF := handler.IsPhysicalFunction(pciBasePath, info.Name())
```

Use `isPF` in all checks instead of calling `IsPhysicalFunction` again.

### 2. Extract driver/vGPU decision into a helper

Move the branching policy into a focused function that returns a decision + reason, and handle logging once in the walk:

```go
type pciDecision int

const (
	pciUse pciDecision = iota
	pciSkip
)

func shouldUsePCIDevice(
	handler HandlerType, // use the actual type
	pciBasePath, devName, pciID, driver string,
	isPF bool,
) (pciDecision, string) {
	if driver == "vfio-pci" {
		if isPF {
			return pciSkip, fmt.Sprintf("Skipping SR-IOV Physical Function %s (%s)", devName, pciID)
		}
		return pciUse, ""
	}

	if driver == "nvidia" {
		if isPF {
			return pciSkip, fmt.Sprintf("Skipping non-vfio-pci Physical Function %s (%s)", devName, pciID)
		}
		if !handler.HasVGPUProfile(pciBasePath, devName) {
			return pciSkip, fmt.Sprintf(
				"Skipping non-vfio-pci device without vGPU profile %s (%s, driver: %s)",
				devName, pciID, driver,
			)
		}
		return pciUse, fmt.Sprintf("Found vGPU-capable VF %s (%s, driver: %s)", devName, pciID, driver)
	}

	return pciSkip, fmt.Sprintf("Not supported driver %s, pci %s", driver, pciID)
}
```

Then the walk callback becomes flatter:

```go
pciID, err := handler.GetDevicePCIID(pciBasePath, info.Name())
if err != nil {
	log.DefaultLogger().Reason(err).Errorf("failed get vendor:device ID for device: %s", info.Name())
	return nil
}

if resourceName, supported := supportedPCIDeviceMap[pciID]; supported {
	driver, err := handler.GetDeviceDriver(pciBasePath, info.Name())
	if err != nil {
		log.DefaultLogger().Reason(err).Errorf("failed to get driver for device: %s", info.Name())
		return nil
	}

	isPF := handler.IsPhysicalFunction(pciBasePath, info.Name())

	decision, msg := shouldUsePCIDevice(handler, pciBasePath, info.Name(), pciID, driver, isPF)
	if decision == pciSkip {
		if msg != "" {
			log.DefaultLogger().Infof("%s", msg)
		}
		return nil
	}
	if msg != "" {
		log.DefaultLogger().Infof("%s", msg)
	}

	pcidev := &PCIDevice{
		pciID:      pciID,
		pciAddress: info.Name(),
	}
	iommuGroup, err := handler.GetDeviceIOMMUGroup(pciBasePath, info.Name())
	if err != nil {
		return nil
	}
	pcidev.iommuGroup = iommuGroup
	pcidev.driver = driver
	pcidev.numaNode = handler.GetDeviceNumaNode(pciBasePath, info.Name())
	pciDevicesMap[resourceName] = append(pciDevicesMap[resourceName], pcidev)
} else {
	log.DefaultLogger().V(9).Infof("Not supported %s", pciID)
}
```

This keeps all existing behavior (including all logging messages and vGPU rules) but removes nested driver branches, deduplicates PF checks, and centralizes policy in a small, testable helper.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

DefaultDisplayOn = true
)

// TODO: Pass in also medata so metada can be constructed also in tests, include if display is possible

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick (typo): Fix minor typos in the TODO comment for clarity.

"medata" and "metada" should both be "metadata" to keep the TODO clear for future readers.

Suggested change
// TODO: Pass in also medata so metada can be constructed also in tests, include if display is possible
// TODO: Pass in also metadata so metadata can be constructed also in tests, include if display is possible

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, exactly — that TODO comment.

If you want to fix it inline, it would look like this:

// TODO: Pass in also metadata so metadata can be constructed also in tests, include if display is possible

@xpivarc

xpivarc commented Feb 20, 2026

Copy link
Copy Markdown
Member Author

@dhiller Are you tracking this problem https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/16890/pull-kubevirt-e2e-k8s-1.35-sig-compute-serial/2024514878953230336
No logs because of

couldn't get current server API group list: Get \"https://127.0.0.1:44329/api?timeout=32s\": dial tcp 127.0.0.1:44329: connect: connection refused"

pcidev.numaNode = handler.GetDeviceNumaNode(pciBasePath, info.Name())
pciDevicesMap[resourceName] = append(pciDevicesMap[resourceName], pcidev)
} else {
log.DefaultLogger().V(9).Infof("Not supported %s", pciID)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
However, V(9) sounds a bit high for this log.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any suggestion? This log fires a lot of times that's why I thing 9 is good here.


// For devices without vfio-pci driver, check if they're NVIDIA vGPU VFs
if driver != "vfio-pci" {
if driver == "nvidia" {

@Barakmor1 Barakmor1 Feb 24, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and instead of this:

if driver == "nvidia" {

// Just force the driver to NOT vfio-pci to ensure they all get ignored.
mockPCI.EXPECT().GetDeviceIOMMUGroup(pciBasePath, gomock.Any()).AnyTimes()
mockPCI.EXPECT().GetDeviceDriver(pciBasePath, gomock.Any()).Return("definitely-not-vfio-pci", nil).AnyTimes()
mockPCI.EXPECT().GetDeviceDriver(pciBasePath, gomock.Any()).Return("vfio-pci", nil).AnyTimes()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain why we need this?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dropped this. @jean-edouard Wat is the rational to not return vfio-pci?

Comment on lines 302 to 303
if err != nil {
return nil

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not related to the pr but I wonder if we should add a log here

Comment on lines +217 to +228
entries, err := os.ReadDir(filepath.Join(basepath, pciAddress))
if err != nil {
return false
}

for _, entry := range entries {
if strings.HasPrefix(entry.Name(), "virtfn") {
return true
}
}

return false

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can just check if sriov_totalvfs file exist iiuc it should only exist when this is PF

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me :)

@Barakmor1

Copy link
Copy Markdown
Member

I think we should add unit tests for discoverPermittedHostPCIDevices as well


// For devices without vfio-pci driver, check if they're NVIDIA vGPU VFs
if driver != "vfio-pci" {
if driver == "nvidia" {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is one of my major concerns ... I'm really worried that we will start accumulating these vendor-specific implementations...

Can you please isolate all this into a helper function, something like shouldPermitNonVfioPCIDevice?

I think we need to follow up with a plugable checker - I'm not sure right now how to implement this, but we could have some kind of a DeviceEligibilityChecker interface
, which will have a ShouldPermit function. We could then have a vendor dedicated pci/checkers/nvidia.go - each one will register itself...
something like that.

@dhiller

dhiller commented Mar 2, 2026

Copy link
Copy Markdown
Contributor

connect: connection refused

We've been seeing clustered failures with this symptom for a while on the serial lane now - example: https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/directory/pull-kubevirt-e2e-k8s-1.34-sig-compute-serial/2013237621370654720

Tracker #16608

@xpivarc

xpivarc commented Mar 5, 2026

Copy link
Copy Markdown
Member Author

@vladikr PTAL

return pciDevicesMap
}

func shouldPermitNvidia(driver, pciBasePath, pciAddress, pciID string) bool {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:(
I think we should avoid using vendor names in functions...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to somehow separate all of these vendor specifc validations
maybe we can create a folder of these validators for each vendor?
something like pkg/virt-handler/device-manager/pci/validators/nvidia.go
Then on the top of the file (around line 275) we will have a generic isDeviceEligible(pcidev, handler) that will loop over these external validators.
Half baked idea... I'll try to write it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is vendor name better in file name than function?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same answer tbh... """because this way it's separate and cleaner - do you really want to spread this all over the codebase? to me this is crazy."""
I would prefer to consolidate this instead of spreading this all over the code base.
This way, there could be different validators for different vendors, and this is clearer and consolidated, rather than multiple functions with vendor names

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vladikr reworked this, PTAL

Comment on lines +275 to 286
// For devices without vfio-pci driver, check if they're NVIDIA vGPU VFs
if driver != "vfio-pci" {
if !shouldPermitNvidia(driver, pciBasePath, info.Name(), pciID) {
log.DefaultLogger().V(9).Infof("Not supported driver %s, pci %s", driver, pciID)
return nil
}
}

pcidev := &PCIDevice{
pciID: pciID,
pciAddress: info.Name(),
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xpivarc
Here is what I can suggest. Create a generic pluggable validator, so each vendor will be separated into its own file and more could be added.

I think we can create these in
pkg/virt-handler/device-manager/pci/validator.go
pkg/virt-handler/device-manager/pci/validators/nvidia.go

Here is what I had in mind:
https://gist.github.com/vladikr/297fd699327e58d8562647f0394c8150

and then here we could do something like:

pcidev := &PCIDevice{
				pciID:      pciID,
				pciAddress: info.Name(),
				driver:     driver,
			}

			if !isDeviceEligible(pcidev, handler) {
				log.DefaultLogger().V(9).Infof("Not supported driver %s, pci %s", driver, pciID)
				return nil
			}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it really worth it? what is the difference?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because this way it's separate and cleaner - do you really want to spread this all over the codebase? to me this is crazy.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly, it's a very simple change - If you want, I can write the code, but I think that this is 100% worth it - as a long term investmet
The last thing this project needs is spreading vendor specific code around the code base - this was my take for a very long time

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree we should not spread it all over the place, can we compromise to move the logic under specific file but not creating unnecessary abstraction?

xpivarc added 3 commits March 30, 2026 18:18
Replace all instances of `strings.Replace(str, old, new, -1)` with
`strings.ReplaceAll(str, old, new)` in mediated_devices_types.go.
This is the idiomatic way to express "replace all occurrences" since
Go 1.12 and improves code readability.

Signed-off-by: Luboslav Pivarc <[email protected]>
Make sure that tests are isolated and the handler
variable is always reverted.

Signed-off-by: Luboslav Pivarc <[email protected]>
This commit adds following functions to existing `DeviceHandler`:
1. HasVGPUProfile - returns true if VF has a vGPU assigned (nvidia only)
2. IsPhysicalFunction - returns true if PCI device is SRIOV PF

Signed-off-by: Luboslav Pivarc <[email protected]>
Some PCI devices can now also enable Display and RamFB,
this is left out to implement.

Signed-off-by: Luboslav Pivarc <[email protected]>
@kubevirt-bot kubevirt-bot added the sig/buildsystem Denotes an issue or PR that relates to changes in the build system. label Mar 31, 2026

@vladikr vladikr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xpivarc Thank you :)

Overall, I like this change, but you know my view, I think it needs to be more generic. I would still introduce validators as a follow up. Functions like shouldPermitNvidia are misleading in my view.

}

func shouldPermitNonVFIOPCI(driver, pciBasePath, pciAddress, pciID string) bool {
return shouldPermitNvidia(driver, pciBasePath, pciAddress, pciID)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you consider a follow up to make this generic?

@vladikr

vladikr commented Apr 2, 2026

Copy link
Copy Markdown
Member

/approve

@kubevirt-bot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vladikr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubevirt-bot kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 2, 2026
@xpivarc

xpivarc commented Apr 8, 2026

Copy link
Copy Markdown
Member Author

/retest

Comment on lines +306 to +309

func shouldPermitNonVFIOPCI(driver, pciBasePath, pciAddress, pciID string) bool {
return shouldPermitNvidia(driver, pciBasePath, pciAddress, pciID)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why this wrap?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was asked by Vladik to have a generic place where we can then evolve with more hw support, e.g amd :)

@xpivarc xpivarc mentioned this pull request Apr 10, 2026
8 tasks
@fossedihelm

Copy link
Copy Markdown
Contributor

/lgtm

@kubevirt-bot kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Apr 10, 2026
@kubevirt-commenter-bot

Copy link
Copy Markdown

Required labels detected, running phase 2 presubmits:
/test pull-kubevirt-e2e-k8s-1.34-windows2016
/test pull-kubevirt-e2e-kind-1.35-vgpu
/test pull-kubevirt-e2e-kind-sriov
/test pull-kubevirt-e2e-k8s-1.35-ipv6-sig-network
/test pull-kubevirt-e2e-k8s-1.34-sig-network
/test pull-kubevirt-e2e-k8s-1.34-sig-storage
/test pull-kubevirt-e2e-k8s-1.34-sig-compute
/test pull-kubevirt-e2e-k8s-1.34-sig-operator

@kubevirt-commenter-bot

Copy link
Copy Markdown

/retest-required
This bot automatically retries required jobs that failed/flaked on
required test lanes of PRs.
Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@kubevirt-bot
kubevirt-bot merged commit efe3433 into kubevirt:main Apr 10, 2026
39 checks passed
@oshoval

oshoval commented Apr 14, 2026

Copy link
Copy Markdown
Contributor

Nice

Note please that once we discuss MDEV vGPU emulation, it case we like the direction, we can do the same for SR-IOV vGPU,

For that will be great if you have a manifest / e2e test to share when time comes, even quick and dirty one that you tested on the real HW,
anyhow it will take some time until wrapping current tasks first

Thanks

lexfrei added a commit to cozystack/website that referenced this pull request Apr 30, 2026
The original draft documented mediated devices (mdev) — that path
does not apply to current NVIDIA data-centre GPUs (L4, L40, L40S,
B100). On the vGPU 17/20 driver branch these GPUs use SR-IOV with
per-VF NVIDIA sysfs; KubeVirt advertises VFs via
permittedHostDevices.pciHostDevices, not mediatedDevices.

This commit rewrites the vGPU section accordingly:

- Add a driver-model note at the top making the mdev/SR-IOV split
  explicit. Pascal–Ampere readers are sent to upstream NVIDIA docs.
- Replace mediatedDeviceTypes / mediatedDevices / gpus: with
  pciHostDevices / hostDevices throughout.
- Document KubeVirt v1.9.0 (ETA July 2026, kubevirt/kubevirt#16890
  shipping in that release) as a hard prerequisite — current released
  tags v1.6.x/1.7.x/1.8.x do not include the patch and backports are
  not planned.
- Replace the legacy NLS block (ServerAddress=, ServerPort=7070,
  FeatureType=1) with the DLS ClientConfigToken flow.
- Switch build instructions to the upstream NVIDIA repo
  (github.com/NVIDIA/gpu-driver-container) — the gitlab repo is
  archived.
- Note explicitly that Talos is not recommended for vGPU (NVIDIA
  redistribution restrictions, siderolabs/extensions#461 closed
  won't-fix); passthrough on Talos is unaffected.
- Add a warning that a 2.4 GiB containerDisk overlay is too small
  to install the GRID guest driver in-place — recommend a CDI
  DataVolume of 20 GiB+ for non-throwaway tests.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
lexfrei added a commit to cozystack/website that referenced this pull request Apr 30, 2026
The original draft documented mediated devices (mdev) — that path
does not apply to current NVIDIA data-centre GPUs (L4, L40, L40S,
B100). On the vGPU 17/20 driver branch these GPUs use SR-IOV with
per-VF NVIDIA sysfs; KubeVirt advertises VFs via
permittedHostDevices.pciHostDevices, not mediatedDevices.

This commit rewrites the vGPU section accordingly:

- Add a driver-model note at the top making the mdev/SR-IOV split
  explicit. Pascal–Ampere readers are sent to upstream NVIDIA docs.
- Replace mediatedDeviceTypes / mediatedDevices / gpus: with
  pciHostDevices / hostDevices throughout.
- Document KubeVirt v1.9.0 (ETA July 2026, kubevirt/kubevirt#16890
  shipping in that release) as a hard prerequisite — current released
  tags v1.6.x/1.7.x/1.8.x do not include the patch and backports are
  not planned.
- Replace the legacy NLS block (ServerAddress=, ServerPort=7070,
  FeatureType=1) with the DLS ClientConfigToken flow.
- Switch build instructions to the upstream NVIDIA repo
  (github.com/NVIDIA/gpu-driver-container) — the gitlab repo is
  archived.
- Note explicitly that Talos is not recommended for vGPU (NVIDIA
  redistribution restrictions, siderolabs/extensions#461 closed
  won't-fix); passthrough on Talos is unaffected.
- Add a warning that a 2.4 GiB containerDisk overlay is too small
  to install the GRID guest driver in-place — recommend a CDI
  DataVolume of 20 GiB+ for non-throwaway tests.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
lexfrei added a commit to cozystack/cozystack that referenced this pull request May 7, 2026
…2323)

## What this PR does

Updates the NVIDIA GPU Operator from v25.3.0 to v26.3.1 and adds an
**experimental** `vgpu` variant for SR-IOV vGPU on Ada Lovelace and
newer GPUs.

Changes:
- Update vendored upstream chart to v26.3.1 (CDI enabled by default,
component version bumps).
- Pin chart version in Makefile for reproducibility.
- Add `values-vgpu.yaml` for the vGPU variant with `defaultWorkload:
vm-vgpu` so the chart activates the vGPU Manager DaemonSet without
per-node labelling.
- Add `vgpu` variant to PackageSource alongside the existing `default`
(passthrough).
- Rename `values-talos.yaml` → `values-passthrough.yaml` and pin
`defaultWorkload: vm-passthrough` for symmetry with the vgpu variant.
The overrides apply to any host OS using vfio-pci, not Talos
specifically.
- Refresh `hack/dcgm-default-counters.csv` to match the new
dcgm-exporter image tag (`4.5.1-4.8.0`) — adds `DCGM_FI_DEV_FB_RESERVED`
and other fields introduced upstream.
`hack/check-gpu-recording-rules.bats` passes against the new fixture.
- Add `hack/check-gpu-operator-variants.bats` that renders both variants
via `helm template` and asserts the pinned ClusterPolicy values, so the
next chart bump cannot silently un-pin them.
- Add internal documentation for the vGPU bring-up.

The `vgpu` variant assumes the user supplies a proprietary vGPU Manager
image via Package CR component values — the .run cannot be
redistributed.

### `vgpu` variant is experimental

NVIDIA's `vgpu-device-manager` walks `/sys/class/mdev_bus/`, which does
not exist on Ada Lovelace and Blackwell — the DaemonSet errors with "no
parent devices found" and is disabled by default. Until an SR-IOV-aware
controller is shipped, profile assignment is an out-of-band step (`echo
<id> > /sys/.../current_vgpu_type` per VF) that must be re-applied after
every node reboot. Without it, `permittedHostDevices.pciHostDevices`
reports zero allocatable resources. The variant ships so the chart bump
and the underlying skeleton (DaemonSet + image plumbing + docs) land
together; production deployments need an automated profile-assignment
mechanism in addition to this PR.

### KubeVirt requirement

KubeVirt advertises VFs via `permittedHostDevices.pciHostDevices` rather
than `mediatedDevices`. This requires kubevirt/kubevirt#16890 ("vGPU:
SRIOV support", merged 2026-04-10) which is `main`-only — released tags
`v1.6.x` / `v1.7.x` / `v1.8.x` do not include it. The patch is targeted
at the next minor KubeVirt release (v1.9.0); track the PR for the actual
release tag. Backports to released branches are not planned.

### Behavior changes inherited from upstream chart v25 → v26

The upstream chart flipped two defaults that are pinned back to `false`
here, so `default` (passthrough) deployments do not silently change
footprint on upgrade:

- `ccManager.enabled` was flipped to `true` upstream — would auto-enable
Confidential Computing on Hopper hardware (H100/H200/B100) after
upgrade.
- `vgpuDeviceManager.enabled` was flipped to `true` upstream — DaemonSet
would render even in passthrough mode.

Both stay opt-in via Package CR overrides. The new bats test guards the
pin.

Other inherited changes that are user-visible and worth pinning down
before upgrade:

- `driver.licensingConfig.configMapName` is deprecated upstream in
favour of `driver.licensingConfig.secretName`. The deprecation marker
lives in the CRD's `description` field (visible via `kubectl explain`) —
there is no helm-render-time warning, the chart template silently
prefers `secretName` and falls back to `configMapName`. Operators with
existing licensing overrides should migrate the key on the next chart
upgrade.
- `driver.version` upstream default jumped from `570.x` to `580.x`.
Invisible on the `default` variant (which sets `driver.enabled: false`),
user-visible for anyone running the upstream Ubuntu driver path.
- `vfioManager.image` switched from
`nvcr.io/nvidia/cuda:12.8.1-base-ubi9` to
`nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0`. Affects every
`default` (passthrough) deployment on upgrade — different image,
different layer, different image-pull surface. Worth pre-mirroring for
air-gapped clusters.
- `dcgmExporter.image` switched to the `-distroless` flavour. Smaller
surface, fewer base-image CVEs, but no shell — operators who used to
`kubectl exec` into the exporter for ad-hoc debugging will need to
switch to `kubectl debug` or container ephemeral images.
- `CDI` (Container Device Interface) is enabled by default in v26.3.1;
this is upstream's recommended setting and is left as-is.
- `node-feature-discovery` subchart source switched from the GitHub
Pages Helm repo to `oci://registry.k8s.io/nfd/charts`. Air-gapped
installations need to mirror the new OCI URL.

### Talos

Talos is **not recommended** for the vGPU variant. NVIDIA does not
publicly distribute the vGPU guest driver — it requires NVIDIA
Enterprise Portal access — and Sidero closed siderolabs/extensions#461
noting that they cannot support vGPU "unless NVIDIA changes their
licensing terms or provides us a way to obtain, test, and distribute the
software". Building a Talos system extension that bakes in the vGPU
driver is therefore not feasible without a private fork that violates
the EULA. Passthrough (`default` variant) on Talos is unaffected.

### Release note

```release-note
[gpu-operator] Update NVIDIA GPU Operator to v26.3.1 and add experimental `vgpu` variant for SR-IOV vGPU on Ada Lovelace / Blackwell (L4, L40, L40S, B100; Ubuntu host; requires KubeVirt v1.9.0+ for `pciHostDevices` advertising). The variant ships skeleton support — profile assignment (`current_vgpu_type` per VF) is currently out of band and must be automated by the operator before production use. On the existing `default` variant: `ccManager` and `vgpuDeviceManager` upstream defaults are pinned back to `false` to preserve prior behavior on chart upgrade; `defaultWorkload` is now pinned to `vm-passthrough` (was unset, falling back to `container`); `driver.licensingConfig.configMapName` is deprecated by upstream in favour of `secretName`; `driver.version` upstream default moves 570 → 580; `vfioManager.image` switches from CUDA UBI9 to k8s-driver-manager (relevant for air-gapped mirrors); `dcgmExporter` image switched to the `-distroless` flavour. NFD subchart source switched to `oci://registry.k8s.io/nfd/charts`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Introduced vGPU support variant with end-to-end configuration guide,
validation procedures, and troubleshooting documentation

* **Updates**
* Upgraded GPU Operator to v26.3.1 and Node Feature Discovery to v0.18.3
* Improved component health checks and added PodDisruptionBudget
availability controls

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
lexfrei added a commit to cozystack/website that referenced this pull request May 7, 2026
## What this PR does

Adds a practical guide for running VMs with NVIDIA vGPU on Cozystack to
the existing GPU passthrough page.

The guide covers the **SR-IOV vGPU path** used by current data-centre
GPUs (L4, L40, L40S, B100) on the vGPU 20.x driver branch. The
mediated-devices path used by older GPUs (Pascal–Ampere) is explicitly
out of scope and the reader is pointed at upstream NVIDIA docs.

Steps documented:

- Build the proprietary vGPU Manager container image from
`github.com/NVIDIA/gpu-driver-container` (the older
`gitlab.com/nvidia/container-images/driver` is archived).
- Deploy GPU Operator with the `vgpu` variant via Package CR (depends on
cozystack/cozystack#2323).
- Assign vGPU profiles to SR-IOV VFs (`current_vgpu_type` sysfs), with a
periodic profile-loader DaemonSet skeleton and an explicit experimental
warning.
- Configure DLS licensing via ClientConfigToken (the legacy NLS /
`ServerAddress=` / `ServerPort=7070` flow no longer applies).
- Patch the KubeVirt CR with `permittedHostDevices.pciHostDevices`
(after kubevirt/kubevirt#16890; first stable KubeVirt release with the
patch is targeted at v1.9.0).
- Sample `VirtualMachine` (raw `kubevirt.io/v1`) using a CDI
`DataVolume` so the rootfs has room for in-VM driver install, with a
`cloudInitNoCloud` disk that drops the licensing token, `gridd.conf`, an
SSH key, and the build dependencies.
- vGPU profile reference table for L40S with the Q/A/B suffix taxonomy.
- Warning about the 2.4 GiB containerDisk root overflow during in-VM
driver install (we observed `SIGBUS` from a write into an mmap of a file
the kernel could no longer extend; the new example sidesteps it via
`DataVolume`).
- Talos is explicitly noted as not recommended for vGPU; passthrough on
Talos is unaffected.

### Release note

```release-note
NONE
```
@lexfrei

lexfrei commented Jul 13, 2026

Copy link
Copy Markdown

Field report for anyone assembling the full stack on top of this: we're running it in a pilot on an 8× H200 SXM (NVSwitch) host — KubeVirt v1.9.0-beta.0, SR-IOV vGPU, currently ~40 vGPU devices across dozens of VMs, mixing MIG-backed (1g/3g) and whole-card profiles. The VF passing itself works well.

The remaining gaps were all on the NVIDIA tooling side; open PRs if anyone hits the same walls: vendor-VFIO VF discovery with per-profile resource names (NVIDIA/kubevirt-gpu-device-plugin#192), NVLink fabric-partition activation for whole-card vGPU on NVSwitch (NVIDIA/kubevirt-gpu-device-plugin#193; multi-GPU NVIDIA/kubevirt-gpu-device-plugin#195), runtime profile rediscovery (NVIDIA/kubevirt-gpu-device-plugin#194), and vGPU device creation on Ada/Hopper+ (NVIDIA/vgpu-device-manager#195).

Thanks for landing this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/handler area/launcher dco-signoff: yes Indicates the PR's author has DCO signed all their commits. lgtm Indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/buildsystem Denotes an issue or PR that relates to changes in the build system. sig/compute size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants