5/12/22, 6:16 PM Configuring GPU Pass-Through for NVIDIA cards | Virtualization Guide :: Reader View
[Link]
/documentation/leap/virtualization/html/book-virtualization/[Link]
Configuring GPU Pass-Through for NVIDIA cards | Virtualization Guide
9-12 minutes
This article describes how to assign an NVIDIA GPU graphics card on the
host machine to a virtualized guest.
GPU pass-through is supported on the AMD64/Intel 64 architecture only.
The host operating system needs to be SLES 12 SP3 or newer.
This article deals with a set of instructions based on V100/T1000 NVIDIA
cards, and is meant for GPU computation purposes only.
Verify that you are using an NVIDIA Tesla product—Maxwell, Pascal,
or Volta.
To be able to manage the host system, you need an additional display card
on the host that you can use when configuring the GPU Pass-Through, or a
functional SSH env
1. Verify that the host operating system is SLES 12 SP3 or newer:
tux >
cat /etc/issue
Welcome to SUSE Linux Enterprise Server 15 (x86_64) - Kernel \r (\l).
2. Verify that the host supports VT-d
technology and that it is already enabled in the firmware settings:
tux >
dmesg | grep -e "Directed I/O"
[ 12.819760] DMAR: Intel(R) Virtualization Technology for Directed I/O
If VT-d is not enabled in the firmware, enable it and reboot the host.
3. Verify that the host has an extra GPU or VGA card:
tux >
lspci | grep -i "vga"
07:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. \
MGA G200e [Pilot] ServerEngines (SEP1) (rev 05)
With a Tesla V100 card:
tux >
lspci | grep -i nvidia
03:00.0 3D controller: NVIDIA Corporation GV100 [Tesla V100 PCIe] (rev a1)
With a T1000 Mobile (available on Dell 5540):
tux >
lspci | grep -i nvidia
01:00.0 3D controller: NVIDIA Corporation TU117GLM [Quadro T1000 Mobile] (rev a1)
IOMMU is disabled by default. You need
to enable it at boot time in the /etc/default/grub
configuration file.
1. For Intel-based hosts:
GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt [Link]=vfio-pci"
For AMD-based hosts:
GRUB_CMDLINE_LINUX="iommu=pt amd_iommu=on [Link]=vfio-pci"
2. When you save the modified /etc/default/grub file,
re-generate the main GRUB 2 configuration file
/boot/grub2/[Link]:
tux >
sudo
grub2-mkconfig -o /boot/grub2/[Link]
3. Reboot the host and verify that IOMMU
is enabled:
tux >
dmesg | grep -e DMAR -e IOMMU
Because we want to assign the NVIDIA card to a VM guest, we need to avoid
use of the card by the host OS's built-in driver for NVIDIA GPUs. The open
source NVIDIA driv
nouveau. Edit the file /etc/modprobe.d/[Link] and append the
following line to its end:
1. Find the card vendor and model IDs. Utilize the bus number identified in
Section A.3.1, “Verify the host environment”, for example
03:00.0:
tux >
lspci -nn | grep 03:00.0
03:00.0 3D controller [0302]: NVIDIA Corporation GV100 [Tesla V100 PCIe] [10de:1db4] (rev a1)
chrome-extension://ecabifbgmdmgdllomnfinbmaellmclnh/data/reader/[Link]?id=583&url=https%3A%2F%[Link].… 1/4
5/12/22, 6:16 PM Configuring GPU Pass-Through for NVIDIA cards | Virtualization Guide :: Reader View
2. Create the file [Link] in the
/etc/modprobe.d/ directory with the following
content:
options vfio-pci ids=10de:1db4
Note
Verify that your card does not need an extra ids=
parameter. For some cards, you must specify the audio device too,
so that device's ID must also be added to the list, oth
will not be able to use the card.
There are three ways you can load the
VFIO driver.
1. Create the file
/etc/[Link].d/[Link] and add the
following content:
add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"
2. Re-generate the initrd file:
tux >
sudo
dracut --force /boot/initrd $(uname -r)
Create the file /etc/modules-load.d/[Link] and
add the following content:
pci_stub
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_intel
To load the driver manually at run-time, execute the following command:
tux >
sudo
modprobe vfio-pci
For Microsoft Windows guests, we recommend disabling MSR (model-specific register) to
avoid the guest crashing. Create the file
/etc/modprobe.d/[Link] and add th
content:
options kvm ignore_msrs=1
For proper GPU Pass-Through functionality, the host needs to boot using UEFI
firmware (that is, not using a legacy-style BIOS boot sequence).
1. Install the qemu-ovmf package which includes UEFI
firmware images:
tux >
sudo
zypper install qemu-ovmf
2. Get the list of OVMF bin and vars
files by filtering the results of the following command:
3. Enable OVMF in the libvirt QEMU configuration in the file
/etc/libvirt/[Link] by using the list obtained
from the previous step. It should look similar to the fo
nvram = [
"/usr/share/qemu/ovmf-x86_64-[Link]:/usr/share/qemu/ovmf-x86_64-[Link]",
"/usr/share/qemu/ovmf-x86_64-[Link]:/usr/share/qemu/ovmf-x86_64-[Link]",
"/usr/share/qemu/ovmf-x86_64-[Link]:/usr/share/qemu/ovmf-x86_64-[Link]",
"/usr/share/qemu/ovmf-x86_64-[Link]:/usr/share/qemu/ovmf-x86_64-[Link]",
"/usr/share/qemu/ovmf-x86_64-[Link]:/usr/share/qemu/ovmf-x86_64-[Link]",
"/usr/share/qemu/ovmf-x86_64-[Link]:/usr/share/qemu/ovmf-x86_64-[Link]",
"/usr/share/qemu/ovmf-x86_64-[Link]:/usr/share/qemu/ovmf-x86_64-[Link]",
"/usr/share/qemu/ovmf-x86_64-[Link]:/usr/share/qemu/ovmf-x86_64-[Link]",
"/usr/share/qemu/ovmf-x86_64-[Link]:/usr/share/qemu/ovmf-x86_64-[Link]",
"/usr/share/qemu/ovmf-x86_64-[Link]:/usr/share/qemu/ovmf-x86_64-[Link]",
"/usr/share/qemu/ovmf-x86_64-[Link]:/usr/share/qemu/ovmf-x86_64-[Link]",
"/usr/share/qemu/ovmf-x86_64-[Link]:/usr/share/qemu/ovmf-x86_64-[Link]",
"/usr/share/qemu/ovmf-x86_64-[Link]:/usr/share/qemu/ovmf-x86_64-[Link]",
For most of the changes in the above steps to take effect, you need to
reboot the host machine:
tux >
sudo
shutdown -r now
This section describes how to configure the guest virtual machine so that it
can use the host's NVIDIA GPU. Use Virtual Machine Manager or
virt-install to install the gue
more details
in Chapter 9, Guest installation.
chrome-extension://ecabifbgmdmgdllomnfinbmaellmclnh/data/reader/[Link]?id=583&url=https%3A%2F%[Link].… 2/4
5/12/22, 6:16 PM Configuring GPU Pass-Through for NVIDIA cards | Virtualization Guide :: Reader View
During the guest VM installation, select and configure the following devices:
Use Q35 chipset if possible.
Install the guest VM using UEFI firmware.
Add the following emulated devices:
Graphic: Spice or VNC
Device: qxl, VGA, or Virtio
Find more information in Section 13.6, “Video”.
Add the host PCI device (03:00.0 in our example) to
the guest. Find more information in
Section 13.12, “Assigning a host PCI device to a VM Guest”.
For best performance, we recommend using virtio drivers for the network
card and storage.
Procedure A.1: RPM-based distributions Report Documentation Bug #
1. Download the driver RPM package from
[Link]
2. Install the downloaded RPM package:
tux >
sudo
rpm -i nvidia-diag-driver-local-repo-sles123-390.30-1.0-1.x86_64.rpm
3. Refresh repositories and install cuda-drivers. This
step will be different for non-SUSE distributions:
tux >
sudo
zypper refresh && zypper install cuda-drivers
4. Reboot the guest VM:
tux >
sudo
shutdown -r now
Procedure A.2: Generic installer Report Documentation Bug #
1. Because the installer needs to compile the NVIDIA driver modules,
install the gcc-c++ and kernel-devel
packages.
2. Disable Secure Boot on the guest, because NVIDIA's driver modules are
unsigned. On SUSE distributions, you can use the YaST GRUB 2 module
to disable Secure Boot
information in
Book “Reference”, Chapter 14 “UEFI (Unified Extensible Firmware Interface)”, Section 14.1.1 “Implementation on openSUSE Leap”.
3. Download the driver installation script from
[Link]
make it executable, and run it to complete the driver
installation:
tux >
chmod +x NVIDIA-Linux-x86_64-[Link]
tux >
sudo
./NVIDIA-Linux-x86_64-[Link]
4. Download CUDA drivers from
[Link]
and install following the on-screen instructions.
Note: Display issues
After you have installed the NVIDIA drivers, the Virtual Machine Manager display will
lose its connection to the guest OS. To access the guest VM, you
must either login via ssh
the
console interface, or install a dedicated VNC server in the guest.
To avoid a flickering screen, stop and disable the display manager:
tux >
sudo
systemctl stop display-manager && systemctl disable display-manager
Procedure A.3: Testing the Linux driver installation Report Documentation Bug #
1. Change the directory to the CUDA sample templates:
tux >
cd /usr/local/cuda-9.1/samples/0_Simple/simpleTemplates
2. Compile and run the simpleTemplates file:
chrome-extension://ecabifbgmdmgdllomnfinbmaellmclnh/data/reader/[Link]?id=583&url=https%3A%2F%[Link].… 3/4
5/12/22, 6:16 PM Configuring GPU Pass-Through for NVIDIA cards | Virtualization Guide :: Reader View
tux >
make && ./simpleTemplates
runTest<float,32>
GPU Device 0: "Tesla V100-PCIE-16GB" with compute capability 7.0
CUDA device [Tesla V100-PCIE-16GB] has 80 Multi-Processors
Processing time: 495.006000 (ms)
Compare OK
runTest<int,64>
GPU Device 0: "Tesla V100-PCIE-16GB" with compute capability 7.0
CUDA device [Tesla V100-PCIE-16GB] has 80 Multi-Processors
Processing time: 0.203000 (ms)
Compare OK
[simpleTemplates] -> Test Results: 0 Failures
chrome-extension://ecabifbgmdmgdllomnfinbmaellmclnh/data/reader/[Link]?id=583&url=https%3A%2F%[Link].… 4/4
