Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
b1b6070
CI: allow specifying custom driver versions in test matrix
leofang Jun 7, 2026
3e016b5
CI: fix Linux driver nsenter re-exec, swap Windows version, enable ci…
leofang Jun 7, 2026
c0ca869
CI: move 'Ensure GPU is working' after 'Install GPU driver' on Linux
leofang Jun 7, 2026
4a23b23
CI: flip two PR-matrix Linux rows to DRIVER=610.43.02
leofang Jun 7, 2026
d33a928
CI: restart nvidia-persistenced on Linux; poll nvidia-smi on Windows
leofang Jun 7, 2026
00896dc
CI: re-enable persistence mode after Linux driver swap
leofang Jun 7, 2026
0d5f0e9
CI: preserve SUID bit when refreshing container nvidia binaries
leofang Jun 7, 2026
3dfaa84
CI: exec nvidia-persistenced directly after Linux driver swap
leofang Jun 7, 2026
701cf2f
CI: pass --user root to nvidia-persistenced after Linux driver swap
leofang Jun 7, 2026
a3f1573
CI: add fast-feedback probe-driver-swap job (workflow_dispatch only)
leofang Jun 7, 2026
c5fef92
CI: drop workflow_dispatch gate on probe-driver-swap so it runs on ev…
leofang Jun 7, 2026
f17dd7f
CI: stop refresh_container_libs from clobbering /run/nvidia-persistenced
leofang Jun 7, 2026
6412f4f
CI: take down nvidia-persistenced via pkill, not systemctl
leofang Jun 7, 2026
2b34f1f
CI: re-bind /run/nvidia-persistenced into container after driver swap
leofang Jun 7, 2026
8d8a9ef
CI: drop install_gpu_driver.sh experiments that turned out non-load-b…
leofang Jun 7, 2026
d2c25eb
Revert: remove the probe-driver-swap fast-feedback job
leofang Jun 7, 2026
fa7940a
CI: address Mike's review comments on PR 2176
leofang Jun 8, 2026
8dfd06e
Merge branch 'main' into leof/ci-custom-driver-versions
leofang Jun 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
CI: allow specifying custom driver versions in test matrix
Extends the DRIVER field in ci/test-matrix.yml beyond 'latest'/'earliest'
to accept an explicit version string (e.g. '580.65.06'). For Linux,
ci/tools/install_gpu_driver.sh (adapted from nv-gha-runners/vm-images
PR #256) swaps the driver in-job via nsenter when the row uses a custom
version; for Windows, ci/tools/install_gpu_driver.ps1 is split into
install + configure_driver_mode, with the install step gated on the
DRIVER value and the mode step always running.

The matrix row is routed to a 'latest' runner image when the DRIVER is
a custom version (the install scripts perform the swap themselves).
Container privileges on Linux (--privileged --pid=host) are added only
on rows with a custom DRIVER. Custom DRIVER + FLAVOR=wsl is rejected
eagerly in the compute-matrix step.

Two existing nightly-numba-cuda rows exercise the new path:
- Linux amd64 / 13.3.0 / l4 -> 580.65.06
- Windows amd64 / 13.3.0 / l4 -> 610.47

Closes #293
Closes #1265
  • Loading branch information
leofang committed Jun 7, 2026
commit b1b6070a823898a01e87c49cb1682f6d65a96eb9
8 changes: 5 additions & 3 deletions .github/workflows/coverage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -275,13 +275,15 @@ jobs:
uses: nv-gha-runners/setup-proxy-cache@main
continue-on-error: true

- name: Update driver
# DRIVER above is 'latest' so install_gpu_driver.ps1 is intentionally
# skipped (it errors on latest/earliest); configure_driver_mode.ps1
# still runs to put the pre-installed driver into TCC mode.
- name: Configure driver mode
shell: powershell
env:
DRIVER_MODE: "TCC"
GPU_TYPE: "a100"
run: |
ci/tools/install_gpu_driver.ps1
ci/tools/configure_driver_mode.ps1

- name: Ensure GPU is working
run: |
Expand Down
28 changes: 24 additions & 4 deletions .github/workflows/test-wheel-linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,13 @@ jobs:
# Read base matrix from YAML file for the specific architecture
TEST_MATRIX=$(yq -o json ".linux[\"${MATRIX_TYPE}\"] | map(select(.ARCH == \"${ARCH}\"))" ci/test-matrix.yml)

# Apply matrix filter and wrap in include structure
MATRIX=$(echo "$TEST_MATRIX" | jq -c '${{ inputs.matrix_filter }} | if (. | length) > 0 then {include: .} else "Error: Empty matrix\n" | halt_error(1) end')
# Apply matrix filter; reject custom DRIVER + FLAVOR=wsl (the
# in-container driver swap doesn't work under WSL); add a
# RUNNER_DRIVER field that maps any custom version back to
# 'latest' (the install script swaps the driver itself, so we
# need to land on the runner that ships with the most recent
# pre-installed driver); wrap in include structure.
Comment thread
leofang marked this conversation as resolved.
Resolved
MATRIX=$(echo "$TEST_MATRIX" | jq -c '${{ inputs.matrix_filter }} | if any(.[]; .DRIVER != "latest" and .DRIVER != "earliest" and .FLAVOR == "wsl") then "Error: custom DRIVER is not supported with FLAVOR=wsl\n" | halt_error(1) else . end | map(. + {RUNNER_DRIVER: (if .DRIVER == "latest" or .DRIVER == "earliest" then .DRIVER else "latest" end)}) | if (. | length) > 0 then {include: .} else "Error: Empty matrix\n" | halt_error(1) end')

echo "MATRIX=${MATRIX}" | tee --append "${GITHUB_OUTPUT}"

Expand All @@ -101,13 +106,16 @@ jobs:
strategy:
fail-fast: false
matrix: ${{ fromJSON(needs.compute-matrix.outputs.MATRIX) }}
runs-on: "${{ matrix.FLAVOR || 'linux' }}-${{ matrix.ARCH }}-gpu-${{ matrix.GPU }}-${{ matrix.DRIVER }}-${{ matrix.GPU_COUNT }}"
runs-on: "${{ matrix.FLAVOR || 'linux' }}-${{ matrix.ARCH }}-gpu-${{ matrix.GPU }}-${{ matrix.RUNNER_DRIVER }}-${{ matrix.GPU_COUNT }}"
# The build stage could fail but we want the CI to keep moving.
if: ${{ github.repository_owner == 'nvidia' && !cancelled() }}
# Our self-hosted runners require a container
# TODO: use a different (nvidia?) container
container:
options: -u root --security-opt seccomp=unconfined --shm-size 16g
# Custom-DRIVER rows need --privileged --pid=host so install_gpu_driver.sh
# can nsenter to the host for the install + refresh the toolkit bind mounts
Comment thread
leofang marked this conversation as resolved.
Resolved
# back inside the container. Stock options for latest/earliest rows.
options: ${{ ((matrix.DRIVER == 'latest' || matrix.DRIVER == 'earliest') && '-u root --security-opt seccomp=unconfined --shm-size 16g') || '-u root --security-opt seccomp=unconfined --shm-size 16g --privileged --pid=host' }}
image: ubuntu:22.04
env:
NVIDIA_VISIBLE_DEVICES: ${{ env.NVIDIA_VISIBLE_DEVICES }}
Expand All @@ -131,6 +139,18 @@ jobs:
dependencies: "jq wget libgl1 libegl1 g++"
dependent_exes: "jq wget"

- name: Install GPU driver
if: ${{ matrix.DRIVER != 'latest' && matrix.DRIVER != 'earliest' }}
env:
DRIVER: ${{ matrix.DRIVER }}
GPU_TYPE: ${{ matrix.GPU }}
Comment thread
leofang marked this conversation as resolved.
Resolved
run: |
# util-linux for nsenter; install_gpu_driver.sh re-execs onto the
# host (requires --privileged --pid=host on the container, set
# conditionally above) and refreshes the toolkit bind mounts here.
apt-get -y install --no-install-recommends util-linux
Comment thread
leofang marked this conversation as resolved.
Resolved
./ci/tools/install_gpu_driver.sh

- name: Set environment variables
env:
BUILD_CUDA_VER: ${{ inputs.build-ctk-ver }}
Expand Down
20 changes: 15 additions & 5 deletions .github/workflows/test-wheel-windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,11 @@ jobs:
# Read base matrix from YAML file for the specific architecture
TEST_MATRIX=$(yq -o json ".windows[\"${MATRIX_TYPE}\"] | map(select(.ARCH == \"${ARCH}\"))" ci/test-matrix.yml)

# Apply matrix filter and wrap in include structure
MATRIX=$(echo "$TEST_MATRIX" | jq -c '${{ inputs.matrix_filter }} | if (. | length) > 0 then {include: .} else "Error: Empty matrix\n" | halt_error(1) end')
# Apply matrix filter; add a RUNNER_DRIVER field that maps any
# custom DRIVER version back to 'latest' (install_gpu_driver.ps1
# swaps the driver itself, so the runner must be the one that
# ships the most recent pre-installed driver); wrap in include.
MATRIX=$(echo "$TEST_MATRIX" | jq -c '${{ inputs.matrix_filter }} | map(. + {RUNNER_DRIVER: (if .DRIVER == "latest" or .DRIVER == "earliest" then .DRIVER else "latest" end)}) | if (. | length) > 0 then {include: .} else "Error: Empty matrix\n" | halt_error(1) end')

echo "MATRIX=${MATRIX}" | tee --append "${GITHUB_OUTPUT}"

Expand All @@ -95,7 +98,7 @@ jobs:
fail-fast: false
matrix: ${{ fromJSON(needs.compute-matrix.outputs.MATRIX) }}
if: ${{ github.repository_owner == 'nvidia' && !cancelled() }}
runs-on: "windows-${{ matrix.ARCH }}-gpu-${{ matrix.GPU }}-${{ matrix.DRIVER }}-${{ matrix.GPU_COUNT }}"
runs-on: "windows-${{ matrix.ARCH }}-gpu-${{ matrix.GPU }}-${{ matrix.RUNNER_DRIVER }}-${{ matrix.GPU_COUNT }}"
steps:
- name: Checkout ${{ github.event.repository.name }}
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
Expand All @@ -106,13 +109,20 @@ jobs:
with:
enable-apt: true

- name: Update driver
- name: Install GPU driver
if: ${{ matrix.DRIVER != 'latest' && matrix.DRIVER != 'earliest' }}
env:
DRIVER_MODE: ${{ matrix.DRIVER_MODE }}
DRIVER: ${{ matrix.DRIVER }}
GPU_TYPE: ${{ matrix.GPU }}
run: |
ci/tools/install_gpu_driver.ps1

- name: Configure driver mode
env:
DRIVER_MODE: ${{ matrix.DRIVER_MODE }}
run: |
ci/tools/configure_driver_mode.ps1

- name: Ensure GPU is working
run: |
nvidia-smi
Expand Down
13 changes: 11 additions & 2 deletions ci/test-matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,16 @@
# Windows entries also include DRIVER_MODE.
#
# Notes:
# - DRIVER accepts:
# * 'latest' - use the runner's pre-installed latest driver (no install step)
# * 'earliest' - use the runner's pre-installed earliest driver (no install step)
# * a version string (e.g. '580.65.06')
# - install that version via ci/tools/install_gpu_driver.sh (Linux)
# or ci/tools/install_gpu_driver.ps1 (Windows) at the start of the
# job. The matrix row is routed to the 'latest' runner image (the
# install scripts swap the driver themselves).
# - DRIVER: 'earliest' does not work with CUDA 12.9.1
# - DRIVER: a custom version is not supported with FLAVOR=wsl on Linux.

linux:
pull-request:
Expand Down Expand Up @@ -74,7 +83,7 @@ linux:
- { MODE: 'nightly-pytorch', ARCH: 'arm64', PY_VER: '3.12', CUDA_VER: '13.0.2', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: 'latest', TORCH_VER: '2.9.1', TORCH_CUDA: 'cu130' }
# nightly-numba-cuda
- { MODE: 'nightly-numba-cuda', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '12.9.1', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: 'latest' }
- { MODE: 'nightly-numba-cuda', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '13.3.0', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: 'latest' }
- { MODE: 'nightly-numba-cuda', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '13.3.0', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: '580.65.06' }

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

- { MODE: 'nightly-numba-cuda', ARCH: 'arm64', PY_VER: '3.12', CUDA_VER: '12.9.1', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: 'latest' }
- { MODE: 'nightly-numba-cuda', ARCH: 'arm64', PY_VER: '3.12', CUDA_VER: '13.3.0', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: 'latest' }
# nightly-standard (arm64 l4×2 — nightly-only per runner team request)
Expand Down Expand Up @@ -113,4 +122,4 @@ windows:
- { MODE: 'nightly-pytorch', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '13.0.2', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: 'latest', DRIVER_MODE: 'TCC', TORCH_VER: '2.9.1', TORCH_CUDA: 'cu130' }
# nightly-numba-cuda
- { MODE: 'nightly-numba-cuda', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '12.9.1', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: 'latest', DRIVER_MODE: 'TCC' }
- { MODE: 'nightly-numba-cuda', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '13.3.0', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: 'latest', DRIVER_MODE: 'TCC' }
- { MODE: 'nightly-numba-cuda', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '13.3.0', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: '610.47', DRIVER_MODE: 'TCC' }
45 changes: 45 additions & 0 deletions ci/tools/configure_driver_mode.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0
#
# configure_driver_mode.ps1 -- set the NVIDIA driver mode on a Windows CI
# runner and cycle the display devices so the new mode takes effect
# without rebooting. Always runs (whether or not install_gpu_driver.ps1
# just ran). When install_gpu_driver.ps1 has run, this single device
# cycle also activates the freshly-installed driver.
#
# Inputs (env):
# DRIVER_MODE One of WDDM, TCC, MCDM.

function Set-DriverMode {

# Map matrix DRIVER_MODE to nvidia-smi -fdm code.
# This assumes we have the prior knowledge on which GPU can use which mode.
$driver_mode = $env:DRIVER_MODE
if ($driver_mode -eq "WDDM") {
Write-Output "Setting driver mode to WDDM..."
nvidia-smi -fdm 0
} elseif ($driver_mode -eq "TCC") {
Write-Output "Setting driver mode to TCC..."
nvidia-smi -fdm 1
} elseif ($driver_mode -eq "MCDM") {
Write-Output "Setting driver mode to MCDM..."
nvidia-smi -fdm 2
} else {
Write-Output "Unknown driver mode: $driver_mode"
exit 1
}

# Only restart NVIDIA display adapters, not other display devices (e.g. QEMU VGA)
$nvidia_devices = Get-PnpDevice -Class Display -FriendlyName "NVIDIA*"
foreach ($device in $nvidia_devices) {
Write-Output "Restarting device: $($device.FriendlyName) ($($device.InstanceId))"
pnputil /disable-device "$($device.InstanceId)"
pnputil /enable-device "$($device.InstanceId)"
}
Comment thread
leofang marked this conversation as resolved.
Resolved
# Give it a minute to settle:
Start-Sleep -Seconds 5
}

# Run the functions
Set-DriverMode
51 changes: 21 additions & 30 deletions ci/tools/install_gpu_driver.ps1
Original file line number Diff line number Diff line change
@@ -1,13 +1,30 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0
#
# install_gpu_driver.ps1 -- install a specific NVIDIA driver version on a
# Windows CI runner. Driver-mode selection and the post-install device
# power-cycle are the responsibility of configure_driver_mode.ps1, which
# the workflow runs immediately after this script (or by itself when
# DRIVER is 'latest'/'earliest' and the runner already brings up the
# right driver).
#
# Inputs (env):
# DRIVER Driver version, e.g. "610.47". Must NOT be 'latest' or
# 'earliest' -- those are runner-pre-installed and the
# workflow is expected to skip this script for them.
# GPU_TYPE Lower-case GPU label from the matrix (e.g. "l4", "rtx4090").
# Selects the data-center vs desktop installer variant.

# Install the driver
function Install-Driver {

# Set the correct URL, filename, and arguments to the installer
# This driver is picked to support Windows 11 & CUDA 13.0
$version = '581.15'
# Driver version is plumbed from the matrix via the DRIVER env var.
$version = $env:DRIVER
if (-not $version -or $version -eq 'latest' -or $version -eq 'earliest') {
Write-Error "DRIVER env var must be a specific version string (e.g. '610.47'); got '$version'."
exit 1
}

# Get GPU type from environment variable
$gpu_type = $env:GPU_TYPE
Expand Down Expand Up @@ -54,33 +71,7 @@ function Install-Driver {
# Install the file with the specified path from earlier
Write-Output 'Running the driver installer...'
Start-Process -FilePath $filepath -ArgumentList $install_args -Wait
Write-Output 'Done!'

# Handle driver mode configuration
# This assumes we have the prior knowledge on which GPU can use which mode.
$driver_mode = $env:DRIVER_MODE
if ($driver_mode -eq "WDDM") {
Write-Output "Setting driver mode to WDDM..."
nvidia-smi -fdm 0
} elseif ($driver_mode -eq "TCC") {
Write-Output "Setting driver mode to TCC..."
nvidia-smi -fdm 1
} elseif ($driver_mode -eq "MCDM") {
Write-Output "Setting driver mode to MCDM..."
nvidia-smi -fdm 2
} else {
Write-Output "Unknown driver mode: $driver_mode"
exit 1
}
# Only restart NVIDIA display adapters, not other display devices (e.g. QEMU VGA)
$nvidia_devices = Get-PnpDevice -Class Display -FriendlyName "NVIDIA*"
foreach ($device in $nvidia_devices) {
Write-Output "Restarting device: $($device.FriendlyName) ($($device.InstanceId))"
pnputil /disable-device "$($device.InstanceId)"
pnputil /enable-device "$($device.InstanceId)"
}
# Give it a minute to settle:
Start-Sleep -Seconds 5
Write-Output 'Install complete; driver mode + device cycle handled by configure_driver_mode.ps1.'
}

# Run the functions
Expand Down
Loading
Loading