Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
b1b6070
CI: allow specifying custom driver versions in test matrix
leofang Jun 7, 2026
3e016b5
CI: fix Linux driver nsenter re-exec, swap Windows version, enable ci…
leofang Jun 7, 2026
c0ca869
CI: move 'Ensure GPU is working' after 'Install GPU driver' on Linux
leofang Jun 7, 2026
4a23b23
CI: flip two PR-matrix Linux rows to DRIVER=610.43.02
leofang Jun 7, 2026
d33a928
CI: restart nvidia-persistenced on Linux; poll nvidia-smi on Windows
leofang Jun 7, 2026
00896dc
CI: re-enable persistence mode after Linux driver swap
leofang Jun 7, 2026
0d5f0e9
CI: preserve SUID bit when refreshing container nvidia binaries
leofang Jun 7, 2026
3dfaa84
CI: exec nvidia-persistenced directly after Linux driver swap
leofang Jun 7, 2026
701cf2f
CI: pass --user root to nvidia-persistenced after Linux driver swap
leofang Jun 7, 2026
a3f1573
CI: add fast-feedback probe-driver-swap job (workflow_dispatch only)
leofang Jun 7, 2026
c5fef92
CI: drop workflow_dispatch gate on probe-driver-swap so it runs on ev…
leofang Jun 7, 2026
f17dd7f
CI: stop refresh_container_libs from clobbering /run/nvidia-persistenced
leofang Jun 7, 2026
6412f4f
CI: take down nvidia-persistenced via pkill, not systemctl
leofang Jun 7, 2026
2b34f1f
CI: re-bind /run/nvidia-persistenced into container after driver swap
leofang Jun 7, 2026
8d8a9ef
CI: drop install_gpu_driver.sh experiments that turned out non-load-b…
leofang Jun 7, 2026
d2c25eb
Revert: remove the probe-driver-swap fast-feedback job
leofang Jun 7, 2026
fa7940a
CI: address Mike's review comments on PR 2176
leofang Jun 8, 2026
8dfd06e
Merge branch 'main' into leof/ci-custom-driver-versions
leofang Jun 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
CI: fix Linux driver nsenter re-exec, swap Windows version, enable ci…
….yml dispatch

- install_gpu_driver.sh: pipe the script body to the host-side bash via
  stdin (bash -s < "$0") instead of re-execing "$0". The script lives
  in the GH workspace mount (container-only), so the relative path
  doesn't resolve after nsenter switches the mount namespace.
  The < "$0" fd is opened before nsenter and survives the flip.
- test-matrix.yml: Windows nightly-numba-cuda row 610.47 -> 596.36
  (610.47 isn't published on the CDN; install hit 404).
- ci.yml: add workflow_dispatch: trigger so the pipeline can be
  re-run manually. The existing should-skip / detect-changes gates
  already handle non-PR events.
  • Loading branch information
leofang committed Jun 7, 2026
commit 3e016b572dcded7701f9b2f12c25cec3cb7e5b1d
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ on:
schedule:
# every 24 hours at midnight UTC
- cron: "0 0 * * *"
workflow_dispatch:
Comment thread
leofang marked this conversation as resolved.
Resolved

jobs:
ci-vars:
Expand Down
2 changes: 1 addition & 1 deletion ci/test-matrix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -122,4 +122,4 @@ windows:
- { MODE: 'nightly-pytorch', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '13.0.2', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: 'latest', DRIVER_MODE: 'TCC', TORCH_VER: '2.9.1', TORCH_CUDA: 'cu130' }
# nightly-numba-cuda
- { MODE: 'nightly-numba-cuda', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '12.9.1', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: 'latest', DRIVER_MODE: 'TCC' }
- { MODE: 'nightly-numba-cuda', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '13.3.0', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: '610.47', DRIVER_MODE: 'TCC' }
- { MODE: 'nightly-numba-cuda', ARCH: 'amd64', PY_VER: '3.12', CUDA_VER: '13.3.0', LOCAL_CTK: '0', GPU: 'l4', GPU_COUNT: '1', DRIVER: '596.36', DRIVER_MODE: 'TCC' }

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8 changes: 7 additions & 1 deletion ci/tools/install_gpu_driver.sh
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,13 @@ refresh_container_libs() {
}

if [ -z "${_NVDRV_NSENTERED:-}" ] && in_container; then
_NVDRV_NSENTERED=1 nsenter -t 1 -m -p -n -i -u -- "$0" \
# Re-exec on the host. The runner-team's `nvgha-driver` script lives at a
# host-side absolute path so `"$0"` survives the mount-namespace flip;
# ours lives in the GH workspace mount (container-only), so we pipe the
# script body in via stdin instead -- the `< "$0"` fd is opened before
# nsenter and stays valid across the namespace switch. Env vars (DRIVER,
# GPU_TYPE, _NVDRV_NSENTERED) are inherited by the host-side bash.
_NVDRV_NSENTERED=1 nsenter -t 1 -m -p -n -i -u -- bash -s < "$0" \
|| { echo "::error::container needs 'options: --privileged --pid=host'" >&2; exit 1; }
refresh_container_libs
else
Expand Down
Loading