Skip to content

Commit d33a928

Browse files
committed
CI: restart nvidia-persistenced on Linux; poll nvidia-smi on Windows
Linux: After install_gpu_driver.sh stops nvidia-persistenced and the apt purge removes the package, the .run installer reinstalls the systemd service but leaves it stopped. cuda.core's test_persistence_mode_enabled fails with NVML_ERROR_UNKNOWN on driver 610.43.02 when the daemon is not running; explicitly start it again at the end of host_install(). Windows: configure_driver_mode.ps1's trailing 'Start-Sleep -Seconds 5' is not enough on slower-coming-back-up multi-GPU rows (observed: 2x H100 MCDM). Replace it with a poll-until-success loop on nvidia-smi with a 60s deadline, matching the runner-team nvgha-driver.ps1 pattern. Previously masked because every Windows row used to run the full install pipeline; with custom-DRIVER plumbing, latest/earliest rows skip the install and the cycle is no longer preceded by warm-up time.
1 parent 4a23b23 commit d33a928

2 files changed

Lines changed: 24 additions & 2 deletions

File tree

‎ci/tools/configure_driver_mode.ps1‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,21 @@ function Set-DriverMode {
3737
pnputil /disable-device "$($device.InstanceId)"
3838
pnputil /enable-device "$($device.InstanceId)"
3939
}
40-
# Give it a minute to settle:
41-
Start-Sleep -Seconds 5
40+
41+
# Poll nvidia-smi until NVML can initialize, or give up after ~60s.
42+
# A fixed sleep is not enough on slower-coming-back-up multi-GPU rows
43+
# (e.g. 2x H100 MCDM) where pnputil enable returns before NVML is
44+
# ready. Pattern borrowed from the runner-team `nvgha-driver.ps1`.
45+
Write-Output "Waiting for nvidia-smi/NVML to come back up after device cycle..."
46+
$deadline = (Get-Date).AddSeconds(60)
47+
do {
48+
Start-Sleep -Seconds 2
49+
& nvidia-smi.exe 2>&1 | Out-Null
50+
} while ($LASTEXITCODE -ne 0 -and (Get-Date) -lt $deadline)
51+
if ($LASTEXITCODE -ne 0) {
52+
Write-Error "nvidia-smi did not return cleanly within 60s of the device cycle"
53+
exit 1
54+
}
4255
}
4356

4457
# Run the functions

‎ci/tools/install_gpu_driver.sh‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,15 @@ host_install() {
114114
&& sh installer.run --silent --dkms --no-questions \
115115
--accept-license --ui=none --no-cc-version-check --kernel-module-type="$KMT" )
116116
modprobe nvidia nvidia_uvm nvidia_modeset
117+
118+
# Restore nvidia-persistenced. We stopped it before the install (and the
119+
# purge may have removed it); the .run installer reinstalls the service.
120+
# Some NVML calls -- e.g. nvmlDeviceSetPersistenceMode -- can fail with
121+
# NVML_ERROR_UNKNOWN on newer drivers when the daemon isn't running, and
122+
# cuda.core's test_persistence_mode_enabled trips on that.
123+
if systemctl list-unit-files 2>/dev/null | grep -q '^nvidia-persistenced\.service'; then
124+
systemctl start nvidia-persistenced || true
125+
fi
117126
}
118127

119128
# Replace the toolkit's bind-mounted nvidia libs/binaries inside this

0 commit comments

Comments
 (0)