Skip to content

Commit 00896dc

Browse files
leofangleofang
committed
CI: re-enable persistence mode after Linux driver swap
Runner-latest L4 images come up with Persistence-M=On (set somewhere in the runner team's image setup, not in cuda-python). Our .run install leaves it Off, which breaks cuda.core's test_persistence_mode_enabled on driver 610.43.02 -- the test calls device.is_persistence_mode_enabled = False on a device that already reports False, and 610.43.02 returns NVML_ERROR_UNKNOWN for that no-op set. Restore the runner baseline by calling `nvidia-smi -pm 1` at the end of host_install() (sets the kernel persistence flag directly via NVML). Also daemon-reload + start nvidia-persistenced.service best-effort so tools that look for the daemon find it; `set -x` around this trailing block so the next run's log confirms which lines fired.
1 parent d33a928 commit 00896dc

1 file changed

Lines changed: 17 additions & 8 deletions

File tree

‎ci/tools/install_gpu_driver.sh‎

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -115,14 +115,23 @@ host_install() {
115115
--accept-license --ui=none --no-cc-version-check --kernel-module-type="$KMT" )
116116
modprobe nvidia nvidia_uvm nvidia_modeset
117117

118-
# Restore nvidia-persistenced. We stopped it before the install (and the
119-
# purge may have removed it); the .run installer reinstalls the service.
120-
# Some NVML calls -- e.g. nvmlDeviceSetPersistenceMode -- can fail with
121-
# NVML_ERROR_UNKNOWN on newer drivers when the daemon isn't running, and
122-
# cuda.core's test_persistence_mode_enabled trips on that.
123-
if systemctl list-unit-files 2>/dev/null | grep -q '^nvidia-persistenced\.service'; then
124-
systemctl start nvidia-persistenced || true
125-
fi
118+
# Restore the runner image's baseline state: persistence mode ENABLED
119+
# plus nvidia-persistenced running. The runner-team's pre-installed
120+
# drivers come up with `Persistence-M: On`, but our .run install leaves
121+
# it Off, which breaks tests that toggle the value (cuda.core's
122+
# test_persistence_mode_enabled hits NVML_ERROR_UNKNOWN when setting
123+
# the mode to its current value on driver 610.43.02).
124+
#
125+
# `nvidia-smi -pm 1` is the load-bearing call -- it sets the kernel-
126+
# level persistence flag directly via NVML (equivalent to what the
127+
# daemon would do on startup). The systemctl block is best-effort: the
128+
# silent .run installer doesn't always drop the systemd unit, so we
129+
# daemon-reload first and tolerate failure on `start`.
130+
set -x
131+
nvidia-smi -pm 1 || true
132+
systemctl daemon-reload 2>/dev/null || true
133+
systemctl start nvidia-persistenced.service 2>/dev/null || true
134+
set +x
126135
}
127136

128137
# Replace the toolkit's bind-mounted nvidia libs/binaries inside this

0 commit comments

Comments
 (0)