Skip to content

Commit 2b34f1f

Browse files
leofangleofang
committed
CI: re-bind /run/nvidia-persistenced into container after driver swap
The container's bind mount of /run/nvidia-persistenced/ is taken at container-start time and pinned to the host directory's then-current inode. Across the install the host directory gets recreated under a fresh inode (the daemon's shutdown + restart cycle replaces it), and the container is stranded on the deleted inode -- socket file shows up with link count 0 inside the container, NVML state-changing calls return NVML_ERROR_UNKNOWN. After refresh_container_libs, umount the stale bind, mkdir the local mount point if missing, and re-bind from /proc/1/root/run/nvidia- persistenced (the host's current view via the privileged container's host-pid-ns access). CAP_SYS_ADMIN required, which custom-DRIVER rows already grant via --privileged --pid=host.
1 parent 6412f4f commit 2b34f1f

1 file changed

Lines changed: 18 additions & 0 deletions

File tree

‎ci/tools/install_gpu_driver.sh‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -227,6 +227,24 @@ if [ -z "${_NVDRV_NSENTERED:-}" ] && in_container; then
227227
_NVDRV_NSENTERED=1 nsenter -t 1 -m -p -n -i -u -- bash -s < "$0" \
228228
|| { echo "::error::container needs 'options: --privileged --pid=host'" >&2; exit 1; }
229229
refresh_container_libs
230+
231+
# Re-bind /run/nvidia-persistenced from host. The container's original
232+
# bind mount of this dir was taken at container-start time and points
233+
# to the host's then-current inode. Even with `pkill` (instead of
234+
# systemctl) the host dir is recreated by the new daemon under a fresh
235+
# inode -- leaving the container's bind mount stranded on a deleted
236+
# inode (socket file shows up with link count 0). Re-do the bind mount
237+
# so in-container NVML clients see the live daemon endpoint. Needs
238+
# CAP_SYS_ADMIN, which we get from the --privileged --pid=host the
239+
# workflow adds for custom-DRIVER rows.
240+
if [ -d /proc/1/root/run/nvidia-persistenced ]; then
241+
set -x
242+
umount /run/nvidia-persistenced 2>/dev/null || true
243+
mkdir -p /run/nvidia-persistenced
244+
mount --bind /proc/1/root/run/nvidia-persistenced /run/nvidia-persistenced
245+
ls -la /run/nvidia-persistenced/ >&2
246+
set +x
247+
fi
230248
else
231249
host_install
232250
fi

0 commit comments

Comments
 (0)