Skip to content

Commit 303fbda

Browse files
juenglinrwgk
juenglin
and
rwgk
authored
ci: install mini-CTK CUPTI under extras/CUPTI and check pathfinder finds it (#3440)
* ci: install mini-CTK CUPTI under extras/CUPTI and check pathfinder finds it fetch_ctk unpacked the cuda_cupti redistrib archive into the top-level toolkit tree, so libcupti landed in lib64 (Linux) or lib/ (Windows). The CTK installer and cuda.pathfinder use extras/CUPTI instead, so pathfinder could not find CUPTI through CUDA_PATH. CI only passed because LD_LIBRARY_PATH found it first, and the default pathfinder tests tolerate missing libraries. Unpack cuda_cupti into extras/CUPTI. On Linux, rename its lib to lib64 as for the top-level tree. On Windows, do the same for archives before 13.4, which put the DLLs directly in lib/; later archives already use lib/<arch>. Tag the cache key with the layout so old caches are not restored. Add a check to the LOCAL_CTK test jobs that fails unless pathfinder loads NVRTC and CUPTI from under CUDA_PATH. * ci: skip the pathfinder CTK check when the checkout predates it Nightly runs check out the older commit their wheels were built from. That commit may lack both the checker and the mini-CTK layout it expects, so run the check only when the checked-out tree contains it. --------- Co-authored-by: Ralf W. Grosse-Kunstleve <[email protected]>
1 parent d93a33c commit 303fbda

4 files changed

Lines changed: 85 additions & 9 deletions

File tree

‎.github/actions/fetch_ctk/action.yml‎

Lines changed: 28 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -118,9 +118,11 @@ runs:
118118
# prefixes its output line with "\", which would end up in the cache key.
119119
MANIFEST_SHA256="$(sha256sum < "$CTK_JSON_FILE" | awk '{print $1}')"
120120
HASH=$(echo -n "${CTK_CACHE_COMPONENTS}" | sha256sum | awk '{print $1}')
121+
# Bump the layout tag (l2) whenever the unpacked tree changes shape, so
122+
# caches saved with an older layout are not restored.
121123
{
122-
echo "CTK_CACHE_KEY=mini-ctk-${{ inputs.cuda-version }}-${{ inputs.host-platform }}-$HASH-m$MANIFEST_SHA256"
123-
echo "CTK_CACHE_FILENAME=mini-ctk-${{ inputs.cuda-version }}-${{ inputs.host-platform }}-$HASH-m$MANIFEST_SHA256.tar.gz"
124+
echo "CTK_CACHE_KEY=mini-ctk-l2-${{ inputs.cuda-version }}-${{ inputs.host-platform }}-$HASH-m$MANIFEST_SHA256"
125+
echo "CTK_CACHE_FILENAME=mini-ctk-l2-${{ inputs.cuda-version }}-${{ inputs.host-platform }}-$HASH-m$MANIFEST_SHA256.tar.gz"
124126
echo "CTK_CACHE_COMPONENTS=${CTK_CACHE_COMPONENTS}"
125127
echo "CTK_BASE_URL=${CTK_BASE_URL}"
126128
echo "CTK_JSON_FILE=${CTK_JSON_FILE}"
@@ -151,16 +153,16 @@ runs:
151153
# "Set up CTK cache variable".
152154
if [[ "${{ inputs.host-platform }}" == linux* ]]; then
153155
function extract() {
154-
tar -xvf $1 -C $CACHE_TMP_DIR --strip-components=1
156+
tar -xvf $1 -C $2 --strip-components=1
155157
}
156158
elif [[ "${{ inputs.host-platform }}" == win* ]]; then
157159
function extract() {
158160
_TEMP_DIR_=$(mktemp -d)
159161
unzip $1 -d $_TEMP_DIR_
160-
cp -r $_TEMP_DIR_/*/* $CACHE_TMP_DIR
162+
cp -r $_TEMP_DIR_/*/* $2
161163
rm -rf $_TEMP_DIR_
162164
# see commit NVIDIA/cuda-python@69410f1d9228e775845ef6c8b4a9c7f37ffc68a5
163-
chmod 644 $CACHE_TMP_DIR/LICENSE
165+
chmod 644 $2/LICENSE
164166
}
165167
fi
166168
function populate_cuda_path() {
@@ -183,17 +185,34 @@ runs:
183185
--component "$CTK_COMPONENT" \
184186
--metadata-path "$CTK_JSON_FILE")"
185187
echo "${CTK_COMPONENT_SHA256} ${CTK_COMPONENT_COMPONENT_FILENAME}" | sha256sum --check --strict -
186-
extract $CTK_COMPONENT_COMPONENT_FILENAME
188+
# cuda.pathfinder looks for CUPTI only under extras/CUPTI, the CTK installer
189+
# layout; the redistrib archive itself has a flat include/ + lib/ layout.
190+
CTK_COMPONENT_DEST="$CACHE_TMP_DIR"
191+
if [[ "$CTK_COMPONENT" == cuda_cupti ]]; then
192+
CTK_COMPONENT_DEST="$CACHE_TMP_DIR/extras/CUPTI"
193+
fi
194+
mkdir -p "$CTK_COMPONENT_DEST"
195+
extract $CTK_COMPONENT_COMPONENT_FILENAME "$CTK_COMPONENT_DEST"
187196
rm $CTK_COMPONENT_COMPONENT_FILENAME
188197
}
189198
190199
# Get headers and shared libraries in place
191200
for item in $(echo $CTK_CACHE_COMPONENTS | tr ',' ' '); do
192201
populate_cuda_path "$item"
193202
done
194-
# TODO: check Windows
195-
if [[ "${{ inputs.host-platform }}" == linux* && -d "${CACHE_TMP_DIR}/lib" ]]; then
196-
mv $CACHE_TMP_DIR/lib $CACHE_TMP_DIR/lib64
203+
CUPTI_DIR="$CACHE_TMP_DIR/extras/CUPTI"
204+
if [[ "${{ inputs.host-platform }}" == linux* ]]; then
205+
for lib_parent in "$CACHE_TMP_DIR" "$CUPTI_DIR"; do
206+
if [[ -d "$lib_parent/lib" ]]; then
207+
mv "$lib_parent/lib" "$lib_parent/lib64"
208+
fi
209+
done
210+
elif [[ "${{ inputs.host-platform }}" == win* ]]; then
211+
# CUPTI archives before 13.4 put the DLLs directly in lib/, where the
212+
# CTK installer uses extras/CUPTI/lib64; later ones use lib/<arch>.
213+
if [[ -d "$CUPTI_DIR/lib" && ! -d "$CUPTI_DIR/lib/x64" && ! -d "$CUPTI_DIR/lib/arm64" ]]; then
214+
mv "$CUPTI_DIR/lib" "$CUPTI_DIR/lib64"
215+
fi
197216
fi
198217
ls -l $CACHE_TMP_DIR
199218

‎.github/workflows/test-wheel-linux.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -411,6 +411,12 @@ jobs:
411411
CUDA_PATHFINDER_TEST_FIND_NVIDIA_BITCODE_LIB_STRICTNESS: see_what_works
412412
run: run-tests pathfinder
413413

414+
# The checker and the mini-CTK layout it expects both come from the
415+
# inputs.sha checkout. Nightly runs may check out a commit without them.
416+
- name: Check cuda.pathfinder loads NVRTC and CUPTI from the mini CTK
417+
if: ${{ inputs.test-mode == 'standard' && env.TEST_PATHFINDER == 'true' && matrix.LOCAL_CTK == '1' && hashFiles('ci/tools/check_pathfinder_ctk_libs.py') != '' }}
418+
run: python ci/tools/check_pathfinder_ctk_libs.py nvrtc cupti
419+
414420
- name: Run cuda.bindings tests
415421
if: ${{ inputs.test-mode == 'standard' && env.TEST_BINDINGS == 'true' && env.SKIP_CUDA_BINDINGS_TEST == '0' }}
416422
env:

‎.github/workflows/test-wheel-windows.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -367,6 +367,13 @@ jobs:
367367
shell: bash --noprofile --norc -xeuo pipefail {0}
368368
run: run-tests pathfinder
369369

370+
# The checker and the mini-CTK layout it expects both come from the
371+
# inputs.sha checkout. Nightly runs may check out a commit without them.
372+
- name: Check cuda.pathfinder loads NVRTC and CUPTI from the mini CTK
373+
if: ${{ inputs.test-mode == 'standard' && env.TEST_PATHFINDER == 'true' && matrix.LOCAL_CTK == '1' && hashFiles('ci/tools/check_pathfinder_ctk_libs.py') != '' }}
374+
shell: bash --noprofile --norc -xeuo pipefail {0}
375+
run: python ci/tools/check_pathfinder_ctk_libs.py nvrtc cupti
376+
370377
- name: Run cuda.bindings tests
371378
if: ${{ inputs.test-mode == 'standard' && env.TEST_BINDINGS == 'true' && env.SKIP_CUDA_BINDINGS_TEST == '0' }}
372379
env:

‎ci/tools/check_pathfinder_ctk_libs.py‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
"""Check that cuda.pathfinder loads the given libraries from the CTK under CUDA_PATH.
5+
6+
The default cuda.pathfinder tests tolerate libraries that cannot be found, so a
7+
layout mismatch between the mini CTK and pathfinder's search directories would
8+
otherwise go unnoticed. The library may be found through any search step, as
9+
long as it comes from under CUDA_PATH.
10+
11+
Usage: python check_pathfinder_ctk_libs.py LIBNAME [LIBNAME ...]
12+
"""
13+
14+
from __future__ import annotations
15+
16+
import os
17+
import sys
18+
19+
from cuda.pathfinder import load_nvidia_dynamic_lib
20+
21+
22+
def _canonical(path: str) -> str:
23+
return os.path.normcase(os.path.realpath(path))
24+
25+
26+
def main(libnames: list[str]) -> int:
27+
root = _canonical(os.environ["CUDA_PATH"])
28+
failed = False
29+
for libname in libnames:
30+
loaded = load_nvidia_dynamic_lib(libname)
31+
if loaded.abs_path is None:
32+
print(f"FATAL: cuda.pathfinder did not report a path for {libname}", file=sys.stderr)
33+
failed = True
34+
continue
35+
path = _canonical(loaded.abs_path)
36+
print(f"{libname}: {path} (found via {loaded.found_via})")
37+
if not path.startswith(root + os.sep):
38+
print(f"FATAL: {libname} was not loaded from CUDA_PATH ({root})", file=sys.stderr)
39+
failed = True
40+
return 1 if failed else 0
41+
42+
43+
if __name__ == "__main__":
44+
sys.exit(main(sys.argv[1:]))

0 commit comments

Comments
 (0)