Skip to content

Commit 4b9dcbd

Browse files
leofangleofang
authored
ci: clean up recurring nightly optional-deps failures (#2968)
* ci: pin numpy<2.5 for nightly-numba-cuda and add cccl for numba-cuda-mlir Two targeted fixes for the recurring failures tracked in #2748: - nightly-numba-cuda: pin numpy<2.5. numba-cuda 0.30.4 references np.row_stack, removed in NumPy 2.5, so every nightly numba-cuda job crashes at collection. Tracked upstream in NVIDIA/numba-cuda#907; drop the cap once a fixed wheel is on PyPI. - nightly-numba-cuda-mlir: add cccl to cuda-toolkit extras. NVRTC currently fails with "catastrophic error: cannot open source file 'nv/target'" when compiling cooperative_groups/details/info.h and curand_kernel.h; the cccl extra pulls nvidia-cuda-cccl, which drops the missing header at nvidia/cu13/include/nv/target. * TEMP: add push trigger to ci-nightly.yml to verify fix on this PR Revert before merging. Same trick as 8d51cf7 * ci: deselect frozen-driver-table test on released cuda-core <=1.2.1 test_frozen_driver_table_covers_all_curesult_members was removed on main by #2792 (2026-09-09) but neither cuda-core-v1.2.0 nor v1.2.1 carry the revert. main cuda-bindings 13.4.1 adds three new CUresult members (MULTICAST_RESOURCE_FULL, INSUFFICIENT_LOADER_VERSION, FABRIC_NOT_READY) not present in the frozen table those releases ship, so the nightly-cuda-core job trips it on every run. Mirror the existing v1.0.1 NvlinkVersion pattern with a version-gated --deselect that drops automatically once the next cuda-core release ships the #2792 revert. * ci: skip numba-cuda IPC tests and mlir test_device_record_copy nightly-numba-cuda: switch from `python -m numba.runtests` to `pytest` (numba-cuda's own CI uses pytest; PR #1987 landed the runtests form as a bring-up fix) so we can skip individual tests. Two workarounds: - pin pytest<9 (matches upstream's numba-cuda cap; NVIDIA/numba-cuda#637 covers the subTest breakage on 9). - --ignore-glob "**/test_ipc.py" — numba-cuda v0.30.4 accesses CUipcMemHandle.reserved, which cuda-bindings intentionally removed; numba-cuda is EOL upstream so no fix is coming (#2748). nightly-numba-cuda-mlir: add --deselect for TestCudaDeviceRecordWithRecord.test_device_record_copy — the WithRecord fixture uses np.recarray (uninitialized memory), so when the float32 field encodes a NaN, `np.testing.assert_equal` trips on NaN != NaN even though the copy round-trip is correct. Filed upstream as NVIDIA/numba-cuda-mlir#341. * ci: move pytest<9 pin for numba-cuda into ci/tools/run-tests Keeps pip constraints in the install step alongside numpy<2.5, instead of a separate `pip install` in the workflow. * ci: downgrade pytest to <9 as a second pip call for numba-cuda cuda_core's test-cuXX group hard-pins pytest==9.1.0, so pip can't satisfy that and pytest<9 in one solve. Split the numba-cuda pytest pin into a second `pip install "pytest<9"` after the main install. * ci: use --import-mode=importlib for numba-cuda pytest run Default `prepend` import mode walks up until it finds a directory without `__init__.py`. numba-cuda's on-disk layout is `site-packages/numba_cuda/numba/cuda/tests/...`, and `numba/` here is a namespace subdir with no `__init__.py`, so pytest treats it as the rootpath and imports each test file as `cuda.tests.<...>`. That collides with cuda-bindings' top-level `cuda` package (which has no `tests` subpackage), giving `ModuleNotFoundError: No module named 'cuda.tests'` at collection. `--import-mode=importlib` sidesteps the sys.path walk and imports each test module by its correct dotted path. * ci: run numba-cuda tests from checked-out source, mirroring mlir Same shape as the numba-cuda-mlir step. run-tests exposes NUMBA_CUDA_VER; the workflow checks out NVIDIA/numba-cuda at the matching tag and runs pytest from numba-cuda-released/testing/, so upstream's testing/pytest.ini kicks in (consider_namespace_packages=true + --pyargs numba.cuda.tests) and we stop caring about the site-packages namespace-package layout. Also mirror the wheel-only gate: in wheels mode, set NUMBA_CUDA_TEST_WHEEL_ONLY=1 so numba-cuda skips tests that need CTK executables. Binary-generated tests self-skip via `@unittest.skipIf(not NUMBA_CUDA_TEST_BIN_DIR, ...)`, so no `make -j` step is needed — mirrors mlir, which has no Makefile. Drops the --import-mode=importlib workaround from the prior commit; pytest.ini's consider_namespace_packages does the job. * ci: swap --ignore-glob for -k 'not TestIpc' in numba-cuda pytest pytest's --ignore-glob="**/test_ipc.py" didn't match through the --pyargs-resolved site-packages path (5 tests still ran and failed with the CUipcMemHandle.reserved AttributeError on linux-64). Match by class name instead — the affected tests live in TestIpcMemory and TestIpcStaged, both starting with "TestIpc", so this is unambiguous. * Revert "TEMP: add push trigger to ci-nightly.yml to verify fix on this PR" This reverts commit 4363950.
1 parent 5b26bcd commit 4b9dcbd

3 files changed

Lines changed: 89 additions & 7 deletions

File tree

‎.github/workflows/test-wheel-linux.yml‎

Lines changed: 40 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -467,9 +467,32 @@ jobs:
467467
pytest -rxXs -v tests/test_utils.py tests/example_tests/
468468
popd
469469
470+
- name: Checkout numba-cuda tests at matching tag
471+
if: ${{ inputs.test-mode == 'nightly-numba-cuda' && env.NUMBA_CUDA_VER != '' }}
472+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
473+
with:
474+
repository: NVIDIA/numba-cuda
475+
ref: v${{ env.NUMBA_CUDA_VER }}
476+
path: numba-cuda-released
477+
470478
- name: Run numba-cuda tests
471-
if: ${{ inputs.test-mode == 'nightly-numba-cuda' }}
472-
run: python -m numba.runtests numba.cuda.tests
479+
if: ${{ inputs.test-mode == 'nightly-numba-cuda' && env.NUMBA_CUDA_VER != '' }}
480+
env:
481+
LOCAL_CTK: ${{ matrix.LOCAL_CTK }}
482+
run: |
483+
if [[ "${LOCAL_CTK}" != 1 ]]; then
484+
# Let numba-cuda skip inputs that require toolkit executables.
485+
export NUMBA_CUDA_TEST_WHEEL_ONLY=1
486+
fi
487+
# Run from numba-cuda's testing/ dir so its pytest.ini kicks in
488+
# (consider_namespace_packages=true + --pyargs numba.cuda.tests).
489+
# -k "not TestIpc": numba-cuda v0.30.4 accesses CUipcMemHandle.reserved,
490+
# which cuda-bindings intentionally removed. numba-cuda is EOL upstream,
491+
# so no fix is coming. See cuda-python#2748. The affected tests all
492+
# live in the two TestIpcMemory/TestIpcStaged classes in test_ipc.py.
493+
pushd numba-cuda-released/testing
494+
pytest -v -k "not TestIpc"
495+
popd
473496
474497
- name: Checkout numba-cuda-mlir tests at matching tag
475498
if: ${{ inputs.test-mode == 'nightly-numba-cuda-mlir' && env.NUMBA_CUDA_MLIR_VER != '' }}
@@ -495,9 +518,13 @@ jobs:
495518
# Skip tests/benchmarks/ and tests/doc_examples/ — they import the
496519
# numba package at collection time, which cuSIMT intentionally does
497520
# not depend on. See NVIDIA/numba-cuda-mlir#136.
521+
# test_device_record_copy (WithRecord variant): flaky NaN vs NaN
522+
# compare from uninitialized np.recarray memory. See
523+
# NVIDIA/numba-cuda-mlir#341.
498524
pytest -rxXs -v \
499525
--ignore=tests/benchmarks \
500526
--ignore=tests/doc_examples \
527+
--deselect 'tests/numba_cuda_tests/cudadrv/test_cuda_devicerecord.py::TestCudaDeviceRecordWithRecord::test_device_record_copy' \
501528
tests/
502529
popd
503530
@@ -520,8 +547,8 @@ jobs:
520547
# patterns that pytest 9.1 rejects; the main-side fix (#2212) has
521548
# not yet shipped in a cuda-core release.
522549
pip install "pytest<9.1"
523-
# Version-gated deselect: drops automatically when a newer
524-
# cuda-core release with the wrapper-mapping update ships.
550+
# Version-gated deselects: drop automatically when a newer
551+
# cuda-core release ships that removes each source.
525552
DESELECTS=()
526553
if python -c "from packaging.version import Version; import sys; sys.exit(0 if Version('${CUDA_CORE_RELEASED_VER}') <= Version('1.0.1') else 1)"; then
527554
# NvlinkVersion: v1.0.1's wrapper mapping predates
@@ -531,6 +558,15 @@ jobs:
531558
--deselect 'tests/test_enum_coverage.py::test_wrapper_covers_all_binding_members[NvlinkVersion]'
532559
)
533560
fi
561+
if python -c "from packaging.version import Version; import sys; sys.exit(0 if Version('${CUDA_CORE_RELEASED_VER}') <= Version('1.2.1') else 1)"; then
562+
# test_frozen_driver_table_covers_all_curesult_members: v1.2.1's
563+
# frozen table (capped at CTK 13.1.1) predates 3 CUresult members
564+
# added by main cuda-bindings 13.4.1. Reverted on main by #2792;
565+
# drop this deselect once a cuda-core release with the revert ships.
566+
DESELECTS+=(
567+
--deselect 'tests/test_utils_enum_explanations_helpers.py::test_frozen_driver_table_covers_all_curesult_members'
568+
)
569+
fi
534570
pytest -rxXs -v --randomly-dont-reorganize \
535571
"${DESELECTS[@]}" \
536572
tests/

‎.github/workflows/test-wheel-windows.yml‎

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -450,10 +450,27 @@ jobs:
450450
pytest -rxXs -v tests/test_utils.py tests/example_tests/
451451
popd
452452
453+
- name: Checkout numba-cuda tests at matching tag
454+
if: ${{ inputs.test-mode == 'nightly-numba-cuda' && env.NUMBA_CUDA_VER != '' }}
455+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
456+
with:
457+
repository: NVIDIA/numba-cuda
458+
ref: v${{ env.NUMBA_CUDA_VER }}
459+
path: numba-cuda-released
460+
453461
- name: Run numba-cuda tests
454-
if: ${{ inputs.test-mode == 'nightly-numba-cuda' }}
462+
if: ${{ inputs.test-mode == 'nightly-numba-cuda' && env.NUMBA_CUDA_VER != '' }}
463+
env:
464+
LOCAL_CTK: ${{ matrix.LOCAL_CTK }}
455465
shell: bash --noprofile --norc -xeuo pipefail {0}
456-
run: python -m numba.runtests numba.cuda.tests
466+
run: |
467+
if [[ "${LOCAL_CTK}" != 1 ]]; then
468+
export NUMBA_CUDA_TEST_WHEEL_ONLY=1
469+
fi
470+
# See linux step for rationale on running from testing/ and -k.
471+
pushd numba-cuda-released/testing
472+
pytest -v -k "not TestIpc"
473+
popd
457474
458475
- name: Checkout numba-cuda-mlir tests at matching tag
459476
if: ${{ inputs.test-mode == 'nightly-numba-cuda-mlir' && env.NUMBA_CUDA_MLIR_VER != '' }}
@@ -476,9 +493,11 @@ jobs:
476493
pushd numba-cuda-mlir-released
477494
pip install --upgrade "pip>=25.1"
478495
pip install --group test
496+
# See linux step for rationale on the --deselect.
479497
pytest -rxXs -v \
480498
--ignore=tests/benchmarks \
481499
--ignore=tests/doc_examples \
500+
--deselect 'tests/numba_cuda_tests/cudadrv/test_cuda_devicerecord.py::TestCudaDeviceRecordWithRecord::test_device_record_copy' \
482501
tests/
483502
popd
484503
@@ -517,6 +536,12 @@ jobs:
517536
--deselect 'tests/test_memory.py::test_non_managed_resources_report_not_managed[pinned]'
518537
)
519538
fi
539+
if python -c "from packaging.version import Version; import sys; sys.exit(0 if Version('${CUDA_CORE_RELEASED_VER}') <= Version('1.2.1') else 1)"; then
540+
# See linux step for rationale (#2792 revert).
541+
DESELECTS+=(
542+
--deselect 'tests/test_utils_enum_explanations_helpers.py::test_frozen_driver_table_covers_all_curesult_members'
543+
)
544+
fi
520545
pytest -rxXs -v --randomly-dont-reorganize \
521546
"${DESELECTS[@]}" \
522547
tests/

‎ci/tools/run-tests‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -162,25 +162,37 @@ elif [[ "${test_module}" == "core" || "${test_module}" == nightly-* ]]; then
162162
elif [[ "${test_module}" == "nightly-numba-cuda" ]]; then
163163
echo "Installing pathfinder + bindings + core + test deps + numba-cuda"
164164
# numba-cuda's test-cuXX group deps (can't use --group for a wheel install):
165+
# numpy<2.5: numba-cuda 0.30.4 references np.row_stack, removed in numpy 2.5. See NVIDIA/numba-cuda#907.
165166
PIP_ARGS+=(
166167
"cuda-toolkit[curand,cublas]==${TEST_CUDA_MAJOR_MINOR}.*"
167168
"numba-cuda[cu${TEST_CUDA_MAJOR}]"
168169
"cupy-cuda${TEST_CUDA_MAJOR}x"
170+
"numpy<2.5"
169171
psutil cffi pytest-xdist pytest-benchmark filecheck ml_dtypes statistics
170172
)
171173
elif [[ "${test_module}" == "nightly-numba-cuda-mlir" ]]; then
172174
echo "Installing pathfinder + bindings + core + numba-cuda-mlir + cupy"
173175
# curand/cublas + cupy: some numba-cuda-mlir tests (e.g.
174176
# test_fortran_contiguous) call cp.random, which dlopens libcurand at
175177
# runtime. Mirror the numba-cuda branch, which installs the same libs.
178+
# cccl: NVRTC needs <nv/target> for cooperative_groups / curand_kernel.h.
176179
PIP_ARGS+=(
177180
"numba-cuda-mlir[cu${TEST_CUDA_MAJOR}]"
178-
"cuda-toolkit[curand,cublas]==${TEST_CUDA_MAJOR_MINOR}.*"
181+
"cuda-toolkit[curand,cublas,cccl]==${TEST_CUDA_MAJOR_MINOR}.*"
179182
"cupy-cuda${TEST_CUDA_MAJOR}x"
180183
)
181184
fi
182185

183186
pip install "${PIP_ARGS[@]}"
187+
188+
if [[ "${test_module}" == "nightly-numba-cuda" ]]; then
189+
# Downgrade pytest for numba-cuda: cuda_core's test-cuXX group hard-pins
190+
# pytest==9.1.0, but numba-cuda's subTest usage breaks under pytest 9.
191+
# Can't add pytest<9 to PIP_ARGS above because pip can't satisfy both
192+
# ==9.1.0 and <9 in one solve. See NVIDIA/numba-cuda#637.
193+
pip install "pytest<9"
194+
fi
195+
184196
echo "Nightly install complete — installed packages:"
185197
pip list
186198
popd
@@ -193,6 +205,15 @@ elif [[ "${test_module}" == "core" || "${test_module}" == nightly-* ]]; then
193205
if [[ -n "${GITHUB_ENV:-}" ]]; then
194206
echo "NUMBA_CUDA_MLIR_VER=${installed_ver}" >> "${GITHUB_ENV}"
195207
fi
208+
elif [[ "${test_module}" == "nightly-numba-cuda" ]]; then
209+
# Expose the installed numba-cuda version so the workflow can
210+
# actions/checkout the matching v<X.Y.Z> tag from NVIDIA/numba-cuda
211+
# (the wheel does not ship test_*.py files at a pytest-discoverable
212+
# top level; upstream's testing/pytest.ini is required to run them).
213+
installed_ver=$(pip show numba-cuda | awk '/^Version:/{print $2}')
214+
if [[ -n "${GITHUB_ENV:-}" ]]; then
215+
echo "NUMBA_CUDA_VER=${installed_ver}" >> "${GITHUB_ENV}"
216+
fi
196217
fi
197218
fi
198219
fi

0 commit comments

Comments
 (0)