Repository navigation
CI: Add CUDA 13.4 support - #10294
Conversation
|
/test mini |
|
The |
Should be fixed by #10295. |
Mirrors cupy#10152 for CUDA 13.4: - schema.yaml: register CUDA 13.4, cuda-python 13.4, and add 13.4 under NCCL 2.30 / cuTENSOR 2.4 / cuSPARSELt 0.9.1 - matrix.yaml, config.pbtxt: add cupy.linux.cuda134{,.multi} + cupy.win.cuda134 as the new '@Push mini' target; demote 13.3 to '@nightly full' - matrix.yaml: bump cupy.linux.cuda13x-cuda-python to CUDA 13.4 + cuda-python 13.4 - ci/versions.yml: build.version 13.3.1 -> 13.4.1 - windows/_flexci.ps1: add 13.4 branch, point 13.x fallback to 13.4 - docs/source/install.rst: append v13.4 to the CUDA Toolkit list The regenerated Dockerfiles / .sh / .gitattributes / config.tags.json / coverage.rst come from '.pfnci/generate.py'.
CUDA gained Windows-on-Arm host support in CTK 13.4, so wire the GHA build lane to also produce cupy-cuda13x win_arm64 wheels alongside the existing linux-64 / linux-aarch64 / win-64 lanes. Follows the NVIDIA/cuda-python#2789 pattern: - New `build-win-arm64` job in ci.yml, gated on `startsWith(CUDA_BUILD_VER, '13.')` so pre-13.4 backport branches cleanly skip it. - Uses only `CUDA_BUILD_VER` (there is no CUDA 12 x arm64 pairing). - Python 3.10 is excluded on `win-arm64` -- CPython upstream has no official 3.10 Windows ARM64 build (nuget-cpython and actions/setup-python's manifest both drop it), so the matrix entry is unbuildable. - Runner: `windows-11-arm` (GitHub-hosted). - `fetch_ctk` action learns the `windows-arm64` redist subdir; supporting Python tools (`env-vars`, `prepare_wheel_build.py`, `wheel_configs.py`, `fetch_ctk_redistrib.py`) get the matching platform entries. - cuTENSOR preload and rapidsai/sccache are skipped for `win-arm64` because NVIDIA has not yet shipped WoA cuTENSOR binaries and rapidsai/sccache has no WoA build. Both are marked in-code as temporary; re-enable when the upstreams catch up. - No test/import step for `win-arm64` -- no GHA runner has an ARM64 GPU. This lane is build-only for now.
The rationale in c0b0180 for skipping sccache on win-arm64 was wrong: rapidsai/sccache v0.17.0-rapids.4 does ship sccache-aarch64-pc-windows-msvc.tar.gz. That target has been in mozilla upstream since at least v0.10.0 (Feb 2025) and the rapidsai fork inherits it from v0.17.0. So the previous commit's 'no build available' comments were factually wrong. Split the sccache URL-triple case into per-Windows-arch branches (win-64 -> x86_64-pc-windows-msvc, win-arm64 -> aarch64-pc-windows-msvc) and undo the win-arm64 skips on sccache install, CIBW_ENVIRONMENT_WINDOWS NVCC wrapper, and the sccache-summary report step.
|
/test mini |
|
/test mini |
|
On Linux, we are blocked by lack of CUDA 13.4 containers, which would be released some time next week. On Windows, we need to add CTK 13.4 to the VM image. |
|
/test mini,windows |
|
/test mini,windows |
|
/test mini,windows |
|
/test mini |
The nvidia/cuda registry ships 13.4.1-* only; the initial 13.4.0 image was never published, so the auto-generated Dockerfiles failed to pull `nvidia/cuda:13.4.0-devel-ubuntu22.04`. Bump `.pfnci/schema.yaml`'s 13.4 full_version to 13.4.1 (matching the precedent set by 12.9 -> 12.9.1) and regenerate the affected Dockerfiles via `.pfnci/generate.py`.
|
/test mini |
NVIDIA's apt repo has no 'libnccl2=2.30.*+cuda13.4' build -- the +cuda13.4 line jumps straight to NCCL 2.31.2. cuda134 / cuda134.multi / cuda13x-cuda-python were failing apt install with E: Version '2.30.*+cuda13.4' for 'libnccl2' was not found Move those matrix rows to NCCL 2.31, keep the existing 13.3 lane on 2.30 (which does ship 2.30.7-1+cuda13.3), and regenerate the affected Dockerfiles via .pfnci/generate.py.
|
/test mini |
|
/test mini,linux |
|
Both Linux and Windows are green now! |
seberg
left a comment
There was a problem hiding this comment.
LGTM, thanks. To tiny nits, that may make sense:
wheel_build.pyandbuild-wheel.yamlstill have awin-64each checks that don't seem to matter currently but should probably be.startswith("win")anyway.- The docs can now also list
NCCL 2.31
(I thought I'd push, but it wasn't enabled.)
Two spots gated on 'host-platform == win-64' need to cover win-arm64
too, since both target Windows:
- build-wheel.yml: the LongPathsEnabled registry check is OS-wide,
not arch-specific -- WoA needs the same guard.
- prepare_wheel_build.py: Windows uses ';' as PATH separator on
every arch, so win-arm64 was picking ':' and generating malformed
CUPY_INCLUDE_PATH / CUPY_LIBRARY_PATH values.
Both cases now match on startsWith('win-'). Other 'win-64' uses in
the tree (runner map, sccache URL triple, PFN CI artifact name)
remain intentionally per-arch.
Matches the .pfnci/matrix.yaml + schema.yaml addition for the CUDA 13.4 lanes.
|
/test mini |
Sorry, I turned it on now. I found agents have a tendency to miss it.. |
Part of #10219.
Adds CUDA 13.4 to the CI matrix (Linux single + multi-GPU, Windows), extends NCCL 2.31 / cuTENSOR 2.4 / cuSPARSELt 0.9.1 / cuda-python 13.4 to CUDA 13.4, and moves the "mini" per-push tag from 13.3 to 13.4 as the new highest supported combination.
The remaining checklist items in #10219 (compiler max cc, install_library, wheel duplicate detection, cupy-release-tools, website) are follow-ups.