Skip to content

CI: Add CUDA 13.4 support - #10294

Merged
leofang merged 9 commits into
cupy:mainfrom
leofang:ci/cuda134
Sep 21, 2026
Merged

leofang merged 9 commits into
cupy:mainfrom
leofang:ci/cuda134

Conversation

@leofang

@leofang leofang commented Sep 9, 2026 •

Copy link
Copy Markdown
Member

Part of #10219.

Adds CUDA 13.4 to the CI matrix (Linux single + multi-GPU, Windows), extends NCCL 2.31 / cuTENSOR 2.4 / cuSPARSELt 0.9.1 / cuda-python 13.4 to CUDA 13.4, and moves the "mini" per-push tag from 13.3 to 13.4 as the new highest supported combination.

The remaining checklist items in #10219 (compiler max cc, install_library, wheel duplicate detection, cupy-release-tools, website) are follow-ups.

@leofang

leofang commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

/test mini

@cupy-ci-trigger cupy-ci-trigger Bot added the ci:triggered Triggers the CI build pipeline (bounced automatically by /test) label Sep 9, 2026
@leofang leofang added this to the v14.3.0 milestone Sep 9, 2026
@leofang

leofang commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

The cuda-drivers 615.71.09 package (just released earlier today) has some issues, tracked in nvbugs 6749972.

@leofang

leofang commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

The cuda-drivers 615.71.09 package (just released earlier today) has some issues, tracked in nvbugs 6749972.

Should be fixed by #10295.

@github-actions github-actions Bot removed the ci:triggered Triggers the CI build pipeline (bounced automatically by /test) label Sep 10, 2026
Mirrors cupy#10152 for CUDA 13.4:

- schema.yaml: register CUDA 13.4, cuda-python 13.4, and add 13.4 under
  NCCL 2.30 / cuTENSOR 2.4 / cuSPARSELt 0.9.1
- matrix.yaml, config.pbtxt: add cupy.linux.cuda134{,.multi} + cupy.win.cuda134
  as the new '@Push mini' target; demote 13.3 to '@nightly full'
- matrix.yaml: bump cupy.linux.cuda13x-cuda-python to CUDA 13.4 + cuda-python 13.4
- ci/versions.yml: build.version 13.3.1 -> 13.4.1
- windows/_flexci.ps1: add 13.4 branch, point 13.x fallback to 13.4
- docs/source/install.rst: append v13.4 to the CUDA Toolkit list

The regenerated Dockerfiles / .sh / .gitattributes / config.tags.json /
coverage.rst come from '.pfnci/generate.py'.
CUDA gained Windows-on-Arm host support in CTK 13.4, so wire the
GHA build lane to also produce cupy-cuda13x win_arm64 wheels alongside
the existing linux-64 / linux-aarch64 / win-64 lanes.

Follows the NVIDIA/cuda-python#2789 pattern:

- New `build-win-arm64` job in ci.yml, gated on
  `startsWith(CUDA_BUILD_VER, '13.')` so pre-13.4 backport branches
  cleanly skip it.
- Uses only `CUDA_BUILD_VER` (there is no CUDA 12 x arm64 pairing).
- Python 3.10 is excluded on `win-arm64` -- CPython upstream has no
  official 3.10 Windows ARM64 build (nuget-cpython and
  actions/setup-python's manifest both drop it), so the matrix
  entry is unbuildable.
- Runner: `windows-11-arm` (GitHub-hosted).
- `fetch_ctk` action learns the `windows-arm64` redist subdir;
  supporting Python tools (`env-vars`, `prepare_wheel_build.py`,
  `wheel_configs.py`, `fetch_ctk_redistrib.py`) get the
  matching platform entries.
- cuTENSOR preload and rapidsai/sccache are skipped for `win-arm64`
  because NVIDIA has not yet shipped WoA cuTENSOR binaries and
  rapidsai/sccache has no WoA build. Both are marked in-code as
  temporary; re-enable when the upstreams catch up.
- No test/import step for `win-arm64` -- no GHA runner has an
  ARM64 GPU. This lane is build-only for now.
The rationale in c0b0180 for skipping sccache on win-arm64 was wrong:
rapidsai/sccache v0.17.0-rapids.4 does ship
sccache-aarch64-pc-windows-msvc.tar.gz. That target has been in mozilla
upstream since at least v0.10.0 (Feb 2025) and the rapidsai fork
inherits it from v0.17.0. So the previous commit's 'no build available'
comments were factually wrong.

Split the sccache URL-triple case into per-Windows-arch branches
(win-64 -> x86_64-pc-windows-msvc, win-arm64 -> aarch64-pc-windows-msvc)
and undo the win-arm64 skips on sccache install, CIBW_ENVIRONMENT_WINDOWS
NVCC wrapper, and the sccache-summary report step.
@leofang

leofang commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

/test mini

@cupy-ci-trigger cupy-ci-trigger Bot added the ci:triggered Triggers the CI build pipeline (bounced automatically by /test) label Sep 10, 2026
@leofang leofang added cat:test Test code / CI prio:high and removed ci:triggered Triggers the CI build pipeline (bounced automatically by /test) labels Sep 10, 2026
@leofang

leofang commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

/test mini

@cupy-ci-trigger cupy-ci-trigger Bot added the ci:triggered Triggers the CI build pipeline (bounced automatically by /test) label Sep 10, 2026
@leofang

leofang commented Sep 10, 2026

Copy link
Copy Markdown
Member Author

On Linux, we are blocked by lack of CUDA 13.4 containers, which would be released some time next week.

On Windows, we need to add CTK 13.4 to the VM image.

@leofang

leofang commented Sep 14, 2026

Copy link
Copy Markdown
Member Author

/test mini,windows

@cupy-ci-trigger cupy-ci-trigger Bot added ci:triggered Triggers the CI build pipeline (bounced automatically by /test) and removed ci:triggered Triggers the CI build pipeline (bounced automatically by /test) labels Sep 14, 2026
@leofang

leofang commented Sep 15, 2026

Copy link
Copy Markdown
Member Author

/test mini,windows

@cupy-ci-trigger cupy-ci-trigger Bot added ci:triggered Triggers the CI build pipeline (bounced automatically by /test) and removed ci:triggered Triggers the CI build pipeline (bounced automatically by /test) labels Sep 15, 2026
@leofang

leofang commented Sep 15, 2026

Copy link
Copy Markdown
Member Author

/test mini,windows

@github-actions github-actions Bot removed the ci:triggered Triggers the CI build pipeline (bounced automatically by /test) label Sep 19, 2026
@leofang

leofang commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

/test mini

@cupy-ci-trigger cupy-ci-trigger Bot added the ci:triggered Triggers the CI build pipeline (bounced automatically by /test) label Sep 19, 2026
The nvidia/cuda registry ships 13.4.1-* only; the initial 13.4.0
image was never published, so the auto-generated Dockerfiles
failed to pull `nvidia/cuda:13.4.0-devel-ubuntu22.04`.

Bump `.pfnci/schema.yaml`'s 13.4 full_version to 13.4.1 (matching
the precedent set by 12.9 -> 12.9.1) and regenerate the affected
Dockerfiles via `.pfnci/generate.py`.
@github-actions github-actions Bot removed the ci:triggered Triggers the CI build pipeline (bounced automatically by /test) label Sep 19, 2026
@leofang

leofang commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

/test mini

@cupy-ci-trigger cupy-ci-trigger Bot added the ci:triggered Triggers the CI build pipeline (bounced automatically by /test) label Sep 19, 2026
NVIDIA's apt repo has no 'libnccl2=2.30.*+cuda13.4' build -- the
+cuda13.4 line jumps straight to NCCL 2.31.2. cuda134 / cuda134.multi
/ cuda13x-cuda-python were failing apt install with

  E: Version '2.30.*+cuda13.4' for 'libnccl2' was not found

Move those matrix rows to NCCL 2.31, keep the existing 13.3 lane on
2.30 (which does ship 2.30.7-1+cuda13.3), and regenerate the affected
Dockerfiles via .pfnci/generate.py.
@github-actions github-actions Bot removed the ci:triggered Triggers the CI build pipeline (bounced automatically by /test) label Sep 19, 2026
@leofang

leofang commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

/test mini

@cupy-ci-trigger cupy-ci-trigger Bot added the ci:triggered Triggers the CI build pipeline (bounced automatically by /test) label Sep 19, 2026
@leofang

leofang commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

/test mini,linux

@cupy-ci-trigger cupy-ci-trigger Bot added ci:triggered Triggers the CI build pipeline (bounced automatically by /test) and removed ci:triggered Triggers the CI build pipeline (bounced automatically by /test) labels Sep 19, 2026
@leofang
leofang marked this pull request as ready for review September 19, 2026 22:47
@leofang
leofang requested a review from a team as a code owner September 19, 2026 22:47
@leofang

leofang commented Sep 20, 2026

Copy link
Copy Markdown
Member Author

Both Linux and Windows are green now!

seberg
seberg previously approved these changes Sep 21, 2026

@seberg seberg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks. To tiny nits, that may make sense:

  1. wheel_build.py and build-wheel.yaml still have a win-64 each checks that don't seem to matter currently but should probably be .startswith("win") anyway.
  2. The docs can now also list NCCL 2.31

(I thought I'd push, but it wasn't enabled.)

Two spots gated on 'host-platform == win-64' need to cover win-arm64
too, since both target Windows:

- build-wheel.yml: the LongPathsEnabled registry check is OS-wide,
  not arch-specific -- WoA needs the same guard.
- prepare_wheel_build.py: Windows uses ';' as PATH separator on
  every arch, so win-arm64 was picking ':' and generating malformed
  CUPY_INCLUDE_PATH / CUPY_LIBRARY_PATH values.

Both cases now match on startsWith('win-'). Other 'win-64' uses in
the tree (runner map, sccache URL triple, PFN CI artifact name)
remain intentionally per-arch.
@github-actions github-actions Bot removed the ci:triggered Triggers the CI build pipeline (bounced automatically by /test) label Sep 21, 2026
Matches the .pfnci/matrix.yaml + schema.yaml addition for the CUDA
13.4 lanes.
@leofang

leofang commented Sep 21, 2026

Copy link
Copy Markdown
Member Author

/test mini

@cupy-ci-trigger cupy-ci-trigger Bot added the ci:triggered Triggers the CI build pipeline (bounced automatically by /test) label Sep 21, 2026
@leofang

leofang commented Sep 21, 2026

Copy link
Copy Markdown
Member Author

(I thought I'd push, but it wasn't enabled.)

Sorry, I turned it on now. I found agents have a tendency to miss it..

@leofang
leofang enabled auto-merge September 21, 2026 14:14
@leofang
leofang merged commit 0b5c56a into cupy:main Sep 21, 2026
110 checks passed
@leofang
leofang deleted the ci/cuda134 branch September 21, 2026 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cat:test Test code / CI ci:triggered Triggers the CI build pipeline (bounced automatically by /test) prio:high

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants