Skip to content

CI: Provide a mechanism to verify build compatibility across all supported CUDA versions #10202

Open

Description

@kmaehashi

In #10058, the CI flow was updated to build wheels only once using the latest CUDA version, and then run tests across multiple CUDA versions. While this optimizes CI resources, we no longer automatically verify that CuPy builds successfully from source against older supported CUDA versions.

Certain key user groups, such as maintainers of modules in HPC environments, frequently build CuPy from source using specific or non-latest CUDA versions. Ensuring build-time compatibility across supported CUDA versions is crucial for their workflows.

While this is not a high-priority blocker, it would be valuable to establish a mechanism to check build compatibility across all supported CUDA versions.

Activity

  1. raman118 commented on Aug 19, 2026

    @raman118
    Contributor

    Hi @kmaehashi, My initial thought is a build-only job (no GPU needed, just compile against each supported CUDA version using nvidia/cuda:X.Y-devel images) that runs on a schedule rather than every PR, to catch build breakage without adding cost to the main CI.

    Before I put together a PR: would you want this as a standalone GitHub Actions workflow, or does it need to plug into the existing FlexCI matrix in .pfnci/matrix.yaml?

  2. leofang commented on Sep 2, 2026

    @leofang
    Member

    To implement this with the new GHA infra instead of reverting back to the old CI infra, we need to be able to cache all sccache blobs & every CTK versions in GHA Cache, which has limited space (we set it to 30 GB; I think paid accounts can go up to 50 GB and enterprise to 200 GB).

    A middle ground is to still use PFN CI to do build tests, and make sccache target GCS instead of GHA Cache as the backend. But I feel if we were to do this it's better to create new, dedicated pipelines for this purpose, so that we can set GPU = 0 and keep the cost low?

    In the long run, though, my vision is that there is very little to no code in CuPy that has tight coupling with CTK code (xref: #9236). With the end goal in mind, my take is it's a reasonable trade-off to defer this until we see an influx of bugs.

    For those who constantly need to build from source, I am certain that they have ways to unblock themselves, e.g. we never have proper conda-based CI tests yet we still survive til this day 🙂 I think our CI should cover the majority of use cases (aka, wheels).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions