Skip to content

[Deps] Use cu13 extra for nvidia cutlass dsl#25576

Merged
Kangyan-Zhou merged 1 commit into
sgl-project:mainfrom
mmangkad:update-cutlass-dsl-cu13-4.5.1
May 21, 2026
Merged

[Deps] Use cu13 extra for nvidia cutlass dsl#25576
Kangyan-Zhou merged 1 commit into
sgl-project:mainfrom
mmangkad:update-cutlass-dsl-cu13-4.5.1

Conversation

@mmangkad

@mmangkad mmangkad commented May 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add [cu13] extra to nvidia-cutlass-dsl dependency since we already default to CUDA 13 and optionally upgrade to 4.5.1

Should fix #25564

Test Plan

CI


CI States

Latest PR Test (Base): ✅ Run #26077315178
Latest PR Test (Extra): ⚠️ Not enabled -- add run-ci-extra label to opt in.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@github-actions github-actions Bot added the dependencies Pull requests that update a dependency file label May 18, 2026
@mmangkad

mmangkad commented May 18, 2026

Copy link
Copy Markdown
Collaborator Author

/rerun-failed-ci

@mmangkad
mmangkad force-pushed the update-cutlass-dsl-cu13-4.5.1 branch 5 times, most recently from 3defae2 to 0d5f6ec Compare May 18, 2026 18:05

@trevor-m trevor-m left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mmangkad !

Comment thread python/pyproject.toml
Switch nvidia-cutlass-dsl to the cu13 extra and update the pin to 4.5.1 for the CUDA 13 default environment.
@mmangkad
mmangkad force-pushed the update-cutlass-dsl-cu13-4.5.1 branch from 0d5f6ec to 59a2ddb Compare May 19, 2026 05:02
@mmangkad
mmangkad requested review from Kangyan-Zhou and trevor-m May 20, 2026 09:31
@Kangyan-Zhou

Copy link
Copy Markdown
Collaborator

/rerun-failed-ci

@Kangyan-Zhou
Kangyan-Zhou merged commit a449ee4 into sgl-project:main May 21, 2026
391 of 415 checks passed
Kangyan-Zhou added a commit that referenced this pull request May 21, 2026
@mmangkad
mmangkad deleted the update-cutlass-dsl-cu13-4.5.1 branch May 21, 2026 03:37
Kangyan-Zhou added a commit to Kangyan-Zhou/sglang that referenced this pull request May 21, 2026
PR sgl-project#25576 bumped nvidia-cutlass-dsl[cu13] from 4.5.0 to 4.5.1. The bump
exposed a latent file-level conflict between -libs-base and -libs-cu13
(both written by the additive [cu13] extra) as a hard GPUModuleOp
TypeError on H100: -libs-cu13's pybind11 binding changed to the new
MLIR-style ((operation: object)) without a matching bump to the Python
wrapper in nvidia-cutlass-dsl, so loading -libs-cu13's .so makes the
wrapper's old-style super().__init__() call fail.

Two changes:

1. Revert the version bump (4.5.1 -> 4.5.0). At 4.5.0 both .so files
   expose a compatible binding, so the same coexistence no longer crashes.
   This removes the active TypeError on H100 and on the CUDA-13 Docker
   image for non-Blackwell users.

2. Add fix_cutlass_dsl_libs() to ci_install_dependency.sh, called from
   main() after download_flashinfer_cache. The function picks the right
   libs package per GPU family even at 4.5.0 to avoid two independent
   regressions that the silent conflict could still hit:

     Blackwell (IS_BLACKWELL=1, CU13):
       Purge -libs-base, force-reinstall -libs-cu13 so its files take
       precedence. -libs-base is CUDA-12.9-built and lacks the sm_110
       arch alias that GB300/B200 need at cutlass import time.

     Non-Blackwell CU13 (H100, H200):
       Purge -libs-cu13, force-reinstall -libs-base. -libs-cu13 carries
       a CUDBG_EXCEPTION_WARP_ILLEGAL_ADDRESS regression in LoRA CUDA-
       graph capture on sm_90 (sgl-project#25743 / reverted by sgl-project#25756).

     Non-CU13: no-op (only -libs-base ever installed).
Kangyan-Zhou added a commit to Kangyan-Zhou/sglang that referenced this pull request May 21, 2026
Revert the version bump from PR sgl-project#25576. At 4.5.1, -libs-cu13's pybind11
binding changed to new MLIR-style ((operation: object)) without a
matching bump to the Python wrapper in nvidia-cutlass-dsl, exposing the
latent file-level conflict between -libs-base and -libs-cu13 (both
written by the additive [cu13] extra) as a hard GPUModuleOp TypeError
at kernel-compile time on CU13 runners.

At 4.5.0 both .so files expose a compatible binding, so the same
coexistence is silent and CI was empirically green on H100 and Blackwell
during the post-sgl-project#25756, pre-sgl-project#25576 window. Going back to 4.5.0 restores
that state.

Supersedes sgl-project#25935 (which proposed the same revert but was closed).
Kangyan-Zhou added a commit to Kangyan-Zhou/sglang that referenced this pull request May 21, 2026
- Use "${REPO_ROOT}/python/pyproject.toml" instead of relative path so the
  version probe doesn't depend on the working directory the script is
  launched from (per gemini-code-assist review).
- Bump nvidia-cutlass-dsl[cu13] 4.5.0 -> 4.5.1 now that the wheel-mix
  TypeError is mitigated by force_reinstall_cutlass_dsl_libs_cu13. This
  re-applies sgl-project#25576 which was rolled back in sgl-project#25938 only because of the
  install-order bug.
alisonshao pushed a commit that referenced this pull request May 21, 2026
- Use "${REPO_ROOT}/python/pyproject.toml" instead of relative path so the
  version probe doesn't depend on the working directory the script is
  launched from (per gemini-code-assist review).
- Bump nvidia-cutlass-dsl[cu13] 4.5.0 -> 4.5.1 now that the wheel-mix
  TypeError is mitigated by force_reinstall_cutlass_dsl_libs_cu13. This
  re-applies #25576 which was rolled back in #25938 only because of the
  install-order bug.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file run-ci

Projects

None yet

3 participants