Skip to content

Add CUDA 12.9.1 x86 nightly binaries #154980

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
tinglvv wants to merge 12 commits into from
Closed

Add CUDA 12.9.1 x86 nightly binaries #154980

tinglvv wants to merge 12 commits into from

Conversation

@tinglvv
Copy link
Collaborator

@tinglvv tinglvv commented Jun 3, 2025 •
edited
Loading

#155196

Adding CUDA 12.9.1 to nightly binaries matrix for linux (x86) builds.
Also add sbsa and libtorch build docker images.

Todo: addition of sbsa and libtorch builds will be follow-up PRs.

cc @atalman @malfet @ptrblck @nWEIdia @eqy

@tinglvv tinglvv requested review from a team and jeffdaily as code owners June 3, 2025 08:13
@pytorch-bot
Copy link

pytorch-bot bot commented Jun 3, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/154980

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 Cancelled Jobs, 3 Pending, 2 Unrelated Failures

As of commit 60dacb9 with merge base c4b93e6 (image):

CANCELLED JOBS - The following jobs were cancelled. Please retry:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added the topic: not user facing topic category label Jun 3, 2025
@tinglvv tinglvv added the ciflow/binaries Trigger all binary build and upload jobs on the PR label Jun 3, 2025
@soulitzer soulitzer added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Jun 3, 2025
nWEIdia
nWEIdia reviewed Jun 3, 2025
View reviewed changes
@tinglvv tinglvv force-pushed the add-cu1290-build branch from 9f866b6 to dfc459e Compare June 5, 2025 00:52
@tinglvv tinglvv mentioned this pull request Jun 5, 2025
Open
12 tasks
@nWEIdia
Copy link
Collaborator

nWEIdia commented Jun 6, 2025

@tinglvv our 12.9.1 is out. Let's update using 12.9.1 bits.

@tinglvv tinglvv force-pushed the add-cu1290-build branch from b3e4cb5 to a2d0d98 Compare June 6, 2025 00:22
@tinglvv
Copy link
Collaborator Author

tinglvv commented Jun 6, 2025

@tinglvv our 12.9.1 is out. Let's update using 12.9.1 bits.

Sure, let me update once the current CI run finishes building, as I just fixed the issue with the docker image not building. It will also require re-uploading the pypi packages which would take a day as well.

@tinglvv tinglvv changed the title Add CUDA 12.9.0 nightly binaries Add CUDA 12.9.1 nightly binaries Jun 9, 2025
@tinglvv tinglvv force-pushed the add-cu1290-build branch 2 times, most recently from b0c830d to 3bc1d0d Compare June 10, 2025 06:19
@atalman
Copy link
Contributor

atalman commented Jun 10, 2025

Hi @tinglvv should we use cudnn == 9.10.2.21 here ?

This was referenced Jun 10, 2025
Closed
Closed
@tinglvv tinglvv force-pushed the add-cu1290-build branch from 3d2fdc0 to 3538cf6 Compare June 10, 2025 23:02
@tinglvv
Copy link
Collaborator Author

tinglvv commented Jun 10, 2025 •
edited
Loading

CUDA 12.9.1 to use
NCCL 2.27.3 per #155233 (Update: Need #155233 to update the release tag version first and then I can update 12.9. Getting error RuntimeError: 12.9 NCCL release tag version v2.26.5-1 does not correspond to wheel version 2.27.3 with current code)
CUDNN 9.10.2.21 per #155576

nWEIdia
nWEIdia reviewed Jun 10, 2025
View reviewed changes
@tinglvv
Copy link
Collaborator Author

tinglvv commented Jun 11, 2025 •
edited
Loading

build failure https://github.com/pytorch/pytorch/actions/runs/15572211125/job/43856156946 Tesla** M60 with CUDA capability sm_52 is not compatible with the current PyTorch installation. The current PyTorch install supports CUDA capabilities sm_75 sm_80 sm_86 sm_90 sm_100 sm_120 compute_120.
62ef609 is needed to update runner for 12.9 too.

@tinglvv tinglvv changed the title Add CUDA 12.9.1 nightly binaries Add CUDA 12.9.1 x86 nightly binaries Jun 11, 2025
atalman
atalman approved these changes Jun 11, 2025
View reviewed changes
Copy link
Contributor

@atalman atalman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@atalman
Copy link
Contributor

atalman commented Jun 11, 2025

@pytorchmergebot merge -f "all required builds are green"

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Jun 11, 2025

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@Skylion007
Copy link
Collaborator

Skylion007 commented Jun 11, 2025

The cusparseLt in this was wrong. :(

@tinglvv
Copy link
Collaborator Author

tinglvv commented Jun 11, 2025

Updating cusparseLt to 0.7.1 in follow-up PR #155709

@tinglvv tinglvv mentioned this pull request Jun 11, 2025
Closed
pytorchmergebot pushed a commit that referenced this pull request Jun 13, 2025
@Skylion007 @pytorchmergebot
[BE]: Sync cusparselt 12.9 with static build and other cuda 12 (#155709)
bbbced9 
followup for #154980

Pull Request resolved: #155709
Approved by: https://github.com/tinglvv, https://github.com/atalman, https://github.com/nWEIdia, https://github.com/cyyever
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Skylion007 Skylion007 Skylion007 left review comments

@nWEIdia nWEIdia nWEIdia left review comments

@eqy eqy eqy approved these changes

@atalman atalman atalman approved these changes

@jeffdaily jeffdaily Awaiting requested review from jeffdaily jeffdaily is a code owner

Assignees

No one assigned

Labels

ciflow/binaries Trigger all binary build and upload jobs on the PR Merged open source topic: not user facing topic category triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@tinglvv @nWEIdia @atalman @pytorchmergebot @Skylion007 @eqy @soulitzer @pytorchbot