[BE]: Update NCCL to 2.27.3 #155233

Skylion007 · 2025-06-05T16:23:32Z

Fixes: #155052 and #153517

This upgrade is needed to effectively use those symmetric memory kernels anyway. Also fixes some nasty NCCL bugs.

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k

pytorch-bot · 2025-06-05T16:23:36Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/155233

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 008adee with merge base 70b68ca ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

nWEIdia

Could you please re-enable the skipped unit test in test_c10d_nccl.py and see if this update would indeed fix #153517 ?

Skylion007 · 2025-06-05T16:41:09Z

Could you please re-enable the skipped unit test in test_c10d_nccl.py and see if this update would indeed fix #153517 ?

I think I reenabled the skipped test, please confirm

nWEIdia · 2025-06-05T16:43:27Z

Upstream CI and runners should be able to cover cuda 12.6 + nccl 2.27.3 cases.
Perhaps on our end, we need to test cuda 12.8 + nccl 2.27.3 on B200 nodes. Is there a way for you to test? If not, we can do the test.

Skylion007 · 2025-06-05T16:47:50Z

Upstream CI and runners should be able to cover cuda 12.6 + nccl 2.27.3 cases. Perhaps on our end, we need to test cuda 12.8 + nccl 2.27.3 on B200 nodes. Is there a way for you to test? If not, we can do the test.

I don't have access to any B200 nodes at the moment, and I don't think torch has any B200 CI runners setup yet

nWEIdia · 2025-06-05T16:52:00Z

Upstream CI and runners should be able to cover cuda 12.6 + nccl 2.27.3 cases. Perhaps on our end, we need to test cuda 12.8 + nccl 2.27.3 on B200 nodes. Is there a way for you to test? If not, we can do the test.

I don't have access to any B200 nodes at the moment, and I don't think torch has any B200 CI runners setup yet

Understood, then let's first wait for cu126 signals and worry about B200 runners later.

Skylion007 · 2025-06-06T16:06:19Z

Upstream CI and runners should be able to cover cuda 12.6 + nccl 2.27.3 cases. Perhaps on our end, we need to test cuda 12.8 + nccl 2.27.3 on B200 nodes. Is there a way for you to test? If not, we can do the test.

I don't have access to any B200 nodes at the moment, and I don't think torch has any B200 CI runners setup yet

Understood, then let's first wait for cu126 signals and worry about B200 runners later.

cu126 runners passed btw, so looks like it fixed it

nWEIdia · 2025-06-06T16:13:02Z

Upstream CI and runners should be able to cover cuda 12.6 + nccl 2.27.3 cases. Perhaps on our end, we need to test cuda 12.8 + nccl 2.27.3 on B200 nodes. Is there a way for you to test? If not, we can do the test.

I don't have access to any B200 nodes at the moment, and I don't think torch has any B200 CI runners setup yet

Understood, then let's first wait for cu126 signals and worry about B200 runners later.

cu126 runners passed btw, so looks like it fixed it

That is good news on the cu126 side!

But for cu128, we are internally still testing with the latest pytorch, please wait before merging, thanks!

kwen2501

Thanks! Indeed good news it fixes the bug.
Let's upgrade!

atalman

lgtm

nWEIdia · 2025-06-06T21:35:39Z

The B200 signal with ~ 2 months old pytorch looks good. Testing with current ( ~ days old) pytorch now.

nWEIdia · 2025-06-10T00:35:18Z

The B200 signal with ~ 2 months old pytorch looks good. Testing with current ( ~ days old) pytorch now.

Update: that more recent pytorch also passes my check with this NCCL 2.27.3 version.

nWEIdia · 2025-06-10T22:29:07Z

Looks like a rebase is needed here.

pytorchmergebot · 2025-06-11T20:53:44Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-06-11T20:59:23Z

Merge failed

Reason: 5 jobs have failed, first few of them are: macos-arm64-binary-wheel / wheel-py3_10-cpu-build, macos-arm64-binary-wheel / wheel-py3_9-cpu-build, macos-arm64-binary-wheel / wheel-py3_12-cpu-build, macos-arm64-binary-wheel / wheel-py3_13-cpu-build, macos-arm64-binary-wheel / wheel-py3_11-cpu-build

Details for Dev Infra team

Raised by workflow job

Skylion007 · 2025-06-11T21:02:55Z

@pytorchbot merge -i

pytorchmergebot · 2025-06-11T21:05:00Z

Merge started

Your change will be merged while ignoring the following 6 checks: macos-arm64-binary-wheel / wheel-py3_10-cpu-build, macos-arm64-binary-wheel / wheel-py3_9-cpu-build, macos-arm64-binary-wheel / wheel-py3_12-cpu-build, macos-arm64-binary-wheel / wheel-py3_13t-cpu-build, macos-arm64-binary-wheel / wheel-py3_13-cpu-build, macos-arm64-binary-wheel / wheel-py3_11-cpu-build

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-06-11T21:10:50Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

Lint / workflow-checks / linux-job

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

Skylion007 · 2025-06-11T22:02:48Z

@pytorchbot merge -i

pytorchmergebot · 2025-06-11T22:07:56Z

Merge started

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-06-11T22:19:39Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

pull / linux-jammy-py3.13-clang12 / test (default, 1, 5, linux.4xlarge)

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

Skylion007 · 2025-06-13T20:53:12Z

@pytorchbot merge

pytorchmergebot · 2025-06-13T20:54:56Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-06-14T02:53:34Z

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

Skylion007 · 2025-06-14T14:59:18Z

@pytorchbot merge

pytorchmergebot · 2025-06-14T15:01:04Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Fixes: pytorch#155052 and pytorch#153517 This upgrade is needed to effectively use those symmetric memory kernels anyway. Also fixes some nasty NCCL bugs. Pull Request resolved: pytorch#155233 Approved by: https://github.com/nWEIdia, https://github.com/kwen2501, https://github.com/atalman, https://github.com/eqy

pytorch-bot bot added ciflow/inductor topic: not user facing topic category labels Jun 5, 2025

Skylion007 requested review from atalman, eqy and nWEIdia June 5, 2025 16:23

pytorchbot added the open source label Jun 5, 2025

nWEIdia requested changes Jun 5, 2025

View reviewed changes

Skylion007 force-pushed the skylion007/update-nccl-2-27-3 branch from 32bc4c4 to f16f184 Compare June 5, 2025 16:40

pytorch-bot bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Jun 5, 2025

Skylion007 requested a review from nWEIdia June 5, 2025 16:40

Skylion007 force-pushed the skylion007/update-nccl-2-27-3 branch from f16f184 to a81c050 Compare June 6, 2025 16:06

kwen2501 self-requested a review June 6, 2025 20:16

kwen2501 approved these changes Jun 6, 2025

View reviewed changes

atalman approved these changes Jun 6, 2025

View reviewed changes

kwen2501 mentioned this pull request Jun 6, 2025

Enables NCCL symmetric memory kernels through mempool registration #155134

Closed

nWEIdia approved these changes Jun 10, 2025

View reviewed changes

eqy approved these changes Jun 10, 2025

View reviewed changes

tinglvv mentioned this pull request Jun 10, 2025

Add CUDA 12.9.1 x86 nightly binaries #154980

Closed

nWEIdia mentioned this pull request Jun 10, 2025

[Testing] multigpu tests are still running against CUDA-11 #154119

Closed

Skylion007 force-pushed the skylion007/update-nccl-2-27-3 branch from a81c050 to 9f6496d Compare June 11, 2025 15:04

pytorchmergebot removed the merging label Jun 11, 2025

pytorchmergebot added the merging label Jun 11, 2025

pytorchmergebot removed the merging label Jun 11, 2025

Skylion007 force-pushed the skylion007/update-nccl-2-27-3 branch from 9d55297 to bf677de Compare June 11, 2025 21:19

pytorchmergebot added the merging label Jun 11, 2025

pytorchmergebot removed the merging label Jun 11, 2025

Skylion007 requested a review from tinglvv June 12, 2025 16:03

Skylion007 force-pushed the skylion007/update-nccl-2-27-3 branch from bf677de to 8c7a226 Compare June 13, 2025 20:46

pytorchmergebot added the merging label Jun 13, 2025

Skylion007 added 2 commits June 14, 2025 10:56

[BE]: Update NCCL to 2.27.3

2c09b17

Update cusparseLt in prep of merge

008adee

Skylion007 force-pushed the skylion007/update-nccl-2-27-3 branch from 7a63164 to 008adee Compare June 14, 2025 14:56

pytorchmergebot added the Merged label Jun 14, 2025

pytorchmergebot closed this in a317c63 Jun 14, 2025

pytorchmergebot removed the merging label Jun 14, 2025

[BE]: Update NCCL to 2.27.3 #155233

[BE]: Update NCCL to 2.27.3 #155233

Uh oh!

Conversation

Skylion007 commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/155233

✅ No Failures

Uh oh!

nWEIdia left a comment

Choose a reason for hiding this comment

Uh oh!

Skylion007 commented Jun 5, 2025

Uh oh!

nWEIdia commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Skylion007 commented Jun 5, 2025

Uh oh!

nWEIdia commented Jun 5, 2025

Uh oh!

Skylion007 commented Jun 6, 2025

Uh oh!

nWEIdia commented Jun 6, 2025

Uh oh!

kwen2501 left a comment

Choose a reason for hiding this comment

Uh oh!

atalman left a comment

Choose a reason for hiding this comment

Uh oh!

nWEIdia commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nWEIdia commented Jun 10, 2025

Uh oh!

nWEIdia commented Jun 10, 2025

Uh oh!

pytorchmergebot commented Jun 11, 2025

Merge started

Uh oh!

pytorchmergebot commented Jun 11, 2025

Merge failed

Uh oh!

Skylion007 commented Jun 11, 2025

Uh oh!

pytorchmergebot commented Jun 11, 2025

Merge started

Uh oh!

pytorchmergebot commented Jun 11, 2025

Merge failed

Uh oh!

Skylion007 commented Jun 11, 2025

Uh oh!

pytorchmergebot commented Jun 11, 2025

Merge started

Uh oh!

pytorchmergebot commented Jun 11, 2025

Merge failed

Uh oh!

Skylion007 commented Jun 13, 2025

Uh oh!

pytorchmergebot commented Jun 13, 2025

Merge started

Uh oh!

pytorchmergebot commented Jun 14, 2025

Uh oh!

Skylion007 commented Jun 14, 2025

Uh oh!

pytorchmergebot commented Jun 14, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Skylion007 commented Jun 5, 2025 •

edited

Loading

pytorch-bot bot commented Jun 5, 2025 •

edited

Loading

nWEIdia commented Jun 5, 2025 •

edited

Loading

nWEIdia commented Jun 6, 2025 •

edited

Loading