[BE]: Update NCCL version to 2.28.7 by Skylion007 · Pull Request #166174 · pytorch/pytorch

Skylion007 · 2025-10-24T17:38:21Z

Update NCCL version to 2.28.7. Fixes issues that prevented 2.28.3 update in #162351

pytorch-bot · 2025-10-24T17:38:25Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/166174

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 Cancelled Jobs

As of commit b6e6ecb with merge base 380d440 ():

CANCELLED JOBS - The following jobs were cancelled. Please retry:

pull / cuda12.8-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu) (gh)
##[error]The operation was canceled.
pull / linux-jammy-py3.10-gcc11 / test (distributed, 1, 2, linux.2xlarge) (gh)
##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

eqy · 2025-10-24T19:57:27Z

We had a number of issues with fixes in-flight, requested a review from @nWEIdia to confirm if 2.28.7 is a good target or if we should wait for a slightly later version.

nWEIdia · 2025-10-24T19:57:56Z

For this version, we have a known performance regression signals, I recommend waiting for a few days for NCCL team's update.

kwen2501 · 2025-11-14T21:15:57Z

@nWEIdia @eqy wondering if you heard any update?

nWEIdia · 2025-11-14T22:39:23Z

@nWEIdia @eqy wondering if you heard any update?

We should be good to go with 2.28.9 https://github.com/NVIDIA/nccl/tree/v2.28.9-1

ngimel · 2025-11-14T22:57:00Z

@nWEIdia can you please make a PR?

nWEIdia · 2025-11-14T23:00:52Z

@nWEIdia can you please make a PR?

Yes, I will do so, just realized that we tagged 2.28.9 but the PYPI wheels are yet to be published. https://pypi.org/project/nvidia-nccl-cu13/

Skylion007 · 2025-11-15T19:42:15Z

@nWEIdia BTW, @atalman Finally added self service PR update wheels: See this PR of mine as an example, will need to land first when you do upgrade: pytorch/test-infra#7415

nWEIdia · 2025-11-18T17:59:39Z

@nWEIdia can you please make a PR?

Hi @ngimel I created the PR: #168091 please help take a look, thanks!

….28.9-1" (This PR will be rebased on #166174) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. [ghstack-poisoned]

This PR updates the NCCL version for CUDA13 from 2.27.7 to 2.28.9. 2.28.9 release notes: https://github.com/NVIDIA/nccl/releases/tag/v2.28.9-1 2.28.7 release notes: https://github.com/NVIDIA/nccl/releases/tag/v2.28.7-1 2.28.3 release notes: https://github.com/NVIDIA/nccl/releases/tag/v2.28.3-1 CUDA 12 remains at 2.27.5 and is untouched by this PR. Reference PR: #166174 Pull Request resolved: #168091 Approved by: https://github.com/atalman

….28.9-1" (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. [ghstack-poisoned]

…dged with nccl 2.28.9-1" (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 [ghstack-poisoned]

….28.9-1" (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 [ghstack-poisoned]

This PR updates the NCCL version for CUDA13 from 2.27.7 to 2.28.9. 2.28.9 release notes: https://github.com/NVIDIA/nccl/releases/tag/v2.28.9-1 2.28.7 release notes: https://github.com/NVIDIA/nccl/releases/tag/v2.28.7-1 2.28.3 release notes: https://github.com/NVIDIA/nccl/releases/tag/v2.28.3-1 CUDA 12 remains at 2.27.5 and is untouched by this PR. Reference PR: #166174 Pull Request resolved: #168091 Approved by: https://github.com/atalman

…dged with nccl 2.28.9-1" (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 [ghstack-poisoned]

….28.9-1" (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 [ghstack-poisoned]

…dged with nccl 2.28.9-1" (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 [ghstack-poisoned]

….28.9-1" (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 [ghstack-poisoned]

…68129) (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 Pull Request resolved: #168129 Approved by: https://github.com/kwen2501, https://github.com/ngimel, https://github.com/atalman

…torch#168129) (This PR will be rebased on pytorch#166174) (There are other PR which updates NCCL version: pytorch#168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves pytorch#167682 Pull Request resolved: pytorch#168129 Approved by: https://github.com/kwen2501, https://github.com/ngimel, https://github.com/atalman

…dged with nccl 2.28.9-1" (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 [ghstack-poisoned]

….28.9-1" (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 [ghstack-poisoned]

…68129) (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 Pull Request resolved: #168129 Approved by: https://github.com/kwen2501, https://github.com/ngimel, https://github.com/atalman

…dged with nccl 2.28.9-1" (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 [ghstack-poisoned]

….28.9-1" (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 [ghstack-poisoned]

…dged with nccl 2.28.9-1" (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 [ghstack-poisoned]

….28.9-1" (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 [ghstack-poisoned]

…dged with nccl 2.28.9-1" (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 [ghstack-poisoned]

….28.9-1" (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 [ghstack-poisoned]

…68129) (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 Pull Request resolved: #168129 Approved by: https://github.com/kwen2501, https://github.com/ngimel, https://github.com/atalman

…68129) (This PR will be rebased on #166174) (There are other PR which updates NCCL version: #168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves #167682 Pull Request resolved: #168129 Approved by: https://github.com/kwen2501, https://github.com/ngimel, https://github.com/atalman (cherry picked from commit c907c77)

…torch#168129) (This PR will be rebased on pytorch#166174) (There are other PR which updates NCCL version: pytorch#168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With pytorch#1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves pytorch#167682 Pull Request resolved: pytorch#168129 Approved by: https://github.com/kwen2501, https://github.com/ngimel, https://github.com/atalman

[BE]: Update NCCL version to 2.28.7

b6e6ecb

Skylion007 requested review from a team and jeffdaily as code owners October 24, 2025 17:38

pytorch-bot bot added ciflow/inductor topic: not user facing topic category labels Oct 24, 2025

Skylion007 added the better-engineering Relatively self-contained tasks for better engineering contributors label Oct 24, 2025

Skylion007 requested review from atalman, malfet and ngimel October 24, 2025 17:39

pytorchbot added the open source label Oct 24, 2025

Skylion007 requested a review from eqy October 24, 2025 19:51

eqy requested a review from nWEIdia October 24, 2025 19:56

Skylion007 closed this Oct 24, 2025

Skylion007 reopened this Oct 28, 2025

jerryzh168 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Oct 30, 2025

kwen2501 self-requested a review November 14, 2025 21:16

nWEIdia mentioned this pull request Nov 18, 2025

[CI][CUDA][Distributed] Update NCCL to 2.28.9 for CUDA13 #168091

Closed

fduwjj mentioned this pull request Nov 25, 2025

[c10d][Sym mem] Make nccl backend full fledged with nccl 2.28.9-1 #168129

Closed

pytorchbot mentioned this pull request Dec 13, 2025

[c10d][Sym mem] Make nccl backend full fledged with nccl 2.28.9-1 #170389

Open

Skylion007 closed this Jan 2, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BE]: Update NCCL version to 2.28.7#166174

[BE]: Update NCCL version to 2.28.7#166174
Skylion007 wants to merge 1 commit intopytorch:mainfrom
Skylion007:skylion007/update-nccl-2-28-7

Skylion007 commented Oct 24, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Oct 24, 2025 •

edited

Loading

Uh oh!

eqy commented Oct 24, 2025

Uh oh!

nWEIdia commented Oct 24, 2025

Uh oh!

kwen2501 commented Nov 14, 2025

Uh oh!

nWEIdia commented Nov 14, 2025

Uh oh!

ngimel commented Nov 14, 2025

Uh oh!

nWEIdia commented Nov 14, 2025

Uh oh!

Skylion007 commented Nov 15, 2025 •

edited

Loading

Uh oh!

nWEIdia commented Nov 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

Skylion007 commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/166174

❌ 2 Cancelled Jobs

Uh oh!

eqy commented Oct 24, 2025

Uh oh!

nWEIdia commented Oct 24, 2025

Uh oh!

kwen2501 commented Nov 14, 2025

Uh oh!

nWEIdia commented Nov 14, 2025

Uh oh!

ngimel commented Nov 14, 2025

Uh oh!

nWEIdia commented Nov 14, 2025

Uh oh!

Skylion007 commented Nov 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nWEIdia commented Nov 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Skylion007 commented Oct 24, 2025 •

edited

Loading

pytorch-bot bot commented Oct 24, 2025 •

edited

Loading

Skylion007 commented Nov 15, 2025 •

edited

Loading