Support ddp zero hook XCCL path #159240

Chao1Han · 2025-07-28T02:50:25Z

XCCL backend no #62300 issue, add xccl path here.

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta

pytorch-bot · 2025-07-28T02:50:29Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/159240

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 62eb2ca with merge base f15ada5 ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

trunk / macos-py3-arm64 / test (default, 2, 3, macos-m1-stable) (gh) (trunk failure)
'Test'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

guangyey

LGTM.
Adding a test case is better.

pytorch-bot · 2025-07-28T03:11:05Z

To add the ciflow label ciflow/xpu please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

Chao1Han · 2025-07-28T03:25:46Z

Test case https://github.com/pytorch/pytorch/blob/main/test/distributed/optim/test_zero_redundancy_optimizer.py could verify ddp zero hook XCCL path, and test file no need device agnostic.

guangyey · 2025-07-28T06:40:54Z

@pytorchbot rebase

pytorchmergebot · 2025-07-28T06:42:25Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-07-28T06:42:28Z

Successfully rebased elastic onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout elastic && git pull --rebase)

Skylion007 · 2025-07-28T16:07:05Z

torch/distributed/algorithms/ddp_comm_hooks/ddp_zero_hook.py

+    # NCCL/HCCL/XCCL backend for now; see https://github.com/pytorch/pytorch/issues/62300
    pg = dist.get_backend(ddp_ref().process_group)  # type: ignore[union-attr]
-    if (pg != dist.Backend.NCCL) and (pg != "hccl"):
+    if (pg != dist.Backend.NCCL) and (pg != "hccl") and (pg != "xccl"):


Shouldn't we just blacklist Gloo in the future if that is the problematic backend?

We don't need to update any XFAIL or SKIPIF in tests, right?

Sure, let me blacklist Gloo and try ci, no need update any tests.

guangyey · 2025-07-29T02:40:55Z

@Skylion007 Thanks very much~

guangyey · 2025-07-30T01:31:30Z

@pytorchbot merge

guangyey · 2025-08-13T02:03:09Z

@pytorchbot rebase -b main

pytorchmergebot · 2025-08-13T02:04:37Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

pytorchmergebot · 2025-08-13T02:04:39Z

Successfully rebased elastic onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout elastic && git pull --rebase)

guangyey · 2025-08-13T02:13:19Z

@pytorchbot merge

pytorch-bot · 2025-08-13T02:13:31Z

To add the ciflow label ciflow/xpu please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

pytorchmergebot · 2025-08-13T02:15:12Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-08-13T08:13:50Z

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

guangyey · 2025-08-13T12:24:40Z

@pytorchbot merge

pytorchmergebot · 2025-08-13T12:26:31Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

XCCL backend no #62300 issue, add xccl path here. Pull Request resolved: #159240 Approved by: https://github.com/guangyey, https://github.com/Skylion007, https://github.com/EikanWang

XCCL backend no pytorch#62300 issue, add xccl path here. Pull Request resolved: pytorch#159240 Approved by: https://github.com/guangyey, https://github.com/Skylion007, https://github.com/EikanWang

pytorch-bot bot added oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (ddp) release notes category labels Jul 28, 2025

guangyey added this to PyTorch Intel Jul 28, 2025

pytorchbot added the open source label Jul 28, 2025

guangyey approved these changes Jul 28, 2025

View reviewed changes

guangyey added the ciflow/xpu Run XPU CI tasks label Jul 28, 2025

pytorch-bot bot removed the ciflow/xpu Run XPU CI tasks label Jul 28, 2025

guangyey added the ciflow/xpu Run XPU CI tasks label Jul 28, 2025

guangyey marked this pull request as ready for review July 28, 2025 03:12

guangyey requested review from d4l3k and kwen2501 July 28, 2025 03:14

guangyey moved this to Review Required in PyTorch Intel Jul 28, 2025

pytorchmergebot force-pushed the elastic branch from f4826ef to 3eb01d6 Compare July 28, 2025 06:42

pytorch-bot bot removed the ciflow/xpu Run XPU CI tasks label Jul 28, 2025

guangyey added the ciflow/xpu Run XPU CI tasks label Jul 28, 2025

guangyey requested a review from ezyang July 28, 2025 06:44

Skylion007 approved these changes Jul 28, 2025

View reviewed changes

pytorch-bot bot removed the ciflow/xpu Run XPU CI tasks label Jul 29, 2025

guangyey added the ciflow/xpu Run XPU CI tasks label Jul 29, 2025

guangyey approved these changes Jul 29, 2025

View reviewed changes

EikanWang approved these changes Jul 29, 2025

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 30, 2025

pytorch-bot bot removed ciflow/trunk Trigger trunk jobs on your pull request ciflow/xpu Run XPU CI tasks labels Aug 13, 2025

Chao1Han added 2 commits August 13, 2025 02:04

support optim cases

e992eaa

blacklist gloo

62eb2ca

pytorchmergebot force-pushed the elastic branch from ec4c619 to 62eb2ca Compare August 13, 2025 02:04

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Aug 13, 2025

guangyey added ciflow/xpu Run XPU CI tasks and removed ciflow/trunk Trigger trunk jobs on your pull request labels Aug 13, 2025

pytorch-bot bot removed the ciflow/xpu Run XPU CI tasks label Aug 13, 2025

guangyey added ciflow/xpu Run XPU CI tasks ciflow/trunk Trigger trunk jobs on your pull request labels Aug 13, 2025

pytorchmergebot added the merging label Aug 13, 2025

pytorchmergebot closed this in 42e51cd Aug 13, 2025

pytorchmergebot added the Merged label Aug 13, 2025

github-project-automation bot moved this from Approved to Done in PyTorch Intel Aug 13, 2025

pytorchmergebot removed the merging label Aug 13, 2025

Support ddp zero hook XCCL path #159240

Support ddp zero hook XCCL path #159240

Uh oh!

Conversation

Chao1Han commented Jul 28, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/159240

✅ You can merge normally! (1 Unrelated Failure)

Uh oh!

guangyey left a comment

Choose a reason for hiding this comment

Uh oh!

pytorch-bot bot commented Jul 28, 2025

Uh oh!

Chao1Han commented Jul 28, 2025

Uh oh!

guangyey commented Jul 28, 2025

Uh oh!

pytorchmergebot commented Jul 28, 2025

Uh oh!

pytorchmergebot commented Jul 28, 2025

Uh oh!

Skylion007 Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Skylion007 Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Chao1Han Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

guangyey commented Jul 29, 2025

Uh oh!

guangyey commented Jul 30, 2025

Uh oh!

guangyey commented Aug 13, 2025

Uh oh!

pytorchmergebot commented Aug 13, 2025

Uh oh!

pytorchmergebot commented Aug 13, 2025

Uh oh!

guangyey commented Aug 13, 2025

Uh oh!

pytorch-bot bot commented Aug 13, 2025

Uh oh!

pytorchmergebot commented Aug 13, 2025

Merge started

Uh oh!

pytorchmergebot commented Aug 13, 2025

Uh oh!

guangyey commented Aug 13, 2025

Uh oh!

pytorchmergebot commented Aug 13, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Chao1Han commented Jul 28, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Jul 28, 2025 •

edited

Loading