Skip to content

[c10d] Make barrier as a custom op#79777

Closed
alanwaketan wants to merge 3 commits intogh/alanwaketan/41/basefrom
gh/alanwaketan/41/head
Closed

[c10d] Make barrier as a custom op#79777
alanwaketan wants to merge 3 commits intogh/alanwaketan/41/basefrom
gh/alanwaketan/41/head

Conversation

@alanwaketan
Copy link
Copy Markdown
Collaborator

@alanwaketan alanwaketan commented Jun 17, 2022 •
edited
Loading

Stack from ghstack (oldest at bottom):

Summary:
This patch makes barrier as a custom op such that it's dispatcher
passable. It's one part of the effort to route comm ops to the dispatcher
such that tracing mechanisms that relies on the dispatcher can trace them,
e.g., LazyTensor and AOTAutograd.

Test Plan:
python test/distributed/test_c10d_nccl.py -k test_nccl_barrier
...and other existing distributed tests.

[c10d] Make alltoall as a custom op
71751b1 
Summary:
This patch makes alltoall as a custom op such that it's dispatcher
passable. It's one part of the effort to route comm ops to the dispatcher
such that tracing mechanisms that relies on the dispatcher can trace them,
e.g., LazyTensor and AOTAutograd.

Test Plan:
python test/distributed/test_c10d_nccl.py -k test_nccl_barrier
...and other existing distributed tests.

[ghstack-poisoned]
This was referenced Jun 17, 2022
Closed
Closed
This was referenced Jun 17, 2022
Closed
Closed
@facebook-github-bot
Copy link
Copy Markdown
Contributor

facebook-github-bot commented Jun 17, 2022 •
edited
Loading

🔗 Helpful links

✅ No Failures (0 Pending)

As of commit f21f40e (more details on the Dr. CI page):

Expand to see more

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

This was referenced Jun 17, 2022
Closed
Closed
Closed
@facebook-github-bot facebook-github-bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Jun 17, 2022
alanwaketan pushed a commit that referenced this pull request Jun 17, 2022
[c10d] Make alltoall as a custom op
67f3f33 
Summary:
This patch makes alltoall as a custom op such that it's dispatcher
passable. It's one part of the effort to route comm ops to the dispatcher
such that tracing mechanisms that relies on the dispatcher can trace them,
e.g., LazyTensor and AOTAutograd.

Test Plan:
python test/distributed/test_c10d_nccl.py -k test_nccl_barrier
...and other existing distributed tests.

ghstack-source-id: aad4231
Pull Request resolved: #79777
@alanwaketan alanwaketan changed the title [c10d] Make alltoall as a custom op [c10d] Make barrier as a custom op Jun 17, 2022
@alanwaketan alanwaketan mentioned this pull request Jun 17, 2022
Closed
Update on "[c10d] Make barrier as a custom op"
c7d7aa1 
Summary:
This patch makes barrier as a custom op such that it's dispatcher
passable. It's one part of the effort to route comm ops to the dispatcher
such that tracing mechanisms that relies on the dispatcher can trace them,
e.g., LazyTensor and AOTAutograd.

Test Plan:
python test/distributed/test_c10d_nccl.py -k test_nccl_barrier
...and other existing distributed tests.

[ghstack-poisoned]
@alanwaketan alanwaketan requested a review from ezyang June 22, 2022 23:45
mrshenli
mrshenli reviewed Jun 24, 2022
View reviewed changes
torch/csrc/distributed/c10d/Ops.cpp
"alltoall_",
dispatch(c10::DispatchKey::CompositeExplicitAutograd, alltoall_));
m.def(
"barrier",
Copy link
Copy Markdown
Contributor

@mrshenli mrshenli Jun 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing a trailing underscore?

Copy link
Copy Markdown
Collaborator Author

@alanwaketan alanwaketan Jun 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

torch/csrc/distributed/c10d/Ops.cpp
AllToAllOptions{std::chrono::milliseconds(timeout)});
}

c10::intrusive_ptr<ProcessGroup::Work> barrier(
Copy link
Copy Markdown
Contributor

@mrshenli mrshenli Jun 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing a trailing underscore?

Copy link
Copy Markdown
Collaborator Author

@alanwaketan alanwaketan Jun 24, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

barrier is not a in-place op. Therefore, it's not necessary to add a trailing _.

Copy link
Copy Markdown
Contributor

@mrshenli mrshenli Jun 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see, trailing score is for inplace ops as in Python APIs. Thought it was a dispatcher convention.

Update on "[c10d] Make barrier as a custom op"
f21f40e 
Summary:
This patch makes barrier as a custom op such that it's dispatcher
passable. It's one part of the effort to route comm ops to the dispatcher
such that tracing mechanisms that relies on the dispatcher can trace them,
e.g., LazyTensor and AOTAutograd.

Test Plan:
python test/distributed/test_c10d_nccl.py -k test_nccl_barrier
...and other existing distributed tests.

[ghstack-poisoned]
@alanwaketan
Copy link
Copy Markdown
Collaborator Author

alanwaketan commented Jun 27, 2022

@pytorchbot merge --green

@alanwaketan
Copy link
Copy Markdown
Collaborator Author

alanwaketan commented Jun 27, 2022

Thanks for approving this pull request, Shen and Wanchao.

@pytorchmergebot
Copy link
Copy Markdown
Collaborator

pytorchmergebot commented Jun 27, 2022

@pytorchbot successfully started a merge job. Check the current status here

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jun 27, 2022

Hey @alanwaketan.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

facebook-github-bot pushed a commit that referenced this pull request Jun 29, 2022
@facebook-github-bot
[c10d] Make barrier as a custom op (#79777) (#79777)
c20830f 
Summary:
This patch makes barrier as a custom op such that it's dispatcher
passable. It's one part of the effort to route comm ops to the dispatcher
such that tracing mechanisms that relies on the dispatcher can trace them,
e.g., LazyTensor and AOTAutograd.

Pull Request resolved: #79777
Approved by: https://github.com/mrshenli, https://github.com/wanchaol

Test Plan:
contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/238eaf20949cefa4cb61a9d61893ab136fbdb3a2

Test plan from GitHub:
python test/distributed/test_c10d_nccl.py -k test_nccl_barrier
...and other existing distributed tests.

Reviewed By: atalman

Differential Revision: D37455834

Pulled By: alanwaketan

fbshipit-source-id: 753702a7bfb50bdca701b209ba3e8becd5ef2921
@facebook-github-bot facebook-github-bot deleted the gh/alanwaketan/41/head branch June 30, 2022 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wanchaol wanchaol wanchaol approved these changes

@mrshenli mrshenli mrshenli approved these changes

@pritamdamania87 pritamdamania87 Awaiting requested review from pritamdamania87

@aazzolini aazzolini Awaiting requested review from aazzolini

@wconstab wconstab Awaiting requested review from wconstab

@ezyang ezyang Awaiting requested review from ezyang

Assignees

No one assigned

Labels

cla signed Merged oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (c10d) release notes category topic: new features topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@alanwaketan @facebook-github-bot @pytorchmergebot @wanchaol @mrshenli