Skip to content

[CP][BE] Remove _AttentionContextParallel #162539

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
fegin wants to merge 6 commits into from
Closed

[CP][BE] Remove _AttentionContextParallel #162539

fegin wants to merge 6 commits into from

Conversation

@fegin
Copy link
Contributor

@fegin fegin commented Sep 9, 2025 •
edited
Loading

@fegin
Update
5eba6de 
[ghstack-poisoned]
@pytorch-bot
Copy link

pytorch-bot bot commented Sep 9, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/162539

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 5844d2a with merge base 7a0f933 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added ciflow/inductor oncall: distributed Add this issue/PR to distributed oncall triage queue labels Sep 9, 2025
This was referenced Sep 9, 2025
Closed
Closed
Closed
Closed
ezyang
ezyang approved these changes Sep 10, 2025
View reviewed changes
Copy link
Contributor

@ezyang ezyang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming you have done BC due diligence.

@fegin
Update
c568d1a 
[ghstack-poisoned]
@fegin fegin added the topic: not user facing topic category label Sep 16, 2025
@fegin
Copy link
Contributor Author

fegin commented Sep 16, 2025

_AttentionContextParallel is never discussed publicly and it is never used in our example or TorchTitan. It is only used by our unittest. We can remove this.

@fegin
Update
57fc3cb 
[ghstack-poisoned]
@fegin
Copy link
Contributor Author

fegin commented Sep 16, 2025

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Sep 16, 2025
@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Sep 16, 2025

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Sep 16, 2025

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

Dig deeper by viewing the failures on hud

Details for Dev Infra team Raised by workflow job

Failing merge rule: Core Maintainers

@fegin
Update
39030d6 
[ghstack-poisoned]
@fegin
Copy link
Contributor Author

fegin commented Sep 16, 2025

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Sep 16, 2025

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Sep 16, 2025

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / win-vs2022-cuda12.6-py3 / build

Details for Dev Infra team Raised by workflow job

@fegin
Update
30d8d90 
[ghstack-poisoned]
This was referenced Sep 16, 2025
Closed
Closed
@fegin
Copy link
Contributor Author

fegin commented Sep 17, 2025

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Sep 18, 2025

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

pytorchmergebot pushed a commit that referenced this pull request Sep 18, 2025
@fegin @pytorchmergebot
[CP] Remove the need of recording cp_dim in the global var (#162540)
096d35c 
This information can be obtained during the dispatching.

Pull Request resolved: #162540
Approved by: https://github.com/ezyang, https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: #162539
pytorchmergebot pushed a commit that referenced this pull request Sep 19, 2025
@fegin @pytorchmergebot
[CP][BE] Correct the names of some tests (#162541)
cd4303a 
We are not doing ring attention but only using allgather to do CP for Flex.
Pull Request resolved: #162541
Approved by: https://github.com/ezyang, https://github.com/Skylion007, https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: #162539, #162540
pytorchmergebot pushed a commit that referenced this pull request Sep 19, 2025
@fegin @pytorchmergebot
[CP][BE] Cosmetic refactors for CP code base (#163115)
0815091 
Summary:
This PR is extracted from #162542, to make the original PR
easier to review. This PR only contains cosmetic changes.

Pull Request resolved: #163115
Approved by: https://github.com/tianyu-l
ghstack dependencies: #162539, #162540, #162541
pytorchmergebot pushed a commit that referenced this pull request Sep 19, 2025
@fegin @pytorchmergebot
[CP][BE] Correct an incorrect docstring (#163131)
093f064 
Pull Request resolved: #163131
Approved by: https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: #162539, #162540, #162541, #163115
pytorchmergebot pushed a commit that referenced this pull request Sep 20, 2025
@fegin @pytorchmergebot
[CP] Fix cuDNN CP LSE dimension bug (#163231)
d55c9d5 
We should only unsqueeze if necessary.

Fix #162743

Pull Request resolved: #163231
Approved by: https://github.com/eqy
ghstack dependencies: #162539, #162540, #162541, #163115, #163131
mansiag05 pushed a commit to mansiag05/pytorch that referenced this pull request Sep 22, 2025
@fegin @mansiag05
[CP][BE] Remove _AttentionContextParallel (pytorch#162539)
091c7d6 
This is not an API we want to support.

Pull Request resolved: pytorch#162539
Approved by: https://github.com/ezyang, https://github.com/tianyu-l
mansiag05 pushed a commit to mansiag05/pytorch that referenced this pull request Sep 22, 2025
@fegin @mansiag05
[CP] Remove the need of recording cp_dim in the global var (pytorch#1…
c9415f2 
…62540)

This information can be obtained during the dispatching.

Pull Request resolved: pytorch#162540
Approved by: https://github.com/ezyang, https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539
mansiag05 pushed a commit to mansiag05/pytorch that referenced this pull request Sep 22, 2025
@fegin @mansiag05
[CP][BE] Correct the names of some tests (pytorch#162541)
61034ce 
We are not doing ring attention but only using allgather to do CP for Flex.
Pull Request resolved: pytorch#162541
Approved by: https://github.com/ezyang, https://github.com/Skylion007, https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539, pytorch#162540
mansiag05 pushed a commit to mansiag05/pytorch that referenced this pull request Sep 22, 2025
@fegin @mansiag05
[CP][BE] Cosmetic refactors for CP code base (pytorch#163115)
83aa65c 
Summary:
This PR is extracted from pytorch#162542, to make the original PR
easier to review. This PR only contains cosmetic changes.

Pull Request resolved: pytorch#163115
Approved by: https://github.com/tianyu-l
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541
mansiag05 pushed a commit to mansiag05/pytorch that referenced this pull request Sep 22, 2025
@fegin @mansiag05
[CP][BE] Correct an incorrect docstring (pytorch#163131)
d148bd2 
Pull Request resolved: pytorch#163131
Approved by: https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541, pytorch#163115
mansiag05 pushed a commit to mansiag05/pytorch that referenced this pull request Sep 22, 2025
@fegin @mansiag05
[CP] Fix cuDNN CP LSE dimension bug (pytorch#163231)
b49fdb0 
We should only unsqueeze if necessary.

Fix pytorch#162743

Pull Request resolved: pytorch#163231
Approved by: https://github.com/eqy
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541, pytorch#163115, pytorch#163131
cleonard530 pushed a commit to cleonard530/pytorch that referenced this pull request Sep 22, 2025
@fegin @cleonard530
[CP][BE] Remove _AttentionContextParallel (pytorch#162539)
bff5f14 
This is not an API we want to support.

Pull Request resolved: pytorch#162539
Approved by: https://github.com/ezyang, https://github.com/tianyu-l
cleonard530 pushed a commit to cleonard530/pytorch that referenced this pull request Sep 22, 2025
@fegin @cleonard530
[CP] Remove the need of recording cp_dim in the global var (pytorch#1…
b3f1d0b 
…62540)

This information can be obtained during the dispatching.

Pull Request resolved: pytorch#162540
Approved by: https://github.com/ezyang, https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539
cleonard530 pushed a commit to cleonard530/pytorch that referenced this pull request Sep 22, 2025
@fegin @cleonard530
[CP][BE] Correct the names of some tests (pytorch#162541)
4221f5e 
We are not doing ring attention but only using allgather to do CP for Flex.
Pull Request resolved: pytorch#162541
Approved by: https://github.com/ezyang, https://github.com/Skylion007, https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539, pytorch#162540
cleonard530 pushed a commit to cleonard530/pytorch that referenced this pull request Sep 22, 2025
@fegin @cleonard530
[CP][BE] Cosmetic refactors for CP code base (pytorch#163115)
87b1690 
Summary:
This PR is extracted from pytorch#162542, to make the original PR
easier to review. This PR only contains cosmetic changes.

Pull Request resolved: pytorch#163115
Approved by: https://github.com/tianyu-l
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541
cleonard530 pushed a commit to cleonard530/pytorch that referenced this pull request Sep 22, 2025
@fegin @cleonard530
[CP][BE] Correct an incorrect docstring (pytorch#163131)
0d0e820 
Pull Request resolved: pytorch#163131
Approved by: https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541, pytorch#163115
cleonard530 pushed a commit to cleonard530/pytorch that referenced this pull request Sep 22, 2025
@fegin @cleonard530
[CP] Fix cuDNN CP LSE dimension bug (pytorch#163231)
74f16c2 
We should only unsqueeze if necessary.

Fix pytorch#162743

Pull Request resolved: pytorch#163231
Approved by: https://github.com/eqy
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541, pytorch#163115, pytorch#163131
dsashidh pushed a commit to dsashidh/pytorch that referenced this pull request Sep 26, 2025
@fegin @dsashidh
[CP][BE] Remove _AttentionContextParallel (pytorch#162539)
482814b 
This is not an API we want to support.

Pull Request resolved: pytorch#162539
Approved by: https://github.com/ezyang, https://github.com/tianyu-l
dsashidh pushed a commit to dsashidh/pytorch that referenced this pull request Sep 26, 2025
@fegin @dsashidh
[CP] Remove the need of recording cp_dim in the global var (pytorch#1…
bc3824a 
…62540)

This information can be obtained during the dispatching.

Pull Request resolved: pytorch#162540
Approved by: https://github.com/ezyang, https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539
dsashidh pushed a commit to dsashidh/pytorch that referenced this pull request Sep 26, 2025
@fegin @dsashidh
[CP][BE] Correct the names of some tests (pytorch#162541)
035519f 
We are not doing ring attention but only using allgather to do CP for Flex.
Pull Request resolved: pytorch#162541
Approved by: https://github.com/ezyang, https://github.com/Skylion007, https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539, pytorch#162540
dsashidh pushed a commit to dsashidh/pytorch that referenced this pull request Sep 26, 2025
@fegin @dsashidh
[CP][BE] Cosmetic refactors for CP code base (pytorch#163115)
2c1c783 
Summary:
This PR is extracted from pytorch#162542, to make the original PR
easier to review. This PR only contains cosmetic changes.

Pull Request resolved: pytorch#163115
Approved by: https://github.com/tianyu-l
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541
dsashidh pushed a commit to dsashidh/pytorch that referenced this pull request Sep 26, 2025
@fegin @dsashidh
[CP][BE] Correct an incorrect docstring (pytorch#163131)
cc3feee 
Pull Request resolved: pytorch#163131
Approved by: https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541, pytorch#163115
dsashidh pushed a commit to dsashidh/pytorch that referenced this pull request Sep 26, 2025
@fegin @dsashidh
[CP] Fix cuDNN CP LSE dimension bug (pytorch#163231)
c49d1a8 
We should only unsqueeze if necessary.

Fix pytorch#162743

Pull Request resolved: pytorch#163231
Approved by: https://github.com/eqy
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541, pytorch#163115, pytorch#163131
@github-actions github-actions bot deleted the gh/fegin/315/head branch October 19, 2025 02:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ezyang ezyang ezyang approved these changes

@tianyu-l tianyu-l tianyu-l approved these changes

Assignees

No one assigned

Labels

ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request Merged oncall: distributed Add this issue/PR to distributed oncall triage queue topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@fegin @pytorchmergebot @ezyang @tianyu-l