Skip to content

[CP] Remove the need of recording cp_dim in the global var #162540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
fegin wants to merge 6 commits into from
Closed

[CP] Remove the need of recording cp_dim in the global var #162540

fegin wants to merge 6 commits into from

Conversation

@fegin
Copy link
Contributor

@fegin fegin commented Sep 9, 2025 •
edited
Loading

@fegin
Update
ef47b55 
[ghstack-poisoned]
@pytorch-bot
Copy link

pytorch-bot bot commented Sep 9, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/162540

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 1 Pending

As of commit 2cbd999 with merge base 7a0f933 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added ciflow/inductor oncall: distributed Add this issue/PR to distributed oncall triage queue labels Sep 9, 2025
This was referenced Sep 9, 2025
Closed
Closed
Closed
Closed
@fegin
Update
543e96a 
[ghstack-poisoned]
@fegin fegin added the topic: not user facing topic category label Sep 16, 2025
@fegin
Update
9c6033e 
[ghstack-poisoned]
@fegin
Update
7f6e074 
[ghstack-poisoned]
@fegin
Update
1c93ee4 
[ghstack-poisoned]
This was referenced Sep 16, 2025
Closed
Closed
@fegin fegin mentioned this pull request Sep 17, 2025
Closed
@fegin
Update
2cbd999 
[ghstack-poisoned]
@fegin fegin mentioned this pull request Sep 18, 2025
Closed
XilunWu
XilunWu approved these changes Sep 18, 2025
View reviewed changes
Copy link
Contributor

@XilunWu XilunWu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: we still have multiple hardcoding sites (seq_dim=2 or seq_dim = 2)

@fegin
Copy link
Contributor Author

fegin commented Sep 18, 2025

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Sep 18, 2025
@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Sep 18, 2025

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

pytorchmergebot pushed a commit that referenced this pull request Sep 19, 2025
@fegin @pytorchmergebot
[CP][BE] Correct the names of some tests (#162541)
cd4303a 
We are not doing ring attention but only using allgather to do CP for Flex.
Pull Request resolved: #162541
Approved by: https://github.com/ezyang, https://github.com/Skylion007, https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: #162539, #162540
pytorchmergebot pushed a commit that referenced this pull request Sep 19, 2025
@fegin @pytorchmergebot
[CP][BE] Cosmetic refactors for CP code base (#163115)
0815091 
Summary:
This PR is extracted from #162542, to make the original PR
easier to review. This PR only contains cosmetic changes.

Pull Request resolved: #163115
Approved by: https://github.com/tianyu-l
ghstack dependencies: #162539, #162540, #162541
pytorchmergebot pushed a commit that referenced this pull request Sep 19, 2025
@fegin @pytorchmergebot
[CP][BE] Correct an incorrect docstring (#163131)
093f064 
Pull Request resolved: #163131
Approved by: https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: #162539, #162540, #162541, #163115
pytorchmergebot pushed a commit that referenced this pull request Sep 20, 2025
@fegin @pytorchmergebot
[CP] Fix cuDNN CP LSE dimension bug (#163231)
d55c9d5 
We should only unsqueeze if necessary.

Fix #162743

Pull Request resolved: #163231
Approved by: https://github.com/eqy
ghstack dependencies: #162539, #162540, #162541, #163115, #163131
mansiag05 pushed a commit to mansiag05/pytorch that referenced this pull request Sep 22, 2025
@fegin @mansiag05
[CP] Remove the need of recording cp_dim in the global var (pytorch#1…
c9415f2 
…62540)

This information can be obtained during the dispatching.

Pull Request resolved: pytorch#162540
Approved by: https://github.com/ezyang, https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539
mansiag05 pushed a commit to mansiag05/pytorch that referenced this pull request Sep 22, 2025
@fegin @mansiag05
[CP][BE] Correct the names of some tests (pytorch#162541)
61034ce 
We are not doing ring attention but only using allgather to do CP for Flex.
Pull Request resolved: pytorch#162541
Approved by: https://github.com/ezyang, https://github.com/Skylion007, https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539, pytorch#162540
mansiag05 pushed a commit to mansiag05/pytorch that referenced this pull request Sep 22, 2025
@fegin @mansiag05
[CP][BE] Cosmetic refactors for CP code base (pytorch#163115)
83aa65c 
Summary:
This PR is extracted from pytorch#162542, to make the original PR
easier to review. This PR only contains cosmetic changes.

Pull Request resolved: pytorch#163115
Approved by: https://github.com/tianyu-l
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541
mansiag05 pushed a commit to mansiag05/pytorch that referenced this pull request Sep 22, 2025
@fegin @mansiag05
[CP][BE] Correct an incorrect docstring (pytorch#163131)
d148bd2 
Pull Request resolved: pytorch#163131
Approved by: https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541, pytorch#163115
mansiag05 pushed a commit to mansiag05/pytorch that referenced this pull request Sep 22, 2025
@fegin @mansiag05
[CP] Fix cuDNN CP LSE dimension bug (pytorch#163231)
b49fdb0 
We should only unsqueeze if necessary.

Fix pytorch#162743

Pull Request resolved: pytorch#163231
Approved by: https://github.com/eqy
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541, pytorch#163115, pytorch#163131
cleonard530 pushed a commit to cleonard530/pytorch that referenced this pull request Sep 22, 2025
@fegin @cleonard530
[CP] Remove the need of recording cp_dim in the global var (pytorch#1…
b3f1d0b 
…62540)

This information can be obtained during the dispatching.

Pull Request resolved: pytorch#162540
Approved by: https://github.com/ezyang, https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539
cleonard530 pushed a commit to cleonard530/pytorch that referenced this pull request Sep 22, 2025
@fegin @cleonard530
[CP][BE] Correct the names of some tests (pytorch#162541)
4221f5e 
We are not doing ring attention but only using allgather to do CP for Flex.
Pull Request resolved: pytorch#162541
Approved by: https://github.com/ezyang, https://github.com/Skylion007, https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539, pytorch#162540
cleonard530 pushed a commit to cleonard530/pytorch that referenced this pull request Sep 22, 2025
@fegin @cleonard530
[CP][BE] Cosmetic refactors for CP code base (pytorch#163115)
87b1690 
Summary:
This PR is extracted from pytorch#162542, to make the original PR
easier to review. This PR only contains cosmetic changes.

Pull Request resolved: pytorch#163115
Approved by: https://github.com/tianyu-l
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541
cleonard530 pushed a commit to cleonard530/pytorch that referenced this pull request Sep 22, 2025
@fegin @cleonard530
[CP][BE] Correct an incorrect docstring (pytorch#163131)
0d0e820 
Pull Request resolved: pytorch#163131
Approved by: https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541, pytorch#163115
cleonard530 pushed a commit to cleonard530/pytorch that referenced this pull request Sep 22, 2025
@fegin @cleonard530
[CP] Fix cuDNN CP LSE dimension bug (pytorch#163231)
74f16c2 
We should only unsqueeze if necessary.

Fix pytorch#162743

Pull Request resolved: pytorch#163231
Approved by: https://github.com/eqy
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541, pytorch#163115, pytorch#163131
dsashidh pushed a commit to dsashidh/pytorch that referenced this pull request Sep 26, 2025
@fegin @dsashidh
[CP] Remove the need of recording cp_dim in the global var (pytorch#1…
bc3824a 
…62540)

This information can be obtained during the dispatching.

Pull Request resolved: pytorch#162540
Approved by: https://github.com/ezyang, https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539
dsashidh pushed a commit to dsashidh/pytorch that referenced this pull request Sep 26, 2025
@fegin @dsashidh
[CP][BE] Correct the names of some tests (pytorch#162541)
035519f 
We are not doing ring attention but only using allgather to do CP for Flex.
Pull Request resolved: pytorch#162541
Approved by: https://github.com/ezyang, https://github.com/Skylion007, https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539, pytorch#162540
dsashidh pushed a commit to dsashidh/pytorch that referenced this pull request Sep 26, 2025
@fegin @dsashidh
[CP][BE] Cosmetic refactors for CP code base (pytorch#163115)
2c1c783 
Summary:
This PR is extracted from pytorch#162542, to make the original PR
easier to review. This PR only contains cosmetic changes.

Pull Request resolved: pytorch#163115
Approved by: https://github.com/tianyu-l
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541
dsashidh pushed a commit to dsashidh/pytorch that referenced this pull request Sep 26, 2025
@fegin @dsashidh
[CP][BE] Correct an incorrect docstring (pytorch#163131)
cc3feee 
Pull Request resolved: pytorch#163131
Approved by: https://github.com/tianyu-l, https://github.com/XilunWu
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541, pytorch#163115
dsashidh pushed a commit to dsashidh/pytorch that referenced this pull request Sep 26, 2025
@fegin @dsashidh
[CP] Fix cuDNN CP LSE dimension bug (pytorch#163231)
c49d1a8 
We should only unsqueeze if necessary.

Fix pytorch#162743

Pull Request resolved: pytorch#163231
Approved by: https://github.com/eqy
ghstack dependencies: pytorch#162539, pytorch#162540, pytorch#162541, pytorch#163115, pytorch#163131
@github-actions github-actions bot deleted the gh/fegin/316/head branch October 19, 2025 02:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ezyang ezyang ezyang approved these changes

@XilunWu XilunWu XilunWu approved these changes

@tianyu-l tianyu-l tianyu-l approved these changes

Assignees

No one assigned

Labels

ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request Merged oncall: distributed Add this issue/PR to distributed oncall triage queue topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@fegin @pytorchmergebot @ezyang @XilunWu @tianyu-l