Skip to content

Fix DDP bug in single process multiple device use cases #36503

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mrshenli wants to merge 10 commits into from
Closed

Fix DDP bug in single process multiple device use cases #36503

mrshenli wants to merge 10 commits into from

Conversation

@mrshenli
Copy link
Contributor

@mrshenli mrshenli commented Apr 13, 2020 •
edited
Loading

Stack from ghstack:

Differential Revision: D21179274

mrshenli pushed a commit that referenced this pull request Apr 13, 2020
Fix DDP bug in single process multiple device use cases
b2f574d 
ghstack-source-id: 727be4e
Pull Request resolved: #36503
@dr-ci
Copy link

dr-ci bot commented Apr 13, 2020 •
edited
Loading

💊 Build failures summary and remediations

As of commit d8299d1 (more details on the Dr. CI page):

  • 2/2 failures possibly* introduced in this PR
    • 1/2 non-CircleCI failure(s)

XLA failure

Job pytorch_xla_linux_bionic_py3_6_clang9_build is failing. Please create an issue with title prefixed by [PT_BREAK] in pytorch/xla and link to to this PR. If you have questions, please reach out to @ailzhang / @dlibenzi / @JackCaoG.

Extra GitHub checks: 1 failed

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

See how this bot performed.

This comment has been revised 31 times.

@mrshenli mrshenli changed the base branch from gh/mrshenli/141/base to release/1.5 April 13, 2020 20:21
@mrshenli mrshenli changed the base branch from release/1.5 to master April 13, 2020 20:21
zhaojuanmao
zhaojuanmao approved these changes Apr 13, 2020
View reviewed changes
zhaojuanmao
zhaojuanmao reviewed Apr 13, 2020
View reviewed changes
Copy link
Contributor

@zhaojuanmao zhaojuanmao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, accidentally clicked approve

zhaojuanmao
zhaojuanmao reviewed Apr 13, 2020
View reviewed changes
@mrshenli mrshenli mentioned this pull request Apr 22, 2020
Closed
mrshenli pushed a commit that referenced this pull request Apr 22, 2020
Fix DDP bug in single process multiple device use cases
cb6b8f5 
ghstack-source-id: 6a1b761
Pull Request resolved: #36503
mrshenli pushed a commit that referenced this pull request Apr 22, 2020
Fix DDP bug in single process multiple device use cases
d6ea229 
ghstack-source-id: 9bd7fa8
Pull Request resolved: #36503
@mrshenli mrshenli requested a review from zhaojuanmao April 22, 2020 16:16
mrshenli pushed a commit that referenced this pull request Apr 22, 2020
Fix DDP bug in single process multiple device use cases
8a1033c 
ghstack-source-id: 3091f2b
Pull Request resolved: #36503
@mrshenli mrshenli changed the title Fix DDP bug in single process multiple device use cases [WIP] Fix DDP bug in single process multiple device use cases Apr 22, 2020
@mrshenli mrshenli mentioned this pull request Apr 22, 2020
Closed
mrshenli pushed a commit that referenced this pull request Apr 22, 2020
Fix DDP bug in single process multiple device use cases
7a03ce4 
ghstack-source-id: 73d1c02
Pull Request resolved: #36503
zhaojuanmao
zhaojuanmao approved these changes Apr 22, 2020
View reviewed changes
Copy link
Contributor

@zhaojuanmao zhaojuanmao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good, I will think about how to refactor the test structures as well.

thanks for fixing it.

@mrshenli mrshenli changed the title [WIP] Fix DDP bug in single process multiple device use cases Fix DDP bug in single process multiple device use cases Apr 22, 2020
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Apr 23, 2020

@mrshenli merged this pull request in 8d6a8d2.

@facebook-github-bot facebook-github-bot deleted the gh/mrshenli/141/head branch April 26, 2020 14:18
@mrshenli mrshenli mentioned this pull request Apr 28, 2020
Closed
@gchanan gchanan mentioned this pull request May 27, 2020
Closed
@mrshenli mrshenli mentioned this pull request May 27, 2020
Merged
mrshenli pushed a commit to mrshenli/pytorch that referenced this pull request May 27, 2020
@mrshenli
Fix DDP bug in single process multiple device use cases (pytorch#36503)
3d55ab3 
This is a commit to merge pytorch#36503 into the 1.5.1 release. It fixes
single-process multi-GPU DDP use cases by explicitly exposing
model replica's parameters to DDP. pytorch#36656 is landed into master at 8d6a8d2.

Summary: Pull Request resolved: pytorch#36503

Test Plan: Imported from OSS

Differential Revision: D21179274

Pulled By: mrshenli

fbshipit-source-id: 0afce30ae0ddda753d1e240584a0f80df9aec4c2
gchanan pushed a commit that referenced this pull request Jun 1, 2020
@gchanan
Fix DDP bug in single process multiple device use cases (#36503)
6151405 
This is a commit to merge #36503 into the 1.5.1 release. It fixes
single-process multi-GPU DDP use cases by explicitly exposing
model replica's parameters to DDP. #36656 is landed into master at 8d6a8d2.

Summary: Pull Request resolved: #36503

Test Plan: Imported from OSS

Differential Revision: D21179274

Pulled By: mrshenli

fbshipit-source-id: 0afce30ae0ddda753d1e240584a0f80df9aec4c2
@mrshenli mrshenli mentioned this pull request Jun 17, 2020
Closed
mrshenli added a commit that referenced this pull request Jun 17, 2020
@mrshenli
Remove obsolete warning message from DDP
37e127a 
Fixed by #36503

[ghstack-poisoned]
mrshenli added a commit that referenced this pull request Jun 17, 2020
@mrshenli
Remove obsolete warning message from DDP
96d9218 
Fixed by #36503

ghstack-source-id: 99d0db5
Pull Request resolved: #40190
facebook-github-bot pushed a commit that referenced this pull request Jun 18, 2020
@mrshenli @facebook-github-bot
Remove obsolete warning message from DDP (#40190)
30364f0 
Summary:
Pull Request resolved: #40190

Fixed by #36503

Test Plan: Imported from OSS

Differential Revision: D22101516

Pulled By: mrshenli

fbshipit-source-id: 9abd6dce602530c11b7fe623ac0f4d556dccc961
@Lornatang Lornatang mentioned this pull request Jun 20, 2020
Closed
xwang233 pushed a commit to xwang233/pytorch that referenced this pull request Jun 20, 2020
@mrshenli @xwang233
Remove obsolete warning message from DDP (pytorch#40190)
a2a218d 
Summary:
Pull Request resolved: pytorch#40190

Fixed by pytorch#36503

Test Plan: Imported from OSS

Differential Revision: D22101516

Pulled By: mrshenli

fbshipit-source-id: 9abd6dce602530c11b7fe623ac0f4d556dccc961
jeffdaily added a commit to ROCm/pytorch that referenced this pull request Jun 25, 2020
@jeffdaily
skip_if_rocm test_rnn in test_c10d_spawn.py
7b43f18 
Test was added a few months back in pytorch#36503 but recently became flaky for ROCm.
@jeffdaily jeffdaily mentioned this pull request Jun 25, 2020
Closed
facebook-github-bot pushed a commit that referenced this pull request Jun 26, 2020
@jeffdaily @facebook-github-bot
skip_if_rocm test_rnn in test_c10d_spawn.py (#40577)
fc8bca0 
Summary:
Test was added a few months back in #36503 but recently became flaky for ROCm.

CC ezyang xw285cornell sunway513
Pull Request resolved: #40577

Differential Revision: D22258196

Pulled By: ezyang

fbshipit-source-id: 8a22b0c17b536b3d42d0382f7737df0f8823ba08
@vfdev-5 vfdev-5 mentioned this pull request Jun 29, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhaojuanmao zhaojuanmao zhaojuanmao approved these changes

@apaszke apaszke Awaiting requested review from apaszke

@pritamdamania87 pritamdamania87 Awaiting requested review from pritamdamania87

Assignees

No one assigned

Labels

Merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@mrshenli @facebook-github-bot @zhaojuanmao @mruberry