Fix DDP incompatibility issue with nn.MultiheadAttention. #26826
Conversation
zhangguanheng66
requested review from
apaszke,
mrshenli and
pietern
as code owners
pytorchbot
added
oncall: distributed
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@zhangguanheng66 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@zhangguanheng66 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@zhangguanheng66 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
pietern
left a comment
pietern
left a comment
There was a problem hiding this comment.
Marking this as changes needed, because the test in test_c10d should be removed.
zhangguanheng66
force-pushed
the
MTH_DDP
branch
from
71569e4 to
10fce78
Compare
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@zhangguanheng66 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@zhangguanheng66 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
cpuhrsch
left a comment
cpuhrsch
left a comment
There was a problem hiding this comment.
Can this break backwards compatibility in any sense?
I think it should be fine for BC. In the new version, we only create a In an old trained model, there is always |
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@zhangguanheng66 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@zhangguanheng66 merged this pull request in eb93200. |
) Summary: Fix issue pytorch#26698. With different query/keys/value dimensions, `nn.MultiheadAttention` has DDP incompatibility issue because in that case `in_proj_weight` attribute is created but not used. Fix it and add a distributed unit test. Pull Request resolved: pytorch#26826 Differential Revision: D17583807 Pulled By: zhangguanheng66 fbshipit-source-id: c393584c331ed4f57ebaf2d4015ef04589c973f6
7 participants
Fix issue #26698.
With different query/keys/value dimensions,
nn.MultiheadAttentionhas DDP incompatibility issue because in that casein_proj_weightattribute is created but not used. Fix it and add a distributed unit test.