Emphasize all DDP forward() outputs must participate in computing loss #20586
Conversation
function outputs must participate in calculating loss.
pytorchbot
added
oncall: distributed
mrshenli
added
the
triaged
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@mrshenli has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
torch/nn/parallel/distributed.py
Outdated
| as being ready to be reduced. Note that all | ||
| ``forward`` outputs must participate in | ||
| calculating loss. Otherwise, those unused | ||
| parameters will not be detected. |
There was a problem hiding this comment.
Note that all
forwardoutputs that are derived from module parameters must participate in calculating loss and later the gradient computation. If they don't, this wrapper will hang waiting for autograd to produce gradients for those parameters. Any outputs derived from module parameters that are otherwise unused can be detached from the autograd graph usingtorch.Tensor.detach.
pietern
left a comment
pietern
left a comment
There was a problem hiding this comment.
LGTM -- added a bit more information to the part that ends up in the documentation.
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@mrshenli has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
5 participants
CC @borguz @chenyangyu1988