[Pipelining] Clean up hooks in zero bubble #138720
Conversation
[ghstack-poisoned]
pytorch-bot
bot
added
the
oncall: distributed
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/138720
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 45e895b with merge base dbd6ada ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
H-Huang
added
release notes: distributed (pipeline)
| retain_graph=retain_graph, | ||
| ) | ||
| # release grad memory early after use | ||
| del param_group["grads"] |
There was a problem hiding this comment.
can you say more about this? Does this meaningfully affect memory usage? What happened if we did not do this delete?
There was a problem hiding this comment.
It's not very significant and saves a few MB. We hold the grad tensors and this would get deleted at the end of the step anyways so this just releases it a bit earlier
There was a problem hiding this comment.
ok, 'saves a few MB' means it is actually meaningful, even if its small.
cc awgu kwen2501 wanchaol fegin fduwjj wz337 wconstab d4l3k c-p-i-o [ghstack-poisoned]
cc awgu kwen2501 wanchaol fegin fduwjj wz337 wconstab d4l3k c-p-i-o [ghstack-poisoned]
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
|
The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command |
|
@pytorchmergebot merge -f "Macos pending" |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Stack from ghstack (oldest at bottom):
cc @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o