[DCP][OSS] Rank local checkpointing in DCP without collectives #147758
+218
−35
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/147758
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 147fb0e with merge base 8eee08d ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
pytorch-bot
bot
added
module: distributed_checkpoint
oncall: distributed
saumishr
force-pushed
the
export-D70112642
branch
from
3521908 to
dbc79b6
Compare
meetv18
added
the
oncall: distributed checkpointing
janeyx99
added
release notes: distributed (checkpoint)
and removed
module: distributed_checkpoint
labels
gkroiz
added a commit
to gkroiz/pytorch
that referenced
this pull request
Mar 9, 2025
saumishr
force-pushed
the
export-D70112642
branch
from
dbc79b6 to
1f25f09
Compare
saumishr
force-pushed
the
export-D70112642
branch
2 times, most recently
from
36ee9c7 to
eb8d1a7
Compare
saumishr
force-pushed
the
export-D70112642
branch
from
eb8d1a7 to
1329b96
Compare
saumishr
force-pushed
the
export-D70112642
branch
from
1329b96 to
4d5cde1
Compare
saumishr
force-pushed
the
export-D70112642
branch
from
4d5cde1 to
54c8484
Compare
saumishr
force-pushed
the
export-D70112642
branch
2 times, most recently
from
78dcad0 to
d3c31ff
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D70112642 |
saumishr
force-pushed
the
export-D70112642
branch
from
d3c31ff to
f62e067
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D70112642 |
saumishr
added a commit
to saumishr/tnt
that referenced
this pull request
Aug 13, 2025
Summary: X-link: pytorch/pytorch#147758 Context: DCP metadata collectives become prohibitively expensive as the job scale grows. This PR introduces rank-local checkpointing which basically saves and loads the checkpoint without any collective. The trade off for now is the dedupe and re-sharding. Support for these would be introduced soon. Reviewed By: meetv18 Differential Revision: D70112642
saumishr
force-pushed
the
export-D70112642
branch
from
f13c835 to
83e249c
Compare
saumishr
added a commit
to saumishr/pytorch
that referenced
this pull request
Aug 13, 2025
…ch#147758) Summary: X-link: meta-pytorch/tnt#991 Context: DCP metadata collectives become prohibitively expensive as the job scale grows. This PR introduces rank-local checkpointing which basically saves and loads the checkpoint without any collective. The trade off for now is the dedupe and re-sharding. Support for these would be introduced soon. Test Plan: E2E UTs Save and load test with internal DCP components: https://www.internalfb.com/mlhub/pipelines/runs/mast/torchx-textray-pretrain_mlm-lv5d7qcfmnqzkd Save and load test with OSS DCP components: https://www.internalfb.com/mlhub/pipelines/runs/mast/torchx-textray-pretrain_mlm-z1vz46vkkgtcld https://www.internalfb.com/mlhub/pipelines/runs/mast/torchx-textray-pretrain_mlm-njvvbn07rv5ckd Reviewed By: meetv18 Differential Revision: D70112642
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D70112642 |
saumishr
added a commit
to saumishr/tnt
that referenced
this pull request
Aug 13, 2025
Summary: X-link: pytorch/pytorch#147758 Context: DCP metadata collectives become prohibitively expensive as the job scale grows. This PR introduces rank-local checkpointing which basically saves and loads the checkpoint without any collective. The trade off for now is the dedupe and re-sharding. Support for these would be introduced soon. Reviewed By: meetv18 Differential Revision: D70112642
saumishr
added a commit
to saumishr/pytorch
that referenced
this pull request
Aug 13, 2025
…ch#147758) Summary: X-link: meta-pytorch/tnt#991 Context: DCP metadata collectives become prohibitively expensive as the job scale grows. This PR introduces rank-local checkpointing which basically saves and loads the checkpoint without any collective. The trade off for now is the dedupe and re-sharding. Support for these would be introduced soon. Test Plan: E2E UTs Save and load test with internal DCP components: https://www.internalfb.com/mlhub/pipelines/runs/mast/torchx-textray-pretrain_mlm-lv5d7qcfmnqzkd Save and load test with OSS DCP components: https://www.internalfb.com/mlhub/pipelines/runs/mast/torchx-textray-pretrain_mlm-z1vz46vkkgtcld https://www.internalfb.com/mlhub/pipelines/runs/mast/torchx-textray-pretrain_mlm-njvvbn07rv5ckd Reviewed By: meetv18 Differential Revision: D70112642
saumishr
force-pushed
the
export-D70112642
branch
from
83e249c to
f6b7c80
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D70112642 |
saumishr
added a commit
to saumishr/tnt
that referenced
this pull request
Aug 13, 2025
Summary: X-link: pytorch/pytorch#147758 Context: DCP metadata collectives become prohibitively expensive as the job scale grows. This PR introduces rank-local checkpointing which basically saves and loads the checkpoint without any collective. The trade off for now is the dedupe and re-sharding. Support for these would be introduced soon. Reviewed By: meetv18 Differential Revision: D70112642
…ch#147758) Summary: X-link: meta-pytorch/tnt#991 Context: DCP metadata collectives become prohibitively expensive as the job scale grows. This PR introduces rank-local checkpointing which basically saves and loads the checkpoint without any collective. The trade off for now is the dedupe and re-sharding. Support for these would be introduced soon. Test Plan: E2E UTs Save and load test with internal DCP components: https://www.internalfb.com/mlhub/pipelines/runs/mast/torchx-textray-pretrain_mlm-lv5d7qcfmnqzkd Save and load test with OSS DCP components: https://www.internalfb.com/mlhub/pipelines/runs/mast/torchx-textray-pretrain_mlm-z1vz46vkkgtcld https://www.internalfb.com/mlhub/pipelines/runs/mast/torchx-textray-pretrain_mlm-njvvbn07rv5ckd Reviewed By: meetv18 Differential Revision: D70112642
saumishr
force-pushed
the
export-D70112642
branch
from
f6b7c80 to
147fb0e
Compare
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D70112642 |
saumishr
added a commit
to saumishr/tnt
that referenced
this pull request
Aug 13, 2025
Summary: X-link: pytorch/pytorch#147758 Context: DCP metadata collectives become prohibitively expensive as the job scale grows. This PR introduces rank-local checkpointing which basically saves and loads the checkpoint without any collective. The trade off for now is the dedupe and re-sharding. Support for these would be introduced soon. Reviewed By: meetv18 Differential Revision: D70112642
Contributor
Author
|
@pytorchmergebot merge |
Collaborator
Merge failedReason: This PR has internal changes and must be landed via Phabricator! Please try reimporting/rexporting the PR! Details for Dev Infra teamRaised by workflow job |
facebook-github-bot
pushed a commit
to meta-pytorch/tnt
that referenced
this pull request
Aug 13, 2025
Summary: Pull Request resolved: #991 X-link: pytorch/pytorch#147758 Context: DCP metadata collectives become prohibitively expensive as the job scale grows. This PR introduces rank-local checkpointing which basically saves and loads the checkpoint without any collective. The trade off for now is the dedupe and re-sharding. Support for these would be introduced soon. Reviewed By: meetv18 Differential Revision: D70112642 fbshipit-source-id: 5558a1d2440e539f87a9b7b6295b4199fb4b448a
Contributor
|
@pytorchbot merge (Initiating merge automatically since Phabricator Diff has merged) |
Collaborator
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
chuanhaozhuge
pushed a commit
that referenced
this pull request
Aug 14, 2025
Summary: DCP metadata collectives become prohibitively expensive as the job scale grows. This PR introduces rank-local checkpointing which basically saves and loads the checkpoint without any collective. The trade off for now is the dedupe and re-sharding. Support for these would be introduced soon. Differential Revision: D70112642 Pull Request resolved: #147758 Approved by: https://github.com/meetv18
chuanhaozhuge
pushed a commit
that referenced
this pull request
Aug 18, 2025
Summary: DCP metadata collectives become prohibitively expensive as the job scale grows. This PR introduces rank-local checkpointing which basically saves and loads the checkpoint without any collective. The trade off for now is the dedupe and re-sharding. Support for these would be introduced soon. Differential Revision: D70112642 Pull Request resolved: #147758 Approved by: https://github.com/meetv18
can-gaa-hou
pushed a commit
to can-gaa-hou/pytorch
that referenced
this pull request
Aug 22, 2025
…ch#147758) Summary: DCP metadata collectives become prohibitively expensive as the job scale grows. This PR introduces rank-local checkpointing which basically saves and loads the checkpoint without any collective. The trade off for now is the dedupe and re-sharding. Support for these would be introduced soon. Differential Revision: D70112642 Pull Request resolved: pytorch#147758 Approved by: https://github.com/meetv18
markc-614
pushed a commit
to markc-614/pytorch
that referenced
this pull request
Sep 17, 2025
…ch#147758) Summary: DCP metadata collectives become prohibitively expensive as the job scale grows. This PR introduces rank-local checkpointing which basically saves and loads the checkpoint without any collective. The trade off for now is the dedupe and re-sharding. Support for these would be introduced soon. Differential Revision: D70112642 Pull Request resolved: pytorch#147758 Approved by: https://github.com/meetv18
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
DCP metadata collectives become prohibitively expensive as the job scale grows. This PR introduces rank-local checkpointing which basically saves and loads the checkpoint without any collective. The trade off for now is the dedupe and re-sharding. Support for these would be introduced soon.
Differential Revision: D70112642
cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @LucasLLC @pradeepfn @kwen2501 @c-p-i-o @MeetVadakkanchery @mhorowitz @ekr0