-
Notifications
You must be signed in to change notification settings - Fork 4.7k
Low-precision master params/grads/optimizer states #7700
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Masahiro Tanaka <[email protected]>
Signed-off-by: Masahiro Tanaka <[email protected]>
Signed-off-by: Masahiro Tanaka <[email protected]>
Signed-off-by: Masahiro Tanaka <[email protected]>
Signed-off-by: Masahiro Tanaka <[email protected]>
Signed-off-by: Masahiro Tanaka <[email protected]>
sfc-gh-truwase
approved these changes
Dec 2, 2025
Signed-off-by: Masahiro Tanaka <[email protected]>
Signed-off-by: Masahiro Tanaka <[email protected]>
Signed-off-by: Masahiro Tanaka <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DeepSpeed optimizer always creates fp32 master params/gradients/optimizer states.
However, we sometimes want to keep them lower precision given torch.autocast support.
This PR allows lower precision master params/grads/optimizer states with bf16/fp16 enabled.
DeepSpeed currently accepts
fp16_master_weights_and_gradientsoption underfp16section (not documented) with ZeRO1/2. This PR extends this for bf16 and also ZeRO3.In
bf16section, we can have new itemsbf16_master_weights_and_gradsandbf16_optimizer_states.Similary to
fp16_master_weights_and_grads,bf16_master_weights_and_gradskeeps master parameters in bf16.bf16_optimizer_stateskeeps optimizer states also in bf16. Here is an example configuration:Note that
bf16_master_weights_and_grads==Trueandbf16_optimizer_states==Falseis supported only with cpu offloading. Also, we don't havefp16_optimizer_statesas it won't be practical. More details are described inconfig-json.mdPreviously,
torch.autocastsupport (torch_autocastsection in config) was not compatible withbf16fp16enabled, but we now accept the combination.This PR also adds some test cases for the configurations as well as the combination with
torch.autocast.