Enable torch.autocast with ZeRO #6993

tohtana · 2025-02-03T07:19:20Z

DeepSpeed supports mixed precision training, but the behavior is different from torch.autocast. DeepSpeed maintains parameters and gradients both in FP32 and a lower precision (FP16/BF16) (NVIDIA Apex AMP style) and computes all modules in the lower precision while torch.autocast maintains parameters in FP32 but computes only certain operators in the lower precision.
This leads to differences in:

performance: torch.autocast needs downcast in forward/backward
memory usage: DeepSpeed needs more memory to keep copies of parameters and gradients in lower precision
accuracy: torch.autocast has a list of modules that can safely be computed in lower precision. Some precision-sensitive operators (e.g. softmax) are computed in FP32.

To align DeepSpeed's behavior with torch.autocast when necessary, this PR adds the integration with torch.autocast with ZeRO. Here is an examples of the configuration.

"torch_autocast": {
  "enabled": true,
  "dtype": "bfloat16",
  "lower_precision_safe_modules": ["torch.nn.Linear", "torch.nn.Conv2d"]
}

Each configuration works as follows:

enabled: Enable the integration with torch.autocast if this is set to True. You don't need to call torch.autocast in your code. The grad scaler is also applied in the DeepSpeed optimizer.
dtype: lower precision dtype passed to torch.autocast. Gradients for allreduce (reduce-scatter) and parameters for allgather (only for ZeRO3) of lower_precision_safe_modules are also downcasted to this dtype.
lower_precision_safe_modules: Downcast for allreduce (reduce-scatter) and allgather (ZeRO3) are applied only to modules specified in this list. (The precision for PyTorch operators in forward/backward follows torch.autocast's policy, not this list.) You can set names of classes with their packages. If you don't set this item, DeepSpeed uses the default list: [torch.nn.Linear, torch.nn.Conv1d, torch.nn.Conv2d, torch.nn.Conv3d].

Note that we only maintain FP32 parameters with this feature enabled. For consistency, you cannot enable fp16 or bf16 in DeepSpeed config.

Fix #6772 --------- Co-authored-by: Logan Adams <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Signed-off-by: Masahiro Tanaka <[email protected]>

…#6967) - Issues with nv-sd updates, will follow up with a subsequent PR Signed-off-by: Masahiro Tanaka <[email protected]>

Signed-off-by: Masahiro Tanaka <[email protected]>

NVIDIA Blackwell GPU generation has number 10. The SM code and architecture should be `100`, but the current code generates `1.`, because it expects a 2 characters string. This change modifies the logic to consider it as a string that contains a `.`, hence splits the string and uses the array of strings. Signed-off-by: Fabien Dupont <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Signed-off-by: Olatunji Ruwase <[email protected]> Signed-off-by: Logan Adams <[email protected]> Signed-off-by: Fabien Dupont <[email protected]> Co-authored-by: Fabien Dupont <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Signed-off-by: Masahiro Tanaka <[email protected]>

1. update intel oneAPI basekit to 2025.0 2. update torch/ipex/oneccl to 2.5 Signed-off-by: Masahiro Tanaka <[email protected]>

Same as [this PR](#6922). [affeb88](affeb88) I noticed the CI updated the DCO check recently. Using the suggested rebase method for sign-off would reintroduce many conflicts, so I opted for a squash merge with sign-off instead. thanks: ) Signed-off-by: inkcherry <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Signed-off-by: Masahiro Tanaka <[email protected]>

Those files have code that gets run when importing them, so in systems that doesn't support triton but have triton installed this causes issues. In general, I think it is better to import triton when it is installed and supported. Signed-off-by: Omar Elayan <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Signed-off-by: Logan Adams <[email protected]> Co-authored-by: Stas Bekman <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Fix #7014 Avoid naming collision on `partition()` --------- Signed-off-by: Olatunji Ruwase <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Fix typos Signed-off-by: Masahiro Tanaka <[email protected]>

BUGFIX for Apple Silicon hostname #6497 --------- Signed-off-by: Fabien Dupont <[email protected]> Signed-off-by: Olatunji Ruwase <[email protected]> Signed-off-by: Logan Adams <[email protected]> Signed-off-by: inkcherry <[email protected]> Signed-off-by: Roman Fitzjalen <[email protected]> Co-authored-by: Logan Adams <[email protected]> Co-authored-by: Fabien Dupont <[email protected]> Co-authored-by: Olatunji Ruwase <[email protected]> Co-authored-by: Liangliang Ma <[email protected]> Co-authored-by: inkcherry <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

- Update existing workflows that use cu121 to cu124. Note, this means that where we download torch latest, we will now be getting torch 2.6 rather than the torch latest 2.5 provided with cuda 12.1. - Note, nv-nightly is failing in master currently due to unrelated errors, so this could be ignored in this PR (nv-nightly tested locally, where it passes with 12.1 and it also passes with 12.4). --------- Signed-off-by: Fabien Dupont <[email protected]> Signed-off-by: Logan Adams <[email protected]> Signed-off-by: Olatunji Ruwase <[email protected]> Signed-off-by: inkcherry <[email protected]> Signed-off-by: Omar Elayan <[email protected]> Co-authored-by: Fabien Dupont <[email protected]> Co-authored-by: Olatunji Ruwase <[email protected]> Co-authored-by: Liangliang Ma <[email protected]> Co-authored-by: inkcherry <[email protected]> Co-authored-by: Omar Elayan <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

This change is required to successfully build fp_quantizer extension on ROCm. --------- Co-authored-by: Logan Adams <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

@tjruwase

cc @tjruwase @jomayeri --------- Co-authored-by: root <root@ftqtmec25000000.taxzvufipdhelhupulxcbvr15f.ux.internal.cloudapp.net> Signed-off-by: Masahiro Tanaka <[email protected]>

Fix #7029 - Add Chinese blog for deepspeed windows - Fix format in README.md Co-authored-by: Logan Adams <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Adding compile support for AIO library on AMD GPUs. --------- Co-authored-by: Olatunji Ruwase <[email protected]> Co-authored-by: Logan Adams <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Make trace cache warnings configurable, and disabled by default. Fix #6985, #4081, #5033, #5006, #5662 --------- Signed-off-by: Olatunji Ruwase <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Update CUDA compute capability for cross compile according to wiki page. https://en.wikipedia.org/wiki/CUDA#GPUs_supported --------- Signed-off-by: Hongwei <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Signed-off-by: Masahiro Tanaka <[email protected]>

Signed-off-by: Logan Adams <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Propagate API change. Signed-off-by: Olatunji Ruwase <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

tohtana · 2025-05-24T03:27:09Z

Then assert is the way to go, Masahiro

Thank you @stas00, then can you approve this PR?

stas00 · 2025-05-27T05:12:54Z

Hmm, I can't just hit approve, that would be defeat the purpose of doing the review.

We have only discussed one small aspect of this PR, which has been resolved, but the rest of the PR I don't know and currently rushing to finish the porting of Ulysses to Hf/DS so until that is done I won't have time to do a serious review.

Signed-off-by: Masahiro Tanaka <[email protected]>

…pSpeed into tohtana/support_autocast

#6993 broke many paths in ZeRO1/2 optimizer. This PR fixes most of the issues the PR caused. Currently we still have one error with tests in `unit/runtime/zero`. ``` ====================================== short test summary info ====================================== FAILED test_zero.py::TestParamPartitioningSkipInit::test[dtype1] - RuntimeError: mat1 and mat2 must have the same dtype, but got Half and BFloat16 ========= 1 failed, 204 passed, 66 skipped, 15 deselected, 5 warnings in 2305.03s (0:38:25) ========= ``` --------- Signed-off-by: Masahiro Tanaka <[email protected]>

DeepSpeed supports mixed precision training, but the behavior is different from `torch.autocast`. DeepSpeed maintains parameters and gradients both in FP32 and a lower precision (FP16/BF16) (NVIDIA Apex AMP style) and computes all modules in the lower precision while `torch.autocast` maintains parameters in FP32 but computes only certain operators in the lower precision. This leads to differences in: - performance: `torch.autocast` needs downcast in forward/backward - memory usage: DeepSpeed needs more memory to keep copies of parameters and gradients in lower precision - accuracy: `torch.autocast` has a list of modules that can safely be computed in lower precision. Some precision-sensitive operators (e.g. softmax) are computed in FP32. To align DeepSpeed's behavior with `torch.autocast` when necessary, this PR adds the integration with `torch.autocast` with ZeRO. Here is an examples of the configuration. ```json "torch_autocast": { "enabled": true, "dtype": "bfloat16", "lower_precision_safe_modules": ["torch.nn.Linear", "torch.nn.Conv2d"] } ``` Each configuration works as follows: - `enabled`: Enable the integration with `torch.autocast` if this is set to `True`. You don't need to call `torch.autocast` in your code. The grad scaler is also applied in the DeepSpeed optimizer. - `dtype`: lower precision dtype passed to `torch.autocast`. Gradients for allreduce (reduce-scatter) and parameters for allgather (only for ZeRO3) of `lower_precision_safe_modules` are also downcasted to this dtype. - `lower_precision_safe_modules`: Downcast for allreduce (reduce-scatter) and allgather (ZeRO3) are applied only to modules specified in this list. (The precision for PyTorch operators in forward/backward follows `torch.autocast`'s policy, not this list.) You can set names of classes with their packages. If you don't set this item, DeepSpeed uses the default list: `[torch.nn.Linear, torch.nn.Conv1d, torch.nn.Conv2d, torch.nn.Conv3d]`. Note that we only maintain FP32 parameters with this feature enabled. For consistency, you cannot enable `fp16` or `bf16` in DeepSpeed config. --------- Signed-off-by: Masahiro Tanaka <[email protected]> Signed-off-by: Fabien Dupont <[email protected]> Signed-off-by: Olatunji Ruwase <[email protected]> Signed-off-by: Logan Adams <[email protected]> Signed-off-by: inkcherry <[email protected]> Signed-off-by: Omar Elayan <[email protected]> Signed-off-by: Roman Fitzjalen <[email protected]> Signed-off-by: Hongwei <[email protected]> Signed-off-by: shaomin <[email protected]> Signed-off-by: Stas Bekman <[email protected]> Signed-off-by: siqi <[email protected]> Signed-off-by: Wei Wu <[email protected]> Signed-off-by: ShellyNR <[email protected]> Signed-off-by: Lai, Yejing <[email protected]> Co-authored-by: Olatunji Ruwase <[email protected]> Co-authored-by: Logan Adams <[email protected]> Co-authored-by: Fabien Dupont <[email protected]> Co-authored-by: Liangliang Ma <[email protected]> Co-authored-by: inkcherry <[email protected]> Co-authored-by: Omar Elayan <[email protected]> Co-authored-by: Stas Bekman <[email protected]> Co-authored-by: Roman Fitzjalen <[email protected]> Co-authored-by: Ramya Ramineni <[email protected]> Co-authored-by: Guanhua Wang <[email protected]> Co-authored-by: root <root@ftqtmec25000000.taxzvufipdhelhupulxcbvr15f.ux.internal.cloudapp.net> Co-authored-by: Hongwei Chen <[email protected]> Co-authored-by: Joe Mayer <[email protected]> Co-authored-by: wukong1992 <[email protected]> Co-authored-by: shaomin <[email protected]> Co-authored-by: loadams <[email protected]> Co-authored-by: siqi654321 <[email protected]> Co-authored-by: siqi <[email protected]> Co-authored-by: Wei Wu <[email protected]> Co-authored-by: Shelly Nahir <[email protected]> Co-authored-by: snahir <[email protected]> Co-authored-by: Yejing-Lai <[email protected]> Co-authored-by: Siddharth Singh <[email protected]> Co-authored-by: Olatunji Ruwase <[email protected]>

deepspeedai#6993 broke many paths in ZeRO1/2 optimizer. This PR fixes most of the issues the PR caused. Currently we still have one error with tests in `unit/runtime/zero`. ``` ====================================== short test summary info ====================================== FAILED test_zero.py::TestParamPartitioningSkipInit::test[dtype1] - RuntimeError: mat1 and mat2 must have the same dtype, but got Half and BFloat16 ========= 1 failed, 204 passed, 66 skipped, 15 deselected, 5 warnings in 2305.03s (0:38:25) ========= ``` --------- Signed-off-by: Masahiro Tanaka <[email protected]>

Vervious · 2025-08-08T04:03:58Z

I'm running into an issue where turning on this feature results in massive grad norms, using zero 2; have you seen this before?

[deepspeed.torch_autocast]
enabled = true  # NOTE: turning this on makes grad norms explode in stage 2 (but not stage 3)
dtype = "bfloat16"

# [deepspeed.bf16]
# enabled = true  # this works properly

[deepspeed.zero_optimization]
stage = 2
allgather_partitions = true
overlap_comm = false
reduce_scatter = true
contiguous_gradients = true
stage3_prefetch_bucket_size = 0
stage3_max_live_parameters = 0
stage3_max_reuse_distance = 0
stage3_gather_16bit_weights_on_model_save = true

Grad norms were reported via model_engine.get_global_grad_norm() and also observed via safe_get_full_grad after a backward call (but before step). Stage 3 seems to have reasonable grad norms. For some reason the loss curves also don't match exactly between stage 2 and 3 (but they match exactly using deepspeed.bf16 instead)

PR deepspeedai#6993 replaces the flat IPG buffers with a dict maintaining type-indexed buckets. The member is also renamed from `_ipg_bucket_flat_buffer` to `ipg_buckets`. Update the bucket clearing logic in `init_z3` accordingly. Signed-off-by: Junjie Mao <[email protected]>

PR #6993 replaces the flat IPG buffers with a dict maintaining type-indexed buckets. The member is also renamed from `_ipg_bucket_flat_buffer` to `ipg_buckets`. Update the bucket clearing logic in `init_z3` accordingly. Signed-off-by: Junjie Mao <[email protected]>

Original PR #6993 by tohtana Original: deepspeedai/DeepSpeed#6993

Merged from original PR #6993 Original: deepspeedai/DeepSpeed#6993

PR #6993 replaces the flat IPG buffers with a dict maintaining type-indexed buckets. The member is also renamed from `_ipg_bucket_flat_buffer` to `ipg_buckets`. Update the bucket clearing logic in `init_z3` accordingly. Signed-off-by: Junjie Mao <[email protected]> Signed-off-by: Guokai Ma <[email protected]>

DeepSpeed supports mixed precision training, but the behavior is different from `torch.autocast`. DeepSpeed maintains parameters and gradients both in FP32 and a lower precision (FP16/BF16) (NVIDIA Apex AMP style) and computes all modules in the lower precision while `torch.autocast` maintains parameters in FP32 but computes only certain operators in the lower precision. This leads to differences in: - performance: `torch.autocast` needs downcast in forward/backward - memory usage: DeepSpeed needs more memory to keep copies of parameters and gradients in lower precision - accuracy: `torch.autocast` has a list of modules that can safely be computed in lower precision. Some precision-sensitive operators (e.g. softmax) are computed in FP32. To align DeepSpeed's behavior with `torch.autocast` when necessary, this PR adds the integration with `torch.autocast` with ZeRO. Here is an examples of the configuration. ```json "torch_autocast": { "enabled": true, "dtype": "bfloat16", "lower_precision_safe_modules": ["torch.nn.Linear", "torch.nn.Conv2d"] } ``` Each configuration works as follows: - `enabled`: Enable the integration with `torch.autocast` if this is set to `True`. You don't need to call `torch.autocast` in your code. The grad scaler is also applied in the DeepSpeed optimizer. - `dtype`: lower precision dtype passed to `torch.autocast`. Gradients for allreduce (reduce-scatter) and parameters for allgather (only for ZeRO3) of `lower_precision_safe_modules` are also downcasted to this dtype. - `lower_precision_safe_modules`: Downcast for allreduce (reduce-scatter) and allgather (ZeRO3) are applied only to modules specified in this list. (The precision for PyTorch operators in forward/backward follows `torch.autocast`'s policy, not this list.) You can set names of classes with their packages. If you don't set this item, DeepSpeed uses the default list: `[torch.nn.Linear, torch.nn.Conv1d, torch.nn.Conv2d, torch.nn.Conv3d]`. Note that we only maintain FP32 parameters with this feature enabled. For consistency, you cannot enable `fp16` or `bf16` in DeepSpeed config. --------- Signed-off-by: Masahiro Tanaka <[email protected]> Signed-off-by: Fabien Dupont <[email protected]> Signed-off-by: Olatunji Ruwase <[email protected]> Signed-off-by: Logan Adams <[email protected]> Signed-off-by: inkcherry <[email protected]> Signed-off-by: Omar Elayan <[email protected]> Signed-off-by: Roman Fitzjalen <[email protected]> Signed-off-by: Hongwei <[email protected]> Signed-off-by: shaomin <[email protected]> Signed-off-by: Stas Bekman <[email protected]> Signed-off-by: siqi <[email protected]> Signed-off-by: Wei Wu <[email protected]> Signed-off-by: ShellyNR <[email protected]> Signed-off-by: Lai, Yejing <[email protected]> Co-authored-by: Olatunji Ruwase <[email protected]> Co-authored-by: Logan Adams <[email protected]> Co-authored-by: Fabien Dupont <[email protected]> Co-authored-by: Liangliang Ma <[email protected]> Co-authored-by: inkcherry <[email protected]> Co-authored-by: Omar Elayan <[email protected]> Co-authored-by: Stas Bekman <[email protected]> Co-authored-by: Roman Fitzjalen <[email protected]> Co-authored-by: Ramya Ramineni <[email protected]> Co-authored-by: Guanhua Wang <[email protected]> Co-authored-by: root <root@ftqtmec25000000.taxzvufipdhelhupulxcbvr15f.ux.internal.cloudapp.net> Co-authored-by: Hongwei Chen <[email protected]> Co-authored-by: Joe Mayer <[email protected]> Co-authored-by: wukong1992 <[email protected]> Co-authored-by: shaomin <[email protected]> Co-authored-by: loadams <[email protected]> Co-authored-by: siqi654321 <[email protected]> Co-authored-by: siqi <[email protected]> Co-authored-by: Wei Wu <[email protected]> Co-authored-by: Shelly Nahir <[email protected]> Co-authored-by: snahir <[email protected]> Co-authored-by: Yejing-Lai <[email protected]> Co-authored-by: Siddharth Singh <[email protected]> Co-authored-by: Olatunji Ruwase <[email protected]>

deepspeedai#6993 broke many paths in ZeRO1/2 optimizer. This PR fixes most of the issues the PR caused. Currently we still have one error with tests in `unit/runtime/zero`. ``` ====================================== short test summary info ====================================== FAILED test_zero.py::TestParamPartitioningSkipInit::test[dtype1] - RuntimeError: mat1 and mat2 must have the same dtype, but got Half and BFloat16 ========= 1 failed, 204 passed, 66 skipped, 15 deselected, 5 warnings in 2305.03s (0:38:25) ========= ``` --------- Signed-off-by: Masahiro Tanaka <[email protected]>

PR deepspeedai#6993 replaces the flat IPG buffers with a dict maintaining type-indexed buckets. The member is also renamed from `_ipg_bucket_flat_buffer` to `ipg_buckets`. Update the bucket clearing logic in `init_z3` accordingly. Signed-off-by: Junjie Mao <[email protected]>

tjruwase and others added 30 commits February 28, 2025 22:53

Use ds-specific module id to avoid conflicts (#6847)

a4fbc3a

Fix #6772 --------- Co-authored-by: Logan Adams <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

add autocast support and ds_config item

9d48dc6

Signed-off-by: Masahiro Tanaka <[email protected]>

prepare ipg buckets for multiple dtypes

8957009

Signed-off-by: Masahiro Tanaka <[email protected]>

switch communication data type

c390c76

Signed-off-by: Masahiro Tanaka <[email protected]>

add gradscaler

458797d

Signed-off-by: Masahiro Tanaka <[email protected]>

Update A6000 workflows to use newer docker container - 24.09 vs 24.03 (…

2984415

…#6967) - Issues with nv-sd updates, will follow up with a subsequent PR Signed-off-by: Masahiro Tanaka <[email protected]>

fix import and formatting

6b6a600

Signed-off-by: Masahiro Tanaka <[email protected]>

convert comm type for z3

2817c02

Signed-off-by: Masahiro Tanaka <[email protected]>

Update GH org references (#6998)

fcdeda3

Signed-off-by: Olatunji Ruwase <[email protected]> Signed-off-by: Logan Adams <[email protected]> Signed-off-by: Fabien Dupont <[email protected]> Co-authored-by: Fabien Dupont <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Update CNAME

b476d07

Signed-off-by: Masahiro Tanaka <[email protected]>

Update CNAME

72f9687

Signed-off-by: Masahiro Tanaka <[email protected]>

[XPU] max1100 workflow update for docker and softwares (#7003)

a2425da

1. update intel oneAPI basekit to 2025.0 2. update torch/ipex/oneccl to 2.5 Signed-off-by: Masahiro Tanaka <[email protected]>

cast dtype for allgather

1c1d43a

Signed-off-by: Masahiro Tanaka <[email protected]>

Update A6000 tests transformers version (#7016)

359c85d

Signed-off-by: Logan Adams <[email protected]> Co-authored-by: Stas Bekman <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Fix ds-chat CI regression (#7015)

735fc2c

Fix #7014 Avoid naming collision on `partition()` --------- Signed-off-by: Olatunji Ruwase <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

[Ulysses tutorial] typos (#7024)

1e7888c

Fix typos Signed-off-by: Masahiro Tanaka <[email protected]>

[ROCm] Enable fp_quantizer on ROCm (#7027)

f1aea5d

This change is required to successfully build fp_quantizer extension on ROCm. --------- Co-authored-by: Logan Adams <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

add gds chinese blog (#7034)

8152824

cc @tjruwase @jomayeri --------- Co-authored-by: root <root@ftqtmec25000000.taxzvufipdhelhupulxcbvr15f.ux.internal.cloudapp.net> Signed-off-by: Masahiro Tanaka <[email protected]>

Add chinese blog for deepspeed windows, and fix format (#7035)

c898ac5

Fix #7029 - Add Chinese blog for deepspeed windows - Fix format in README.md Co-authored-by: Logan Adams <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

AIO on ROCM (#7023)

e3ea926

Adding compile support for AIO library on AMD GPUs. --------- Co-authored-by: Olatunji Ruwase <[email protected]> Co-authored-by: Logan Adams <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Control trace cache warnings (#7039)

e946615

Make trace cache warnings configurable, and disabled by default. Fix #6985, #4081, #5033, #5006, #5662 --------- Signed-off-by: Olatunji Ruwase <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Update setup.py handling of ROCm cupy (#7051)

acc6a1e

Signed-off-by: Masahiro Tanaka <[email protected]>

nv-ds-chat breaks with latest transformers (#7052)

bda1430

Signed-off-by: Logan Adams <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Rename aio_thread_count to intra_op_parallelism (#7056)

c184b16

Propagate API change. Signed-off-by: Olatunji Ruwase <[email protected]> Signed-off-by: Masahiro Tanaka <[email protected]>

Masahiro Tanaka and others added 7 commits June 2, 2025 06:52

Merge branch 'master' into tohtana/support_autocast

a7d8737

Signed-off-by: Masahiro Tanaka <[email protected]>

Merge branch 'master' into tohtana/support_autocast

a01facd

Merge branch 'master' into tohtana/support_autocast

1a36621

Merge branch 'master' into tohtana/support_autocast

be11834

Signed-off-by: Masahiro Tanaka <[email protected]>

Merge branch 'master' into tohtana/support_autocast

81dfab6

Merge branch 'master' into tohtana/support_autocast

52ab5d2

Signed-off-by: Masahiro Tanaka <[email protected]>

Merge branch 'tohtana/support_autocast' of github.com:deepspeedai/Dee…

e1b8474

…pSpeed into tohtana/support_autocast

tjruwase approved these changes Jun 19, 2025

View reviewed changes

tjruwase and others added 2 commits June 18, 2025 22:49

Merge branch 'master' into tohtana/support_autocast

3914509

Merge branch 'master' into tohtana/support_autocast

e9154d4

tohtana enabled auto-merge (squash) June 19, 2025 20:23

tohtana merged commit ed5f737 into master Jun 19, 2025
12 checks passed

tohtana deleted the tohtana/support_autocast branch June 19, 2025 21:36

tohtana mentioned this pull request Jun 21, 2025

Fix release of IPG buffer #7376

Merged

eternalNight mentioned this pull request Sep 30, 2025

DeepCompile: Fix IPG bucket clearing #7610

Merged

snorkelopstesting1-a11y pushed a commit to snorkel-marlin-repos/deepspeedai_DeepSpeed_pr_6993_98d95b49-81a8-4e37-a589-1378c216295f that referenced this pull request Oct 2, 2025

Enable torch.autocast with ZeRO

6e1c01e

Original PR #6993 by tohtana Original: deepspeedai/DeepSpeed#6993

snorkelopstesting1-a11y mentioned this pull request Oct 2, 2025

Enable torch.autocast with ZeRO snorkel-marlin-repos/deepspeedai_DeepSpeed_pr_6993_98d95b49-81a8-4e37-a589-1378c216295f#1

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enable torch.autocast with ZeRO #6993

Enable torch.autocast with ZeRO #6993

Uh oh!

tohtana commented Feb 3, 2025 •

edited

Loading

Uh oh!

tohtana commented May 24, 2025

Uh oh!

stas00 commented May 27, 2025 •

edited

Loading

Uh oh!

Uh oh!

Vervious commented Aug 8, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Enable torch.autocast with ZeRO #6993

Enable torch.autocast with ZeRO #6993

Uh oh!

Conversation

tohtana commented Feb 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tohtana commented May 24, 2025

Uh oh!

stas00 commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Vervious commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

tohtana commented Feb 3, 2025 •

edited

Loading

stas00 commented May 27, 2025 •

edited

Loading

Vervious commented Aug 8, 2025 •

edited

Loading