Skip to content

enable FP16Optimizer for fp16 deepspeed training.#547

Merged
JingyaHuang merged 6 commits into
huggingface:mainfrom
AdamLouly:adamlouly/fp16_optimizer_support
Dec 8, 2022
Merged

enable FP16Optimizer for fp16 deepspeed training.#547
JingyaHuang merged 6 commits into
huggingface:mainfrom
AdamLouly:adamlouly/fp16_optimizer_support

Conversation

@AdamLouly

@AdamLouly AdamLouly commented Dec 5, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do?

This PR enables FP16 Optimizer when fp16 training is enabled.

Benchmark validation.

This is the logs from both runs with and without fp16 training flag.

Roberta-large 

with fp16 optimizer :
***** train metrics *****
  epoch                    =      13.33
  train_loss               =     0.5796
  train_runtime            = 0:02:46.34
  train_samples            =       3668
  train_samples_per_second =    307.796
  train_steps_per_second   =      1.202

without fp16 optimizer
***** train metrics *****
  epoch                    =      13.33
  train_loss               =     0.5796
  train_runtime            = 0:02:54.16
  train_samples            =       3668
  train_samples_per_second =    293.974
  train_steps_per_second   =      1.148

The recipe that I ran on this test:
"optimum/examples/onnxruntime/training/text-classification/run_glue.py"

command used to run the recipe:
python -m torch.distributed.launch --nproc_per_node=4 /home/adamlouly/optimum/examples/onnxruntime/training/text-classification/run_glue.py --model_name_or_path roberta-large --task_name MRPC --max_seq_length 128 --learning_rate 3e-6 --do_train --output_dir /dev/optim_al --overwrite_output_dir --max_steps 200 --logging_steps 20 --per_device_train_batch_size 64 --fp16 --deepspeed /home/adamlouly/optimum/aml_ds_config_zero_1.json

config json file:

{ "fp16": { "enabled": true, "loss_scale": 0, "loss_scale_window": 1000, "hysteresis": 2, "min_loss_scale": 1 }, "zero_optimization": { "stage": 1, "allgather_partitions": true, "allgather_bucket_size": 200000000, "overlap_comm": true, "reduce_scatter": true, "reduce_bucket_size": 200000000, "contiguous_gradients": false, "cpu_offload": false }, "zero_allow_untested_optimizer": true, "optimizer": { "type": "AdamW", "params": { "lr": "auto", "betas": "auto", "eps": "auto", "weight_decay": "auto" } }, "scheduler": { "type": "WarmupLR", "params": { "warmup_min_lr": "auto", "warmup_max_lr": "auto", "warmup_num_steps": "auto" } }, "steps_per_print": 2000, "train_batch_size": "auto", "train_micro_batch_size_per_gpu": "auto", "gradient_accumulation_steps": "auto", "wall_clock_breakdown": false }

@AdamLouly

AdamLouly commented Dec 5, 2022

Copy link
Copy Markdown
Contributor Author

@JingyaHuang can you please approve the workflows?

@HuggingFaceDocBuilderDev

HuggingFaceDocBuilderDev commented Dec 6, 2022 •
edited
Loading

Copy link
Copy Markdown

The documentation is not available anymore as the PR was closed or merged.

@JingyaHuang

JingyaHuang commented Dec 6, 2022

Copy link
Copy Markdown
Contributor

Hi @AdamLouly,

Thanks for the PR!

Just to have a further understanding of the fp16 optimizer wrapper, it works both for DeepSpeed and Apex right? In Transformers, while using Apex as backend for mixed precision, the optimizer is wrapped in the function _wrap_model,

optimum/optimum/onnxruntime/trainer.py

Line 1488 in eff4c2b

model, self.optimizer = amp.initialize(model, self.optimizer, opt_level=self.args.fp16_opt_level)

will it make sense also to wrap this?

And under the hood, what does the wrapper optimize? Is it compatible regardless of the optimizer used?

@AdamLouly

AdamLouly commented Dec 6, 2022

Copy link
Copy Markdown
Contributor Author

_wrap_model

Yes I think we have to wrap the apex one as well, thanks for pointing it out,

to answer your question:

  • FP16_Optimizer is a simple wrapper to replace inefficient FP16_Optimizer function calls implemented by libraries for example
    Apex, DeepSpeed, Megatron-LM.
  • these are the supported Optimizers.
    "megatron.fp16.fp16.FP16_Optimizer": LegacyMegatronLMModifier, "deepspeed.runtime.zero.stage2.FP16_DeepSpeedZeroOptimizer": DeepSpeedZeROModifier, "deepspeed.runtime.zero.stage_1_and_2.DeepSpeedZeroOptimizer": DeepSpeedZeROModifier, "apex.amp.optimizer.unique_name_as_id": ApexAMPModifier,

JingyaHuang
JingyaHuang reviewed Dec 7, 2022
View reviewed changes

@JingyaHuang JingyaHuang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @AdamLouly , let's remove the exceptions, and then we can merge it!

Comment thread optimum/onnxruntime/trainer.py
Comment thread optimum/onnxruntime/trainer.py
@AdamLouly

AdamLouly commented Dec 8, 2022

Copy link
Copy Markdown
Contributor Author

Updated the perf results on the description using roberta-large

@AdamLouly
AdamLouly requested a review from JingyaHuang December 8, 2022 19:43
@AdamLouly

AdamLouly commented Dec 8, 2022

Copy link
Copy Markdown
Contributor Author

@JingyaHuang formatter trainer.py

@AdamLouly

AdamLouly commented Dec 8, 2022

Copy link
Copy Markdown
Contributor Author

@JingyaHuang do you use black for lint? that's what I used and it still fails, thoughts?

@JingyaHuang

JingyaHuang commented Dec 8, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

@JingyaHuang do you use black for lint? that's what I used and it still fails, thoughts?

@AdamLouly, yes we use black and isort. You can use directly ‘make style’ for formatting.

@AdamLouly

AdamLouly commented Dec 8, 2022

Copy link
Copy Markdown
Contributor Author

@JingyaHuang do you use black for lint? that's what I used and it still fails, thoughts?

@AdamLouly, yes we use black and I sort. You can use directly ‘make style’ for formatting.

make style reformat other files that I have not change as well.

@JingyaHuang

JingyaHuang commented Dec 8, 2022

Copy link
Copy Markdown
Contributor

@AdamLouly,
you could either format trainer.py with the latest black and isort. Or apply make style, but only commit trainer.py.

JingyaHuang
JingyaHuang approved these changes Dec 8, 2022
View reviewed changes

@JingyaHuang JingyaHuang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for iterating, will merge it once the CI pass! @AdamLouly

@JingyaHuang
JingyaHuang merged commit 5c90cf1 into huggingface:main Dec 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JingyaHuang JingyaHuang JingyaHuang approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@AdamLouly @HuggingFaceDocBuilderDev @JingyaHuang