Is your feature request related to a problem? Please describe.
Add schedulers for training transformer-based models (e.g. ViT). These schedulers include:

• WarmupConstantSchedule
• WarmupLinearSchedule
• WarmupCosineSchedule
• WarmupCosineWithHardRestartsSchedule

Describe the solution you'd like
Currently, Pytorch does not have implementations for these schedulers. However, HuggingFace library provides such implementations:
https://huggingface.co/transformers/v1.2.0/_modules/pytorch_transformers/optimization.html