🐛 Bug

Although the official step order seems to have changed to calling scheduler.step() after each iteration/epoch instead of before, this causes CosineAnnealingWarmRestarts to have an extraneous step at iteration/epoch 0, instead of starting at the maximum specified learning rate.

To Reproduce

Steps to reproduce the behavior:

1. Initialize a scheduler with T_0=1, T_mult=2.
2. Loop through using a for loop, first printing the lr, then stepping.

scheduler = CosineAnnealingWarmRestarts(optimizer, T_0=1, T_mult=2)
for i in range(9):
... print(i)
... print(scheduler.get_lr())
... scheduler.step()
...
0
[0.0]
1
[0.5]
2
[0.5]
3
[0.25]
4
[0.5]
5
[0.42677669529663687]
6
[0.25]
7
[0.07322330470336313]
8
[0.5]

Expected behavior

For cases such as the one above, would expect the restart to occur at the 8th iteration/epoch (index 7 above), but instead occurs at the 9th iteration/epoch (index 8 above). This is probably due to the resetting of last_epoch in the init, since it resets the last_epoch to the initial user given parameter value, despite the fact that the call to the super() init sets it to user given parameter value + 1.

Environment

• PyTorch Version (e.g., 1.0): 1.1
• OS (e.g., Linux): Windows/WSL
• How you installed PyTorch (conda, pip, source): conda
• Build command you used (if compiling from source):
• Python version: 3.7
• CUDA/cuDNN version: 10
• GPU models and configuration: GTX 1070
• Any other relevant information: N/A

Additional context

N/A