I propose to have publicly-facing functions of closed-form scheduling formulas depending only on input iteration number.

This way people who are having troubles with current LR scheduler design (IMO statefulness (both in fields and in the param groups themselves), problems of overriding learning rates after restoring from checkpoint, inability to handle only specific param groups) can still use the tested formulas and wouldn't need to copy this piece as well.

These formulas are useful also for people doing schedulers for quantities other than LR's.

E.g. that would be torch.schedulers.polynomial(iteration) or something like this

This also makes more sense given that optimizers have already gained a functional form as well.

To complement these functions, I propose to create stateless helper functions to work directly with optimizer param groups: setting new values obtained from formulas and getting the last lr. At least, this low-level API should be usable by anyone and will avoid people writing/copying these helpers from scratch.

cc @vincentqb @jbschlosser @albanD