StepSize Class #1592

In #1586, I presented the new PGA class that implements the following

$$x_{k+1} = prox_{\gamma_{k}g}(x_{k} - \gamma_{k}D(x_{k})\nabla f(x_{k}))$$

Similar to #1588 , where a Preconditioner class is passed for D, I have implemented a new class called StepSizeMethod that changes the step size $\gamma_{k}$. Actually, it changes the Lispchitz constant L for the function f.

It is a ABC + DataClass. At the moment, there are ConstantStepSize and ArmijoStepSize children classes. The constant case returns the constant step size that the user passes or the step size from the L attribute of the function f, if it exists. The ArmijoStepSize is

Since StepSizeMethod works as a Callable class, it has access to self which is the actual algorithm. I have implemented the Backtracking which is for ISTA/FISTA (non-smooth) objectives , Barzilai-Borwein for deterministic or stochastic optimisation, a nice step-size method from Malitsky, Mishchenko which later extended to the non-smooth case by Patrinos et al. Also, our users can define their own step size rule, for instance

from Twyman et al . Also, from SklearnSGD

Example: using Armijo for GD and Accelerated GD = FISTA (g = 0)

Example: GD and GD-Armijo for MRI recon

from this notebook
@ckolbPTB . Atm, it is tested but not unittested.