Linear models and neural nets use a number of optimizer functions/solvers e.g. (fmin_l_bfgs_b (scipy), newton_cg etc). For some solvers (e.g. lbfgs) the individual estimators then needs to check that the convergence happened, and print the appropriate warning. This is done more or less well and sometimes differently depending on the estimator (e.g. #9274 (comment) recently). Other solvers check the convergence internally.

scipy has done some work to unify optimizers with the scipy.optimize.minimize interface. I think it might make sense to use that API when possible (I haven't done a detailed analysis yet to detect possible issues), and check the resulting OptimizeResult object.

This should allow to,

• remove some redundant code in estimators by reusing the same helper functions.
• increase confidence that we are properly checking for the successful convergence
• although it doesn't directly address it, having a common solvers API might help if we want to tackle the issue of determining equivalent stopping criterion (tol) between solvers.
• would help to support third party solvers in the future.

TBH, I'm not sure how much of this can really be generalized outside of lbfgs, that can use a few helper functions to check convergence.

Also related to #9405 that includes a Iteratively Reweighted Least Squares (IRLS) solver.

Comments are welcome.