What if we add an abort to this in everything but GMRES. Can we do a runtime check on the ordering?

This is like a 10% performance increase in total simulation time. The option PELE_CVODE_FORCE_YCORDER opens up the full set of solvers for any advanced users, but I think we should automatically default CYOrder when possible.

I'm ok with this being a compile time option, but I think the behavior here is kind of confusing. If I understand properly, this basically means a user can't choose anything besides GMRES or MAGMA on GPU without modifying source code, because the PELE_CVODE_FORCE_YCORDER define is not exposed to the user though the GNUmakefile in the way that for example PELE_USE_MAGMA is. I think it would be better for users to control this more directly.

Unfortunately, I think the documentation on the chemistry integrators is so out of date that it's not even possible to add an explanation there. We really do need to address that at some point, as right now its very difficult for users to know which combinations of YC/CYOrder, EOS, hardware and solve type are valid, let alone what might be expected to be the fastest.

In each Pele code we would add PELE_CVODE_FORCE_YCORDER which is simple, just like PELE_USE_MAGMA.

Shouldn't that be added to Make.PelePhysics as well?