Similarly to #11155, we should avoid casting to float 64.
Now that #11155 introduce SAG and SAGA for float 32 as well, we can reuse the same solver to handle the fused type.