PR AMReX-Codes#4520 broke Microphysics's autodiff math. It could be fixed on the
Microphysics side. However, we can fix it on the amrex side and we should
use std::enable_if anyway to limit the data types.

PR AMReX-Codes#4520 broke Microphysics's autodiff math. It could be fixed on the
Microphysics side. However, we can fix it on the amrex side and we should
use std::enable_if anyway to limit the data types.

PR #4520 broke Microphysics's autodiff math. It could be fixed on the
Microphysics side. However, we can fix it on the amrex side and we
should use std::enable_if anyway to limit the data types.

---------

Co-authored-by: Axel Huebl <[email protected]>

Basic build system support for AMReX SIMD in superbuilds.

See:
- AMReX-Codes/amrex#4520 (requires this PR)
- BLAST-ImpactX/impactx#1002 (needed for this PR
to be finished)

## Add `ParallelForSIMD<T>`

This adds another template overload to `ParallelForSIMD`.

A typical user pattern for maximum controls so far is:
```C++
#ifdef AMREX_USE_SIMD
if constexpr (amrex::simd::is_vectorized<T>) {
    amrex::ParallelForSIMD<T::simd_width>(np, pushSingleParticle);
} else
#endif
{
    amrex::ParallelFor(np, pushSingleParticle);  // GPU & non-SIMD CPU
}
```

This simplifies it to:
```C++
amrex::ParallelForSIMD<T>(np, pushSingleParticle);
```
indicating there _might_ be a SIMD path if `T` (e.g., a functor)
implements it.

One can still call `ParallelForSIMD` with an explicit SIMD width (int),
as before.

## Generalized Particle Load/Store

A typical SIMD user pattern for particle SoA kernels was:
```C++
SIMDParticleReal<SIMD_WIDTH> part_x;
part_x.copy_from(&m_part_x[i], stdx::element_aligned);

el.compute(part_x);

#ifdef AMREX_USE_SIMD
if constexpr (is_nth_arg_non_const<&el::compute, n>)
    part_x.copy_to(&m_part_x[i], stdx::element_aligned);
#endif
```

This simplifies it to:
```C++
decltype(auto) x = load_1d(m_part_x, i);

el.compute(x);

store_1d<&el::compute, 0>(x, m_part_x, i);
```
and can now also be used for the GPU path, where for now the load is a
transparent pointer forward/deref and the store is a no-OP.

## Combined

Using these patterns together, one can now write single-source
SIMD-CPU/non-SIMD-CPU/GPU kernels, e.g.,
BLAST-ImpactX/impactx#1279

Follow-up to #4520

## Checklist

The proposed changes:
- [ ] fix a bug or incorrect behavior in AMReX
- [x] add new capabilities to AMReX
- [ ] changes answers in the test suite to more than roundoff level
- [ ] are likely to significantly affect the results of downstream AMReX
users
- [x] include documentation in the code and/or rst files, if appropriate

## Summary

Needed to cover SIMD features and generic SIMD/non-SIMD patterns.
Bravely vibe coded, but now reviewed and improved for sensibility.

## Additional background

#4924 #4607 #4600  #4520

## Checklist

The proposed changes:
- [ ] fix a bug or incorrect behavior in AMReX
- [ ] add new capabilities to AMReX
- [ ] changes answers in the test suite to more than roundoff level
- [ ] are likely to significantly affect the results of downstream AMReX
users
- [ ] include documentation in the code and/or rst files, if appropriate

---------

Co-authored-by: Weiqun Zhang <[email protected]>