Description

When running PeleLMeX on Frontier at large scale (>256 nodes), some jobs failed during MPI initialization with errors such as:

srun: error: frontier02517: task 1529: Segmentation fault (core dumped) srun: Terminating StepId=3784394.0

or

inet_recv: unexpected socket EOF on frontier0XXXX _pmi_network_allgather:_pmi_inet_recv from target failed Fatal error in PMPI_Init: Other MPI error, PMI_Allgather failed: -1

The failures were intermittent at small scale but consistent at 256+ nodes. HIP was being initialized implicitly and concurrently across all ranks after MPI_Init or during early GPU-aware MPI setup. At very large node counts, this caused contention and race conditions in ROCm’s SDMA and HSA bring-up routines.

Fix

An explicit call to

hipInit(0);

is now issued before MPI or AMReX initialization. This ensures that the HIP runtime and HSA driver are fully brought up on each rank before MPI communication setup begins, preventing concurrent lazy initialization of HIP contexts across thousands of ranks.

Validation

The fix was tested on Frontier (OLCF) for 8 separate runs at 512 nodes.
All runs completed successfully without the previous startup crashes.