The Enzyme environment can now be instantiate correctly, while still using MPICH_jll v5, which confirms the proposed solution is effective.

I am using this opportunity to learn about this issue from @giordano .

In my use case I would like to use MPITrampoline which trampolines to a custom CUDA-aware MPI that is bundled with NVHPC. Not asking because I am too lazy to check it myself but I would like to understand the problem. Is this expected to work against all kinds of MPI?

So what happens when XESMF is loaded after Oceananigans, given that I've configured the system to use MPITrampoline? Does the conda try to compile something with the MPI that I specify?

Before that when I was running scripts which involve XESMF I also ran into HDF5_jll, NetCDF_jll compilation failures because it complains that I was not using the right MPI (which is the conda's version I believe)

The problem is that the Python package brings its own libmpi:

julia> using XESMF, Libdl

julia> filter(contains("libmpi.so"), dllist())[1]
"/home/runner/work/Oceananigans.jl/Oceananigans.jl/.CondaPkg/.pixi/envs/default/lib/libmpi.so.12"

When then another library on julia's side needs a libmpi.so.12, pixi's one would be already loaded, and if the two aren't compatible (e.g. libmpifort.so from MPICH_jll v5 expects libmpi.so.12 to provide the symbol MPIR_fortran_false, which was introduced in mpich v5.0.0, then there's a problem if pixi's mpich is older).

If you were using MPITrampoline, I believe that'd still try to load libmpi.so.12 at some point, but if you have XESMF already loaded then you'd have the wrong libmpi.so.12 around, which goes boom. By loading Oceananigans (which loads MPI.jl automatically) first, we're now forcing julia's libmpi.so.12 to be loaded first, not letting pixi's library to get into the way.