When we use system provided libraries like openmpi spack doesn't use the module file defined in the packages.yaml:
https://github.com/ACCESS-NRI/spack-config/blob/main/common/gadi/packages.yaml#L218-L224
but writes it's own
System:
-------------------------------------------------------------------
/apps/Modules/modulefiles/openmpi/4.1.7:
setenv HCOLL_ENABLE_MCAST 0
conflict mpi lam mpich openmpi intel-mpi
setenv OMPI_BASE /apps/openmpi/4.1.7
setenv OMPI_ROOT /apps/openmpi/4.1.7
setenv OMPI_VERSION 4.1.7
setenv VT_MAX_FLUSHES 0
setenv VT_PFORM_LDIR /scratch/tm70/aph502/tmp
setenv OMPI_MCA_orte_tmpdir_base /tmp
prepend-path PATH /apps/openmpi/4.1.7/bin
prepend-path C_INCLUDE_PATH /apps/openmpi/4.1.7/include
prepend-path CPLUS_INCLUDE_PATH /apps/openmpi/4.1.7/include
prepend-path CPATH /apps/openmpi/4.1.7/include
prepend-path FPATH /apps/openmpi/4.1.7/include
prepend-path LIBRARY_PATH /apps/openmpi/4.1.7/lib/profilers
prepend-path LD_LIBRARY_PATH /apps/openmpi/4.1.7/lib/profilers
prepend-path LD_RUN_PATH /apps/openmpi/4.1.7/lib/profilers
prepend-path LIBRARY_PATH /apps/openmpi/4.1.7/lib
prepend-path LD_LIBRARY_PATH /apps/openmpi/4.1.7/lib
prepend-path LD_RUN_PATH /apps/openmpi/4.1.7/lib
prepend-path MANPATH /apps/openmpi/4.1.7/share/man
setenv UCC_LOG_LEVEL ERROR
setenv UCC_TLS ^sharp
prepend-path PATH /apps/openmpi/wrapper
prepend-path PATH /apps/openmpi/wrapper/fortran
setenv OPENMPI_BASE /apps/openmpi/4.1.7
setenv OPENMPI_ROOT /apps/openmpi/4.1.7
setenv OPENMPI_VERSION 4.1.7
module-whatis {openmpi, version 4.1.7}
-------------------------------------------------------------------Spack:
-------------------------------------------------------------------
/g/data/vk83/apps/spack/0.22/release/modules/linux-rocky8-x86_64_v4/openmpi/4.1.7-alvuihq:
module-whatis {An open source Message Passing Interface implementation.}
prepend-path PATH /apps/openmpi/4.1.7/bin
prepend-path MANPATH /apps/openmpi/4.1.7/share/man
prepend-path PKG_CONFIG_PATH /apps/openmpi/4.1.7/lib/pkgconfig
prepend-path CMAKE_PREFIX_PATH /apps/openmpi/4.1.7/.
setenv MPICC /apps/openmpi/4.1.7/bin/mpicc
setenv MPICXX /apps/openmpi/4.1.7/bin/mpic++
setenv MPIF77 /apps/openmpi/4.1.7/bin/mpif77
setenv MPIF90 /apps/openmpi/4.1.7/bin/mpif90
setenv MPIFC /apps/openmpi/4.1.7/bin/mpifort
append-path MANPATH {}
-------------------------------------------------------------------and the diff:
2c2
< /apps/Modules/modulefiles/openmpi/4.1.7:
---
> /g/data/vk83/apps/spack/0.22/release/modules/linux-rocky8-x86_64_v4/openmpi/4.1.7-alvuihq:
4,11c4
< setenv HCOLL_ENABLE_MCAST 0
< conflict mpi lam mpich openmpi intel-mpi
< setenv OMPI_BASE /apps/openmpi/4.1.7
< setenv OMPI_ROOT /apps/openmpi/4.1.7
< setenv OMPI_VERSION 4.1.7
< setenv VT_MAX_FLUSHES 0
< setenv VT_PFORM_LDIR /scratch/tm70/aph502/tmp
< setenv OMPI_MCA_orte_tmpdir_base /tmp
---
> module-whatis {An open source Message Passing Interface implementation.}
13,22d5
< prepend-path C_INCLUDE_PATH /apps/openmpi/4.1.7/include
< prepend-path CPLUS_INCLUDE_PATH /apps/openmpi/4.1.7/include
< prepend-path CPATH /apps/openmpi/4.1.7/include
< prepend-path FPATH /apps/openmpi/4.1.7/include
< prepend-path LIBRARY_PATH /apps/openmpi/4.1.7/lib/profilers
< prepend-path LD_LIBRARY_PATH /apps/openmpi/4.1.7/lib/profilers
< prepend-path LD_RUN_PATH /apps/openmpi/4.1.7/lib/profilers
< prepend-path LIBRARY_PATH /apps/openmpi/4.1.7/lib
< prepend-path LD_LIBRARY_PATH /apps/openmpi/4.1.7/lib
< prepend-path LD_RUN_PATH /apps/openmpi/4.1.7/lib
24,31c7,14
< setenv UCC_LOG_LEVEL ERROR
< setenv UCC_TLS ^sharp
< prepend-path PATH /apps/openmpi/wrapper
< prepend-path PATH /apps/openmpi/wrapper/fortran
< setenv OPENMPI_BASE /apps/openmpi/4.1.7
< setenv OPENMPI_ROOT /apps/openmpi/4.1.7
< setenv OPENMPI_VERSION 4.1.7
< module-whatis {openmpi, version 4.1.7}
---
> prepend-path PKG_CONFIG_PATH /apps/openmpi/4.1.7/lib/pkgconfig
> prepend-path CMAKE_PREFIX_PATH /apps/openmpi/4.1.7/.
> setenv MPICC /apps/openmpi/4.1.7/bin/mpicc
> setenv MPICXX /apps/openmpi/4.1.7/bin/mpic++
> setenv MPIF77 /apps/openmpi/4.1.7/bin/mpif77
> setenv MPIF90 /apps/openmpi/4.1.7/bin/mpif90
> setenv MPIFC /apps/openmpi/4.1.7/bin/mpifort
> append-path MANPATH {}More recent versions of the system openmpi are built with support for NVIDIA SHARP, but this is turned off in the module file because it is currently unsupported. This isn't the case when using the spack module and so it produces a large number of spurious error files:
ACCESS-NRI/ACCESS-OM3#143
We have raised this with upstream spack developers in the telecon, why doesn't spack just use the module file specific in packages.yaml? There was no answer AFAIK.
We could add the missing environment variables to our spack configuration to avoid this issue.
When we use system provided libraries like
openmpispackdoesn't use the module file defined in thepackages.yaml:https://github.com/ACCESS-NRI/spack-config/blob/main/common/gadi/packages.yaml#L218-L224
but writes it's own
System:
------------------------------------------------------------------- /apps/Modules/modulefiles/openmpi/4.1.7: setenv HCOLL_ENABLE_MCAST 0 conflict mpi lam mpich openmpi intel-mpi setenv OMPI_BASE /apps/openmpi/4.1.7 setenv OMPI_ROOT /apps/openmpi/4.1.7 setenv OMPI_VERSION 4.1.7 setenv VT_MAX_FLUSHES 0 setenv VT_PFORM_LDIR /scratch/tm70/aph502/tmp setenv OMPI_MCA_orte_tmpdir_base /tmp prepend-path PATH /apps/openmpi/4.1.7/bin prepend-path C_INCLUDE_PATH /apps/openmpi/4.1.7/include prepend-path CPLUS_INCLUDE_PATH /apps/openmpi/4.1.7/include prepend-path CPATH /apps/openmpi/4.1.7/include prepend-path FPATH /apps/openmpi/4.1.7/include prepend-path LIBRARY_PATH /apps/openmpi/4.1.7/lib/profilers prepend-path LD_LIBRARY_PATH /apps/openmpi/4.1.7/lib/profilers prepend-path LD_RUN_PATH /apps/openmpi/4.1.7/lib/profilers prepend-path LIBRARY_PATH /apps/openmpi/4.1.7/lib prepend-path LD_LIBRARY_PATH /apps/openmpi/4.1.7/lib prepend-path LD_RUN_PATH /apps/openmpi/4.1.7/lib prepend-path MANPATH /apps/openmpi/4.1.7/share/man setenv UCC_LOG_LEVEL ERROR setenv UCC_TLS ^sharp prepend-path PATH /apps/openmpi/wrapper prepend-path PATH /apps/openmpi/wrapper/fortran setenv OPENMPI_BASE /apps/openmpi/4.1.7 setenv OPENMPI_ROOT /apps/openmpi/4.1.7 setenv OPENMPI_VERSION 4.1.7 module-whatis {openmpi, version 4.1.7} -------------------------------------------------------------------Spack:
------------------------------------------------------------------- /g/data/vk83/apps/spack/0.22/release/modules/linux-rocky8-x86_64_v4/openmpi/4.1.7-alvuihq: module-whatis {An open source Message Passing Interface implementation.} prepend-path PATH /apps/openmpi/4.1.7/bin prepend-path MANPATH /apps/openmpi/4.1.7/share/man prepend-path PKG_CONFIG_PATH /apps/openmpi/4.1.7/lib/pkgconfig prepend-path CMAKE_PREFIX_PATH /apps/openmpi/4.1.7/. setenv MPICC /apps/openmpi/4.1.7/bin/mpicc setenv MPICXX /apps/openmpi/4.1.7/bin/mpic++ setenv MPIF77 /apps/openmpi/4.1.7/bin/mpif77 setenv MPIF90 /apps/openmpi/4.1.7/bin/mpif90 setenv MPIFC /apps/openmpi/4.1.7/bin/mpifort append-path MANPATH {} -------------------------------------------------------------------and the diff:
More recent versions of the system
openmpiare built with support for NVIDIA SHARP, but this is turned off in the module file because it is currently unsupported. This isn't the case when using thespackmodule and so it produces a large number of spurious error files:ACCESS-NRI/ACCESS-OM3#143
We have raised this with upstream spack developers in the telecon, why doesn't spack just use the module file specific in
packages.yaml? There was no answer AFAIK.We could add the missing environment variables to our spack configuration to avoid this issue.