building zoltan fails if openmpi has been compiled with pmi/slurm-support (i.e. without --enable-static) #8979
Description
Configuration of zoltan fails if openmpi has been compiled with +pmi schedulers=slurm (and thus without --enable-static)
Steps to reproduce the issue
$ spack install openmpi+pmi schedulers=slurm
$ spack install zoltan ^openmpi+pmi schedulers=slurm
==> Building zoltan [Package]
==> Executing phase: 'install'
==> Error: ProcessError: Command exited with status 1:
'/tmp/spack/var/spack/stage/zoltan-3.83-fixlptk6olvzooozkph2jjiutyvewaxm/Zoltan_v3.83/configure' '--prefix=/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/zoltan-3.83-fixlptk6olvzooozkph2jjiutyvewaxm' '--with-cflags=-O3 -fPIC' '--with-cxxflags=-O3 -fPIC' '--with-fcflags=-O3 -fPIC' '--enable-f90interface' '--enable-mpi' 'RANLIB=echo' '--with-ar=$(CXX) -shared $(LDFLAGS) -o' '--with-libs=-lgfortran' 'CC=/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/bin/mpicc' 'CXX=/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/bin/mpic++' 'FC=/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/bin/mpif90' '--with-mpi=/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr' '--with-mpi-libs=-lmpi_usempi_ignore_tkr -lmpi_usempif08 -lmpi_cxx -lompitrace -lmpi_mpifh -lmpi -lmca_common_ompio'
2 errors found in build log:
62 checking whether to use zoltan-examples... yes
63 checking whether additional library search paths defined... no
64 checking whether additional include search paths defined... no
65 checking how to get verbose linking output from /tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/bin/mpif90... -v
66 checking for Fortran libraries of /tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/bin/mpif90... -L/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/
openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib -L/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/hwloc-1.11.9-fvh2jnikqex54g4qath4umb4tclk7k6p/lib -L/apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.
5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.2.0 -L/apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.2
.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.2.0/../../.. -lmpi_usem
pi_ignore_tkr -lmpi_usempif08 -lmpi_cxx -lompitrace -lmpi_mpifh -lmpi -lmca_common_ompio -lgfortran -lm -lquadmath -lpthread
67 checking for dummy main to link with Fortran libraries... unknown
>> 68 configure: error: in `/tmp/hpcins0h/spack-stage/spack-stage-3IwHvo/Zoltan_v3.83/build':
>> 69 configure: error: linking to Fortran libraries from C fails
70 See `config.log' for more details
configure:6640: checking how to get verbose linking output from /tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/bin/mpif90
...
/apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/libexec/gcc/x86_64-pc-linux-gnu/8.2.0/collect2 -plugin /apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh
3f63qrlb4mw6pmzzmgrit/libexec/gcc/x86_64-pc-linux-gnu/8.2.0/liblto_plugin.so -plugin-opt=/apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/libexec/gcc/x86_64-pc-linux-gnu/8.2
.0/lto-wrapper -plugin-opt=-fresolution=/tmp/ccQPzMXR.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lquadmath -plugin-opt=-pass-through=-lm -plugin-opt=-pass-throu
gh=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -rpath /apps/SPACK/opt/linux-centos7-x8
6_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib:/apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib64 --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-l
inux-x86-64.so.2 -o conftest /lib/../lib64/crt1.o /lib/../lib64/crti.o /apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.2.0/crtbegin.o -L/tmp/s
pack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib -L/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/hwloc-1.11.9-fvh2jnikqex54g4qath4umb4tclk7k6p/lib -L/tmp/spack/
opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib -L/apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.
2.0 -L/apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.2.0/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/apps/SPACK/opt/linux-centos
7-x86_64/gcc-4.8.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.2.0/../../.. /tmp/ccexUSTn.o -lmpi_usempi_ignore_tkr -lmpi_usempif08 -lmpi_cxx -lompitrace -lmpi_mpifh -lmpi -lmca_common_
ompio -lgfortran -lm -rpath /tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/hwloc-1.11.9-fvh2jnikqex54g4qath4umb4tclk7k6p/lib -rpath /tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw5
4uqkj5u5jfvscgjeib6jr/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm -lgcc_s -lgcc -lquadmath -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc /apps/SPACK/opt/linux-centos7-x86_64/gcc-4.8
.5/gcc-8.2.0-rhsxipz3bgh3f63qrlb4mw6pmzzmgrit/lib/gcc/x86_64-pc-linux-gnu/8.2.0/crtend.o /lib/../lib64/crtn.o
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_progress_is_registered'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `ompi_io_ompio_decode_datatype'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_bytes_per_agg'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_file_delete'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_coll_timing_info'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_component_progress'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_set_aggregator_props'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_fview_based_grouping'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_ompio_request_t_class'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `ompi_io_ompio_generate_current_file_view'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_get_bytes_per_agg'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_grouping_option'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_sharedfp_lazy_open'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_cycle_buffer_size'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_get_num_aggregators'
/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib/libmca_common_ompio.so: undefined reference to `mca_io_ompio_finalize_initial_grouping'
collect2: error: ld returned 1 exit statusInformation on your system
- CentOS-7.5
- current develop branch without modifications
Analysis and possible solutions
Compiling openmpi without --enabled-shared results in "libmca_*.so" files ending up in openmpi's lib directory, including e.g. libmca_common_ompio.so.
The function get_mpi_libs() from packages/zoltan/package.py looks for any file in the MPI's library directory matching r'^(lib)((\w*)mpi(\w*))\.((a)|({0}))$'.format(dso_suffix).
libmca_common_ompio.so is matched by that regular expression, however, should not be specified for (explicit) linkage.
I did not find a clever way to extend the regular expression to not match libmca* or ompio. Thus, my solution would be to introduce a blacklist and skip any of those, e.g. something like
def get_mpi_libs(self):
mpi_libs = set()
+ # never add one of the blacklisted libs even if they match the regex
+ blacklisted_libs = [ 'mca_common_ompio' ]
for lib_path in glob.glob(join_path(self.spec['mpi'].prefix.lib, '*')):
mpi_lib_match = re.match(
r'^(lib)((\w*)mpi(\w*))\.((a)|({0}))$'.format(dso_suffix),
os.path.basename(lib_path))
- if mpi_lib_match:
+ if mpi_lib_match and mpi_lib_match.group(2) not in blacklisted_libs:
mpi_libs.add(mpi_lib_match.group(2))
return list(mpi_libs)
Any better idea?
At least for openmpi an empty mpi_libs list would be o.k. as the mpi-wrapper automatically pick up the required libs during the configure strep.
openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/bin/mpif90 --showme:link
-fexceptions -pthread -I/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib -L/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/hwloc-1.11.9-fvh2jnikqex54g4qath4umb4tclk7k6p/lib -Wl,-rpath -Wl,/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/hwloc-1.11.9-fvh2jnikqex54g4qath4umb4tclk7k6p/lib -Wl,-rpath -Wl,/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib -L/tmp/spack/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-3.1.1-6vwf3bvmnw54uqkj5u5jfvscgjeib6jr/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
Thus, an other solution might be
def get_mpi_libs(self):
if '^openmpi' in self.spec:
return list()
mpi_libs = set()
...
Opinions before I create a pull request for either of these?