Issue with buildcache and intel package (symlinks) #10062
Description
While using intel package installed via buildcache, I see :
$ /gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-01/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/bin/icc --vers
ion
/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-01/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/bin/icc: error while loading shared libraries: libintelremotemon.so: cannot open shared object file: No such file or directory
Here is what happening :
Intel compiler package has (many) symlinks:
$ ll /gpfs/bbp.cscs.ch/apps/compilers/install/intel-18.0.1/bin
total 64
drwxrwxr-x+ 2 kumbhar bbp 4096 Aug 14 15:38 ./
drwxrwxr-x+ 11 kumbhar bbp 32768 Aug 14 15:38 ../
lrwxrwxrwx 1 kumbhar bbp 57 Aug 14 15:38 codecov -> ../compilers_and_libraries_2018/linux/bin/intel64/codecov*
lrwxrwxrwx 1 kumbhar bbp 58 Aug 14 15:38 compilervars.csh -> ../compilers_and_libraries_2018/linux/bin/compilervars.csh*
lrwxrwxrwx 1 kumbhar bbp 57 Aug 14 15:38 compilervars.sh -> ../compilers_and_libraries_2018/linux/bin/compilervars.sh*
lrwxrwxrwx 1 kumbhar bbp 53 Aug 14 15:38 fpp -> ../compilers_and_libraries_2018/linux/bin/intel64/fpp*
lrwxrwxrwx 1 kumbhar bbp 53 Aug 14 15:38 icc -> ../compilers_and_libraries_2018/linux/bin/intel64/icc*
The bin directory itself has .so files:
$ ll /gpfs/bbp.cscs.ch/apps/compilers/install/intel-18.0.1/compilers_and_libraries_2018.1.163/linux/bin/intel64/
total 128960
drwxr-xr-x+ 2 kumbhar bbp 4096 Sep 26 21:54 ./
.....
-rwxr-xr-x+ 1 kumbhar bbp 4009096 Aug 14 15:38 icc*
-rwxr-xr-x+ 1 kumbhar bbp 315667 Aug 14 15:38 libcilkrts.so.5*
-rwxr-xr-x+ 1 kumbhar bbp 1915896 Aug 14 15:38 libintelremotemon.so*
The compiler binary has RPATH entry as:
$ objdump -x /gpfs/bbp.cscs.ch/apps/compilers/install/intel-18.0.1/bin/icc | grep RPATH
RPATH $ORIGIN
which helps to link those libraries :
$ ldd /gpfs/bbp.cscs.ch/apps/compilers/install/intel-18.0.1/compilers_and_libraries_2018.1.163/linux/bin/intel64/icc
linux-vdso.so.1 => (0x00007fffedb05000)
libintelremotemon.so => /gpfs/bbp.cscs.ch/apps/compilers/install/intel-18.0.1/compilers_and_libraries_2018.1.163/linux/bin/intel64/libintelremotemon.so (0x00007fffed730000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007fffed4f7000)
But when we create buildcache tarball, the symlinks are replaced with actual content:
$ ll intel-18.0.3-kwbptw7qxxylgkaoj5nm4cmn2nfwq4hw/bin/
total 29184
drwxr-x---+ 2 kumbhar bbp 4096 Dec 8 11:14 ./
drwxr-x---+ 22 kumbhar bbp 4096 Dec 8 11:14 ../
-rwxr-x---+ 1 kumbhar bbp 1929456 Dec 8 11:14 codecov*
-rwxr-x---+ 1 kumbhar bbp 5157 Dec 8 11:14 compilervars.csh*
-rwxr-x---+ 1 kumbhar bbp 4830 Dec 8 11:14 compilervars.sh*
-rwxr-x---+ 1 kumbhar bbp 1736696 Dec 8 11:14 fpp*
-rwxr-x---+ 1 kumbhar bbp 4026824 Dec 8 11:14 icc*
-rwxr-x---+ 1 kumbhar bbp 5157 Dec 8 11:14 iccvars.csh*
-rwxr-x---+ 1 kumbhar bbp 4830 Dec 8 11:14 iccvars.sh*
-rwxr-x---+ 1 kumbhar bbp 4026824 Dec 8 11:14 icpc*
and then if we try to use icc, we don't have libintelremotemon.so in $ORIGIN.
Steps to reproduce the issue
$ spack install [email protected] %gcc
$ spack buildcache create -r -d home [email protected] %gcc
$ tar -xvf home/build_cache/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3/linux-rhel7-x86_64-gcc-4.8.5-intel-18.0.3-kwbptw7qxxylgkaoj5nm4cmn2nfwq4hw.spack
$ tar -xvzf linux-rhel7-x86_64-gcc-4.8.5-intel-18.0.3-kwbptw7qxxylgkaoj5nm4cmn2nfwq4hw.tar.gz
$ intel-18.0.3-kwbptw7qxxylgkaoj5nm4cmn2nfwq4hw/bin/icc
intel-18.0.3-kwbptw7qxxylgkaoj5nm4cmn2nfwq4hw/bin/icc: error while loading shared libraries: libintelremotemon.so: cannot open shared object file: No such file or directoryI wonder if its safe to preserve relative symlinks within the install prefix in buildcache.
Also, note that the module file generated is different :
# 1999-10-01 is built from buildcahce while 1999-10-02 from source
$ diff /gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-01/modules/tcl/linux-rhel7-x86_64/intel/18.0.3 /gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/modules/tcl/linux-rhel7-x86_64/intel/18.0.3
2c2
< ## Module file created by spack (https://github.com/spack/spack) on 2018-12-08 08:47:50.902149
---
> ## Module file created by spack (https://github.com/spack/spack) on 2018-12-08 09:42:46.868419
15,21c15,24
< prepend-path PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-01/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/bin"
< prepend-path MANPATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-01/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/man"
< prepend-path LD_LIBRARY_PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-01/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/lib"
< prepend-path CMAKE_PREFIX_PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-01/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/"
< prepend-path CLASSPATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-01/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib/mpi.jar"
< setenv I_MPI_ROOT "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-01/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/mpi"
< setenv PSTLROOT "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-01/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/pstl"
---
> prepend-path PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/bin"
> prepend-path MANPATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/man"
> prepend-path LD_LIBRARY_PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/lib"
> prepend-path CMAKE_PREFIX_PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/"
> prepend-path CLASSPATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib/mpi.jar"
> prepend-path INTEL_LICENSE_FILE "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/licenses:/opt/intel/licenses:/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/data/intel/licenses"
> setenv I_MPI_ROOT "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/mpi"
> prepend-path NLSPATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64/locale/%l_%t/%N"
> setenv PSTLROOT "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/pstl"
> setenv TBBROOT "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/tbb"
24,25c27,32
< prepend-path LD_LIBRARY_PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-01/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/mpi/mic/lib"
< prepend-path LD_LIBRARY_PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-01/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib"
---
> prepend-path LD_LIBRARY_PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/tbb/lib/intel64/gcc4.7"
> prepend-path LD_LIBRARY_PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/tbb/lib/intel64/gcc4.7"
> prepend-path LD_LIBRARY_PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/mpi/mic/lib"
> prepend-path LD_LIBRARY_PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib"
> prepend-path LD_LIBRARY_PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64_lin"
> prepend-path LD_LIBRARY_PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64"
28c35,36
< prepend-path MANPATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-01/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/mpi/man"
---
> prepend-path MANPATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/mpi/man"
> prepend-path MANPATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/man/common"
35c43,44
< prepend-path PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-01/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/mpi/intel64/bin"
---
> prepend-path PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/mpi/intel64/bin"
> prepend-path PATH "/gpfs/bbp.cscs.ch/apps/hpc/test/jenkins/deployment/install/compilers/1999-10-02/linux-rhel7-x86_64/gcc-4.8.5/intel-18.0.3-kwbptw7qxx/compilers_and_libraries_2018.3.222/linux/bin/intel64"
@gartung : thoughts?
fyi : @matz-e