Corrupt installation directory during simultaneous spack installs #9394
Description
I attempted to run 2 spack installs simultaneously. I thought that the packages to be built by the commands were entirely disjoint sets, but it turned out that a recent git pull pulled in a newer version of cmake (which is required by both commands). So the first spack install began building the new cmake and the second spack install blocked waiting on the completion of that install. Once the first spack install built and installed cmake, I believe that the second spack install began immediately building and installing cmake, causing the following warning from the first spack install:
==> Successfully installed cmake
Fetch: 7.85s. Build: 3m 52.90s. Total: 4m 0.74s.
[+] /usr/workspace/wsb/herbein1/packages/toss3/toss3/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/cmake-3.12.2-vj5tmvsxlu3d2a7vbmsbj7l5nv3kncas
==> Warning: Dependency missing due to corrupt install directory:
/usr/workspace/wsb/herbein1/packages/toss3/toss3/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/cmake-3.12.2-vj5tmvsxlu3d2a7vbmsbj7l5nv3kncas
Install prefix exists but contains no spec.yaml:
/usr/workspace/wsb/herbein1/packages/toss3/toss3/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/cmake-3.12.2-vj5tmvsxlu3d2a7vbmsbj7l5nv3kncas
The output from the second spack install related to cmake is:
-- Installing: /usr/workspace/wsb/herbein1/packages/toss3/toss3/spack/opt/spack/linux-rhel7-x86_64/gcc-8.2.0/cmake-3.12.2-vj5tmvsxlu3d2a7vbmsbj7l5nv3kncas/share/cmake-3.12/completions/ctest
==> Warning: Module file already exists : skipping creation
file : /usr/workspace/wsb/herbein1/packages/toss3/toss3/spack/share/spack/lmod/linux-rhel7-x86_64/gcc/8.2.0/cmake/3.12.2-vj5.lua
spec : [email protected]%[email protected]~doc+ncurses+openssl+ownlibs patches=dd3a40d4d92f6b2158b87d6fb354c277947c776424aa03f6dc8096cf3135f5d0 ~qt arch=linux-rhel7-x86_64 ^[email protected]%[email protected]~symlinks~termlib arch=linux-rhel7-x86_64 ^[email protected].
2o%[email protected]+systemcerts arch=linux-rhel7-x86_64 ^[email protected]%[email protected] arch=linux-rhel7-x86_64 ^[email protected]%[email protected]+optimize+pic+shared arch=linux-rhel7-x86_64
==> Warning: Module file already exists : skipping creation
file : /usr/workspace/wsb/herbein1/packages/toss3/toss3/spack/share/spack/modules/linux-rhel7-x86_64/cmake-3.12.2-gcc-8.2.0-vj5tmvs
spec : [email protected]%[email protected]~doc+ncurses+openssl+ownlibs patches=dd3a40d4d92f6b2158b87d6fb354c277947c776424aa03f6dc8096cf3135f5d0 ~qt arch=linux-rhel7-x86_64 ^[email protected]%[email protected]~symlinks~termlib arch=linux-rhel7-x86_64 ^[email protected].
2o%[email protected]+systemcerts arch=linux-rhel7-x86_64 ^[email protected]%[email protected] arch=linux-rhel7-x86_64 ^[email protected]%[email protected]+optimize+pic+shared arch=linux-rhel7-x86_64
==> Warning: Module file already exists : skipping creation
file : /usr/workspace/wsb/herbein1/packages/toss3/toss3/spack/share/spack/dotkit/linux-rhel7-x86_64/cmake-3.12.2-gcc-8.2.0-vj5tmvs.dk
spec : [email protected]%[email protected]~doc+ncurses+openssl+ownlibs patches=dd3a40d4d92f6b2158b87d6fb354c277947c776424aa03f6dc8096cf3135f5d0 ~qt arch=linux-rhel7-x86_64 ^[email protected]%[email protected]~symlinks~termlib arch=linux-rhel7-x86_64 ^[email protected].
2o%[email protected]+systemcerts arch=linux-rhel7-x86_64 ^[email protected]%[email protected] arch=linux-rhel7-x86_64 ^[email protected]%[email protected]+optimize+pic+shared arch=linux-rhel7-x86_64
==> Successfully installed cmake
Fetch: 0.08s. Build: 3m 50.76s. Total: 3m 50.84s.
Both builds continued on normally, and as far as I can tell, they finished successfully. I believe the second spack install left the cmake installation in an OK state. I assume nothing bad happened with the corrupt cmake installation because the package dependent on cmake (i.e., jansson) was not rebuilt by spack and the already installed version used. I also assume this was caused by the parallel execution of the spack installs, but maybe something else caused it.
Steps to reproduce the issue
spack install flux-sched@master # in tmux pane 1
spack install flux-sched@master ^[email protected] # in tmux pane 2Error Message
See above messages in summary
Information on your system
This includes:
- which platform you are using: Quartz at LLNL - TOSS3