buildcache relocation: rpath not re-written correctly #31241
Description
Steps to reproduce
Trying to update the E4S CI stack via this PR:
There are many failed jobs, with the same type of error. For this issue let us look at this one, where we cannot build libxau because one of it's build dependencies pkg-config is not usable:
The salient part of the failure log occurs during the configure phase where we see pkg-config cannot find it's library libpkgconf.so.3:
...
>> 88 checking pkg-config is at least version 0.9.0... /home/software/spa
ck/__spack_path_placeholder__/__spack_path_placeholder__/__spack_pa
th_placeholder__/__spack_path_placeholder__/__spack_path_placeholde
r__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
ath_placeholder__/__spack_path_placeholder__/__spack_path_placehold
er__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_
path_placeholder__/__spack_path_placeholder__/__spack_path_placehol
der__/__spack_path_placeholder__/__spack_path_placeholder__/__spack
_path_placeholder__/__spa/linux-ubuntu22.04-x86_64/gcc-11.2.0/pkgco
nf-1.8.0-wsqf3pdlaknj3vt7bq7skjmztdd5flps/bin/pkg-config: error whi
le loading shared libraries: libpkgconf.so.3: cannot open shared ob
ject file: No such file or directory
...
I am able to reproduce the error locally following the reproduction steps:
To reproduce this build locally, run:
spack ci reproduce-build https://gitlab.spack.io/api/v4/projects/2/jobs/2710862/artifacts [--working-dir <dir>]
Picking up right after the error is reproduced, here is what I notice about the state of things:
root@3f0149fcccec:/# spack location -i pkgconfig
/home/software/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spa/linux-ubuntu22.04-x86_64/gcc-11.2.0/pkgconf-1.8.0-wsqf3pdlaknj3vt7bq7skjmztdd5flps
root@3f0149fcccec:/# ldd $(spack location -i pkgconfig)/bin/pkg-config
linux-vdso.so.1 (0x00007ffec89b2000)
libpkgconf.so.3 => not found
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f13dafcf000)
/lib64/ld-linux-x86-64.so.2 (0x00007f13db20c000)
root@3f0149fcccec:/# objdump -x $(spack location -i pkgconfig)/bin/pkg-config | grep -i rpath
RPATH /home/software/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spa/morepadding/linux-ubuntu22.04-x86_64/gcc-11.2.0/pkgconf-1.8.0-wsqf3pdlaknj3vt7bq7skjmztdd5flps/lib:/home/software/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spa/morepadding/linux-ubuntu22.04-x86_64/gcc-11.2.0/pkgconf-1.8.0-wsqf3pdlaknj3vt7bq7skjmztdd5flps/lib64
If you look closely at the above output, you will see the observed RPATH of pkg-config is not correct!
Observed RPATH:
/home/software/spack/<padded-path>/morepadding/linux-ubuntu22.04-x86_64/...
Correct RPATH:
/home/software/spack/<padded-path>/linux-ubuntu22.04-x86_64/....
Notice the observed RPATH contains .../morepadding/... which should have been removed when it was installed from the cache.
During the reproduction, you see this message associated with the relocation of pkg-config from the cache:
...
==> [2022-06-22-11:51:29.532251] Fetching s3://spack-binaries-prs/pr31032_e4s-june-sync/build_cache/linux-ubuntu22.04-x86_64/gcc-11.2.0/pkgconf-1.8.0/linux-ubuntu22.04-x86_64-gcc-11.2.0-pkgconf-1.8.0-wsqf3pdlaknj3vt7bq7skjmztdd5flps.spack
==> [2022-06-22-11:51:31.906954] old relative prefix morepadding/linux-ubuntu22.04-x86_64/gcc-11.2.0/pkgconf-1.8.0-wsqf3pdlaknj3vt7bq7skjmztdd5flps
new relative prefix linux-ubuntu22.04-x86_64/gcc-11.2.0/pkgconf-1.8.0-wsqf3pdlaknj3vt7bq7skjmztdd5flps
relative rpaths False
==> [2022-06-22-11:51:31.944803] Relocating package from
/home/software/spack/[padded-to-506-chars]/__spa to /home/software/spack/[padded-to-506-chars]/__spa.
...
This message seems wrong:
Relocating package from /home/software/spack/[padded-to-506-chars]/__spa to /home/software/spack/[padded-to-506-chars]/__spa.
Shouldn't we be relocating:
- FROM:
/home/software/spack/[padded-to-506-chars]/__spa/morepadding - TO:
/home/software/spack/[padded-to-506-chars]/__spa
??
@scottwittenburg @wspear @gartung
Error message
No response
Information on your system
- Spack: 0.19.0.dev0 (0a27c80)
- Python: 3.10.4
- Platform: linux-ubuntu22.04-cascadelake
- Concretizer: clingo
General information
- I have run
spack debug reportand reported the version of Spack/Python/Platform - I have searched the issues of this repo and believe this is not a duplicate
- I have run the failing commands in debug mode and reported the output