Thanks to @philippfriese well over a year ago, too bad we missed it.

TL;DR configure scripts source site config files either from $CONFIG_SITE, /usr/local/share/config.site or /usr/local/etc/config.site. This is used in openSUSE and RHEL. They set libdir to something different.

Spack does not prevent this, result in build failure and/or non-determinism.

I can confirm this exact error on two other machines. The first machine runs a CentOS 8 on an Arm ThunderX2, the second OpenSUSE 15.5 on an Arm Ampere Altra Max.

On both, this error occurs during installation of Python, with both Spack 0.20 and 0.21.0.dev0 (195f965).

Curiously, I found out that this issue is somehow related to the CONFIG_SITE variable set on both systems in file /etc/profile.d/site.sh to value /usr/share/site/aarch64-unknown-linux-gnu. That file apparently tries to adjust the values of several environment variables, among them libdir and libexecdir. It is invoked by Autoconf during ./configure, which shows up in my logs if I run spack install --verbose [email protected]:

==> [2023-10-27-17:35:10.454666] './configure' [...]

configure: loading site script /usr/share/site/aarch64-unknown-linux-gnu

checking build system type... aarch64-unknown-linux-gnu

[...]

I presume that this script changes some environment variables in a way that breaks the Python package.

Crucially, unsetting CONFIG_SITE prior to running spack install python fixes the issue on my end. I didn't investigate further why exactly this happens, as the logs turn dry at that level of drilldown.

Originally posted by @philippfriese in #40110