(with apologies to Djikstra and everyone else who's recycled that title meme)

TL;DR: the library that one of my spack binaries uses depends on what other spack packages I've module load-ed. YIKES. See also #3926.

I was trying to understand why @JusticeForMikeBrown was having trouble building bowtie2 (see #3950) when I've built it successfully with [email protected].

His problem with [email protected] was zlib related; I checked the package and noticed that it doesn't have a dependency on zlib. Perhaps it should, I thought. Wonder what zlib my "production" copy was linked against?

$ ldd bowtie2-align-l  | grep libz
	libz.so.1 => /blah/spack/v0.0.8/opt/spack/linux-centos7-x86_64/gcc-5.4.0/zlib-1.2.11-ec535e2ikkpl7hd4y454t3yydjqorja6/lib/libz.so.1 (0x00002aaaaaf32000)

That surprised me, because there's no zlib dependency in the package.

Sure enough, it's because I have something else module load-ed that has the side effect of adding zlib's directory to LD_LIBRARY_PATH.

$ (unset LD_LIBRARY_PATH; ldd bowtie2-align-l) | grep libz
	libz.so.1 => /lib64/libz.so.1 (0x00002aaaaaf2f000)

My "newer" version of CentOS has a /lib64/libz.so.1 that includes gzbuffer (nm didn't help, library's stripped...):

$ strings /lib64/libz.so.1 | grep buffer
gzbuffer
buffer error

so it (probably) works for me either way.

But imagine if there were two versions of a library (perhaps something mathematical) that give different results. Now you have a program giving different results depending on what other Spack applications are also loaded.

THAT would be fun to track down (assuming you even noticed...).

W.R.T. the main problem, bowtie2 should probably have a dependency on a new-ish version of zlib, but stuff like this is why LD_LIBRARY_PATH is a slippery tool to reach for.

I'll argue that this kind of unpredictability is a bigger negative than being helpful and always setting LD_LIBRARY_PATH. This comment in the docs isn't actually correct:

Spack avoids library misconfiguration by using RPATH to link dependencies. When a user links a library or runs a program, it is tied to the dependencies it was built with, so there is no need to manipulate LD_LIBRARY_PATH at runtime.

clipped from here

What would happen if LD_LIBRARY_PATH became opt-in, packages that need it specify it in their package definitions?

Looking at the list of cases where RPATH support doesn't work, it seems like 1) is not relevant (I think it's referring to PERL5LIB, etc...) and 3) are simply bugs. That leaves 2), python extensions. Is RPATH unworkable there or just not yet working?