Will this result in RPATHs in compiled Python extensions? That would be a big benefit.

@skosukhin could you please address the flake8 errors from the CI output?

@citibeth, I think that in some cases it does this even now. I don't know how it's in general yet, but I can tell how it's for the package I'm working on (which extends python and uses numpy's distutils, which are based on the original distutils). The building scenario of the package is the following:

1. Compile file by file with the executable $CC. In our case the variable $CC is set by spack and refers to its wrapper (if $CC is not set then distutils take the value from _sysconfigdata.py). So here everything works as it should.
2. Link everything with a single call of the executable $LDSHARED. In our case the environment variable is not set, so distutils take the value from _sysconfigdata.py, which was changed at the final step of the installation of the python package to something like "/usr/bin/gcc-4.9 -shared". So this linking step is done by the original compiler directly. After that we depend on how the original compiler is configured. If it's configured to call the linker just by its name "ld" then the spack's wrapper comes into play since it's the first "ld" in the $PATH. And it appends the required rpath arguments. On the other hand the compiler can be configured to call the linker from a particular directory. Then we don't get rpathes.

With this PR python package sets environment variable $LDSHARED for the setup_py executable, which should be called by python extension packages. The value of $LDSHARED refers to the compiler the python was build with, which is the spack's wrapper. That means that the final linking is done by the spack's wrapper in the "ccld" mode, so we get the required rpath arguments. So, we get rpaths in a larger number of cases. I'm not sure if we always get them.

I look forward to trying it out. Someday once it's merged, I'll plan on checking my full Python stack for RPATHs.

@skosukhin Do you know if this works with Python2, Python3 or both? See #2350

@skosukhin Is this PR still WIP? I see you've posted new commits recently.

@citibeth, This works for both in my case.

@becker33, I consider this as finished now. Although I am not sure about two things:

1. Did I choose the right way to keep metadata?
2. Is it ok to keep the cache (the attribute _distutils_data_cache) the way I did it?

Why not add _distutils_data_cache = None at the top of the package file? That would allow you to avoid using hasattr and setattr, and it looks like in this case it would also simplify your code, since you end up using setattr to explicitly set the default value to None.

@lee218llnl: can you take a quick look at this before we merge it in? If it's ok with you, feel free to merge.

When I test this, I encounter the following error on activate:

bash-4.1$ ./bin/spack find
==> 22 installed packages.
-- linux-rhel6-x86_64 / [email protected] -------------------------------
[email protected] [email protected] [email protected] [email protected]
[email protected] [email protected] [email protected] [email protected]
[email protected] [email protected] [email protected] [email protected]
[email protected] [email protected] [email protected] [email protected]
[email protected] [email protected] [email protected]
[email protected] [email protected] [email protected]

bash-4.1$ ./bin/spack activate py-numpy
==> Error: Can only (de)activate extensions for installed packages.

I will do some debugging on my end to see what is causing this, but a pull of develop does not demonstrate this problem. @skosukhin have you tried activating a package with these changes?

@lee218llnl This branch did not include #2524. It does now. Please, try again.

This looks good to me. I can also see with a scipy build that it uses the proper value of LDSHARED.