Python command, libraries, and headers by adamjstewart · Pull Request #3367 · spack/spack

adamjstewart · 2017-03-04T22:53:24Z

Fixes #3478.
Fixes #3594.
Fixes #4051.

Motivation

Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically.

I originally got into this mess with #3274, where I discovered that Boost could not be built with Python 3 because the executable is called python3 and we were telling it to use python. I got deeper into this mess when I started hacking on #3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers.

Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use python, spec['python'].prefix.lib, and spec['python'].prefix.include. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to spec['blas'].libs and friends.

Prefix

For most packages in Spack, we assume that the installation directory is spec['python'].prefix. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, which python would return /usr/local/bin/python, and most users would erroneously assume that /usr/local is the installation directory. If you peruse through #2173, you'll immediately see why this is not the case. Homebrew actually installs Python in /usr/local/Cellar/python/2.7.12_2 and symlinks the executable to /usr/local/bin/python. PYTHONHOME (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not /usr/local.

Normally I would say, "sounds like user error, make sure to use the real installation directory in your packages.yaml". But I think we can make a special case for Python. That's what we decided in #2173 anyway. If we change our minds, I would be more than happy to simplify things.

To solve this problem, I created a spec['python'].home attribute that works the same way as spec['python'].prefix but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite spec['python'].prefix? I think it's currently immutable.

Command

In general, Python 2 comes with both python and python2 commands, while Python 3 only comes with a python3 command. But this is up to the OS developers. For example, /usr/bin/python on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking:

If the spec is for Python 3, try searching for the python3 command.
If the spec is for Python 2, try searching for the python2 command.
If neither are found, try searching for the python command.

Libraries

Spack installs Python libraries in spec['python'].prefix.lib. Except on openSUSE 13, where it installs to spec['python'].prefix.lib64 (see #2295 and #2253). On my CentOS 6 machine, the Python libraries are installed in /usr/lib64. Both need to work.

The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing:

lib/libpython2.7.dylib

For Python 3.6 on CentOS 6, I'm seeing:

lib/libpython3.so
lib/libpython3.6m.so.1.0
lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0

Notice the m after the version number. Yeah, that's a thing.

Headers

In Python 2.7, I'm seeing:

include/python2.7/pyconfig.h

In Python 3.6, I'm seeing:

include/python3.6m/pyconfig.h

It looks like all Python 3 installations have this m. Tested with Python 3.2 and 3.6 on macOS and CentOS 6

Spack has really nice support for libraries (find_libraries and LibraryList), but nothing for headers. Fixed.

Reviewing

This is obviously a big change, and will need thorough testing. I'm looking at the following people for reviews:

@alalazo @davydden I believe you did most of the existing work on LibraryList and find_libraries. Can you make sure my header-equivalents look solid?
@citibeth You're one of the few Spack users who makes heavy use of Python 3. Can you check that things work?
@BarrySmith You went through hell to get Spack working for your Homebrew-installed Python. Can you confirm that my changes still work the way you need?
@mdavezac You got me into this mess. Can you test Boost for me?

adamjstewart · 2017-03-04T22:54:20Z

lib/spack/llnl/util/filesystem.py

+
+    .. code-block:: python
+
+       >>> find('/usr/local/bin', 'python', recurse=False)


I'm honestly considering just converting this to a wrapper around the system find. Thoughts? Either way, I think this would be a very useful command to have.

adamjstewart · 2017-03-04T22:55:48Z

lib/spack/llnl/util/filesystem.py

-    def __init__(self, libraries):
-        self.libraries = list(libraries)
+    def __init__(self, files):
+        self.files = list(dedupe(files))


@alalazo One change I made was to remove duplicates from the list when creating the object. Is there any reason this would be a problem?

adamjstewart · 2017-03-04T22:57:12Z

lib/spack/llnl/util/filesystem.py

+    """Returns an iterable object containing a list of full paths to
+    headers if found.
+
+    Accepts any glob characters accepted by fnmatch:


New features, yay! This allows you to search for headers like *.h or libraries like *.so.

adamjstewart · 2017-03-04T23:00:31Z

var/spack/repos/builtin/packages/python/package.py

+        elif os.path.exists(os.path.join(self.prefix.bin, 'python')):
+            exe = 'python'
+        else:
+            msg = 'Cannot locate python executable in {0}'


By including the search directory in the error message, I'm hoping that users who erroneously add /usr/bin/ to their packages.yaml will see that they actually need to use /usr.

adamjstewart · 2017-03-04T23:01:37Z

var/spack/repos/builtin/packages/python/package.py

+        if headers:
+            return headers
+
+        msg = 'Cannot locate python headers in {0}'


This error message will be helpful for users who use an externally installed system python but don't have the development headers installed.

davydden

looks very nice! Thanks for extensive tests of the new HeaderList class. When the PR will be ready to merge, i can try building boost+python on Sierra with clang.

citibeth · 2017-03-05T01:32:40Z

While the infrastructure in this PR looks useful, I suspect it's not really the right approach for Python. Python already has ways to tell you about its configuration (the sysconfig module), and I think we should use those Python-standard methods. All Spack should be doing is finding the python/python2/python3 executable; and then it should be called to determine the rest.

https://docs.python.org/3/library/sysconfig.html

sysconfig is Python3. For Python2, similar functionality is provided in distutils:

https://bugs.launchpad.net/python-uinput/+bug/1032773
http://stackoverflow.com/questions/11825516/are-the-paths-to-the-python-standard-library-and-to-the-dist-packages-different

Here's an example of where I use sysconfig in a CMake build:

https://github.com/citibeth/icebin/blob/develop/cmake/FindPython.cmake

citibeth

Please try using Python's distutils/sysconfig before we resort to digging around in distros.

adamjstewart · 2017-03-05T02:17:25Z

I'll take a look next week and see what I can do. Thanks @citibeth!

BarrySmith · 2017-03-05T04:36:18Z

var/spack/repos/builtin/packages/foam-extend/package.py

-            'PYTHON_DIR': self.spec['python'].prefix,
-            'PYTHON_BIN_DIR': self.spec['python'].prefix.bin,
+            'PYTHON_DIR': self.spec['python'].home,
+            'PYTHON_BIN_DIR': self.spec['python'].home.bin),


home.bin),

Has an extra stray ) that my python doesn't like.

alalazo · 2017-03-05T07:40:48Z

var/spack/repos/builtin/packages/python/package.py

        # Need this to allow python build to find the Python installation.
-        env['PYTHONHOME'], env['PYTHONPATH'] = prefix, prefix
-        env['MACOSX_DEPLOYMENT_TARGET'] = '10.6'
+        spack_env['PYTHONHOME'] = prefix


Shouldn't this (and the lines below) be:

spack_env.set('PYTHONHOME', prefix)

?

I wonder if we should allow both modes. Fixed.

wscullin · 2017-03-09T00:33:01Z

So, to chime in, not all Python 3.x builds will have *m. This is an ABI flag, defined in PEP 3149 as part of the aftermath of PEP 384. The most common POSIX-y ABI flags we see are:

d: built with Py_DEBUG set
m: uses pymalloc
u: with wide unicode support

I've seen in dev environments:
a: forced memory alignment

and for modules the entire string may bloat up to include interpreters and platform triplets.

mdavezac · 2017-03-09T08:01:59Z

var/spack/repos/builtin/packages/python/package.py

-        env['MACOSX_DEPLOYMENT_TARGET'] = '10.6'
+        spack_env.set('PYTHONHOME', prefix)
+        spack_env.set('PYTHONPATH', prefix)
+        spack_env.set('MACOSX_DEPLOYMENT_TARGET', '10.6')


There is a way to get this number for a specific platform since 2.3, apparently:

from platform import mac_ver mac_ver()[0]

It seems to return an empty string on other platforms, so it should be safe to use.

@mathstuf It looks like you added this in #188. Can you comment on this? Was it added because RPATH support didn't work otherwise in 10.5 and earlier?

I would assume so. Python's build is…unique to say the least :) .

mdavezac · 2017-03-09T08:56:30Z

var/spack/repos/builtin/packages/boost/package.py


        if '+python' in spec:
-            options.append('--with-python=%s' % python_exe)
+            options.append('--with-python=%s' % spec['python'].executable)


I'm getting an error here saying 'executable' is not defined.
I am using an external python here. Do these attributes get defined in that case?

Looks the problem also exists for Spack-built Python. Apparently spec['python'] and spec['python'].package are very different. I think I have a fix. I'll push it out after some more testing.

FYI: spec['python'] is a Spec, which is basically the sub-DAG. spec['python'].package is an instance of Package, which is the build system. If you want a decent way to think about the difference, here you go:

Think of the Spec as a descriptor -- it's an arbitrary DAG describing a build configuration. You can instantiate a Spec for foo^bar+baz%intel without knowing anything about packages Foo or Bar. You cannot concretize or actually build that Spec without a Package called Foo and another called Bar.

The Package does two things: at the class level, it gives you metadata used to concretize Specs (depends_on(), variant(), etc.). An instance of Package needs to be constructed with a Spec, and it provides you with the build system for that package and its dependencies.

We could probably separate (2) out, but IMHO the metaphor of classes-as-package-definitions is a nice one, and it makes it easier for packagers to think about how they can code things up in Python. We stole that from Homebrew.

mdavezac · 2017-03-09T09:01:09Z

var/spack/repos/builtin/packages/python/package.py

+
+        cmd = 'from distutils.sysconfig import get_config_var; '
+        cmd += self.print_string("get_config_var('{0}')".format(key))
+


I'm getting an error here about 4 rather than 1 argument.
It would seem that the line is missing a pair of brackets. But since it's used elsewhere, it might be something less benign, due to the fact that I'm trying to install boost +python with python an external package?

Looks like this problem exists for Spack-built Python as well. The problem was that I forgot to make self.python a @property. Fixed.

mdavezac · 2017-03-09T09:02:26Z

var/spack/repos/builtin/packages/boost/package.py


        if '+python' in spec:
-            options.append('--with-python=%s' % python_exe)
+            options.append('--with-python=%s' % spec['python'].executable)


When installing with python as an external package, spack complains 'executable' is not defined in 'spec' of type 'Spec'.

adamjstewart · 2017-03-09T16:11:49Z

Thanks @wscullin! I think @citibeth's approach (now implemented) will be much more flexible since it asks Python where it's libraries will be installed and named.

adamjstewart · 2017-03-09T19:06:57Z

lib/spack/spack/spec.py

        descriptor `__get__` method
    """
-    name = 'lib' + spec.name
-    shared = '+shared' in spec


@alalazo Previously, any package without a shared variant would search for static libraries. I changed this behavior below.

yes, indeed. This fixes the issue I encountered as well.

adamjstewart · 2017-03-09T19:07:47Z

lib/spack/spack/spec.py

        descriptor `__get__` method
    """
-    return '-I' + spec.prefix.include
+    name = 'lib' + spec.name


I wonder if we should just search for all libraries instead of just lib{name}. My default header handler searches for all *.h files.

adamjstewart · 2017-03-09T19:08:28Z

lib/spack/spack/test/database.py

    assert len(libraries) == 1

-    cppflags_expected = '-I' + spec.prefix.include
-    assert spec['zmpi'].cppflags == cppflags_expected


spec['zmpi'].headers.cpp_flags wasn't working for this test. Possibly because it's a fake installation and doesn't actually contain any header files.

@adamjstewart I would rather modify the mocking if it needs to look "more real" than removing a test.

How/where would I do that?

@adamjstewart: You could change what install --fake puts in the target directory to include a dummy header (it contains a dummy library at the moment -- not sure about the header).

mathstuf · 2017-03-09T19:09:57Z

Notice the m after the version number. Yeah, that's a thing.

These are called "abiflags" and there are multiple ones available. m for "multibyte", d for "debug". I think there are a few more too (u?). Luckily, they are always sorted alphabetically when there are multiples.

adamjstewart · 2017-03-09T19:12:14Z

Thanks for the testing everyone! I think I've resolved most of the bugs at this point. I was able to install boost+python ^[email protected]. Still need to test other versions of Python and external packages/system installations. Still working on a way to cache the results of get_config_vars so that we don't have to eat the startup cost of Python multiple times.

mdavezac · 2017-03-10T10:16:53Z

I've just checks boost+python with an external python on a mac (installed the standard mac packager homebrew). Unfortunately,

'LIBDIR': '/usr/local/opt/python/Frameworks/Python.framework/Versions/2.7/lib'
'LDLIBRARY': 'Python.framework/Versions/2.7/Python',

so LDLIBRARY is not just the filename ('Python' is a symlink to a .dylib), so the current code fails to find the python library.

One solution might be to check if

'PYTHONFRAMEWORKPREFIX': '/usr/local/opt/python/Frameworks'

is empty or not, and then check whether PYTHONFRAMEWORKPREFIX/LDLIBRARY is a thing:

isfile(join(get_config_vars("PYTHONFRAMEWORKPREFIX")[0], get_config_vars("LDLIBRARY")[0]))

adamjstewart · 2017-03-10T13:04:19Z

Ugh, it sounds like macOS Python is drastically different from Linux Python. I'll take a look.

adamjstewart · 2017-03-15T02:02:00Z

Did some comparisons across Anaconda, Homebrew, and system installations of Python 2 and 3:

anaconda2-python2.txt:prefix=/Users/Adam/anaconda2
anaconda3-python3.txt:prefix=/Users/Adam/anaconda3
homebrew-python2.txt:prefix=/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7
homebrew-python3.txt:prefix=/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5
usr-bin-python2.txt:prefix=/System/Library/Frameworks/Python.framework/Versions/2.7

anaconda2-python2.txt:LIBDIR=/Users/Adam/anaconda2/lib
anaconda3-python3.txt:LIBDIR=/Users/Adam/anaconda3/lib
homebrew-python2.txt:LIBDIR=/usr/local/opt/python/Frameworks/Python.framework/Versions/2.7/lib
homebrew-python3.txt:LIBDIR=/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.5/lib
usr-bin-python2.txt:LIBDIR=/System/Library/Frameworks/Python.framework/Versions/2.7/lib

anaconda2-python2.txt:LIBRARY=libpython2.7.a
anaconda3-python3.txt:LIBRARY=libpython3.5m.a
homebrew-python2.txt:LIBRARY=libpython2.7.a
homebrew-python3.txt:LIBRARY=libpython3.5m.a
usr-bin-python2.txt:LIBRARY=libpython2.7.a

naconda2-python2.txt:LDLIBRARY=libpython2.7.dylib
anaconda3-python3.txt:LDLIBRARY=libpython3.5m.dylib
homebrew-python2.txt:LDLIBRARY=Python.framework/Versions/2.7/Python
homebrew-python3.txt:LDLIBRARY=Python.framework/Versions/3.5/Python
usr-bin-python2.txt:LDLIBRARY=Python.framework/Versions/2.7/Python

Everything looks correct except LDLIBRARY. At this point I'm thinking about just running find_libraries on LIBDIR with libpython{0}*'.format(self.version.up_to(2)).

adamjstewart · 2017-03-15T16:11:24Z

@mdavezac Can you try this again?

adamjstewart · 2017-04-28T18:48:25Z

Ping @tgamblin 😄

tgamblin · 2017-04-29T22:55:31Z

@adamjstewart: you can also from six import iteritems

adamjstewart · 2017-04-29T22:59:38Z

Is there any benefit to that? Do you think we'll have problems with memory usage on Python 2?

tgamblin · 2017-04-30T00:20:58Z

Is there any benefit to that? Do you think we'll have problems with memory usage on Python 2?

For small stuff, no. But .items() creates a new array, while iteritems does not. So if it's called frequently it may slow things down.

alalazo · 2017-05-05T13:11:23Z

lib/spack/llnl/util/filesystem.py

+def _find_recursive(root, search_files):
+    found_files = []
+
+    for path, _, list_files in os.walk(root):


@adamjstewart I think here you got the order of the for loops wrong: if you start walking the directory tree first and then search each file in a single sub-directory you may scramble the original order of search_files.

I think you're completely right, good spot! To be fair, the original method I stole this from did it this way too. I don't think we ever noticed because most of the time the libraries are all in the same directory.

I think you're completely right, good spot! To be fair, the original method I stole this from did it this way too

Hey, don't underestimate the fact that I was using a dictionary for that. I just didn't want to traverse the directories multiple times, but I recorded what I found there and reconstructed it in order. At least I think I did it 😄

Do you want to submit a PR? Or should I take a stab at it?

I'll let you handle it, should only be a couple lines to change.

… output) PR spack#3367 inadvertently changed the semantics of _find_recursive and _find_non_recursive so that the returned list are not ordered as the input search list. This commit restores the original semantic, and adds tests to verify it.

… output) (#72) PR spack#3367 inadvertently changed the semantics of _find_recursive and _find_non_recursive so that the returned list are not ordered as the input search list. This commit restores the original semantic, and adds tests to verify it.

…ut) (#4156) PR #3367 inadvertently changed the semantics of _find_recursive and _find_non_recursive so that the returned list are not ordered as the input search list. This commit restores the original semantic, and adds tests to verify it.

@alalazo

## Motivation Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically. I originally got into this mess with spack#3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on spack#3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers. Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends. ## Prefix For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through spack#2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`. Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in spack#2173 anyway. If we change our minds, I would be more than happy to simplify things. To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable. ## Command In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking: If the spec is for Python 3, try searching for the `python3` command. If the spec is for Python 2, try searching for the `python2` command. If neither are found, try searching for the `python` command. ## Libraries Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see spack#2295 and spack#2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work. The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing: ``` lib/libpython2.7.dylib ``` For Python 3.6 on CentOS 6, I'm seeing: ``` lib/libpython3.so lib/libpython3.6m.so.1.0 lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0 ``` Notice the `m` after the version number. Yeah, that's a thing. ## Headers In Python 2.7, I'm seeing: ``` include/python2.7/pyconfig.h ``` In Python 3.6, I'm seeing: ``` include/python3.6m/pyconfig.h ``` It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6 Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.

… output) (spack#4156) PR spack#3367 inadvertently changed the semantics of _find_recursive and _find_non_recursive so that the returned list are not ordered as the input search list. This commit restores the original semantic, and adds tests to verify it.

@alalazo

## Motivation Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically. I originally got into this mess with spack#3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on spack#3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers. Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends. ## Prefix For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through spack#2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`. Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in spack#2173 anyway. If we change our minds, I would be more than happy to simplify things. To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable. ## Command In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking: If the spec is for Python 3, try searching for the `python3` command. If the spec is for Python 2, try searching for the `python2` command. If neither are found, try searching for the `python` command. ## Libraries Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see spack#2295 and spack#2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work. The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing: ``` lib/libpython2.7.dylib ``` For Python 3.6 on CentOS 6, I'm seeing: ``` lib/libpython3.so lib/libpython3.6m.so.1.0 lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0 ``` Notice the `m` after the version number. Yeah, that's a thing. ## Headers In Python 2.7, I'm seeing: ``` include/python2.7/pyconfig.h ``` In Python 3.6, I'm seeing: ``` include/python3.6m/pyconfig.h ``` It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6 Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.

… output) (spack#4156) PR spack#3367 inadvertently changed the semantics of _find_recursive and _find_non_recursive so that the returned list are not ordered as the input search list. This commit restores the original semantic, and adds tests to verify it.

@alalazo

## Motivation Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically. I originally got into this mess with spack#3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on spack#3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers. Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends. ## Prefix For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through spack#2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`. Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in spack#2173 anyway. If we change our minds, I would be more than happy to simplify things. To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable. ## Command In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking: If the spec is for Python 3, try searching for the `python3` command. If the spec is for Python 2, try searching for the `python2` command. If neither are found, try searching for the `python` command. ## Libraries Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see spack#2295 and spack#2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work. The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing: ``` lib/libpython2.7.dylib ``` For Python 3.6 on CentOS 6, I'm seeing: ``` lib/libpython3.so lib/libpython3.6m.so.1.0 lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0 ``` Notice the `m` after the version number. Yeah, that's a thing. ## Headers In Python 2.7, I'm seeing: ``` include/python2.7/pyconfig.h ``` In Python 3.6, I'm seeing: ``` include/python3.6m/pyconfig.h ``` It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6 Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.

… output) (spack#4156) PR spack#3367 inadvertently changed the semantics of _find_recursive and _find_non_recursive so that the returned list are not ordered as the input search list. This commit restores the original semantic, and adds tests to verify it.

This PR includes the following changes: * Added JDK 10 * Changed the JDK version numbers according to the consensus reached in #2284 * Added spec['java'].home and spec['java'].libs, similar to #3367 (JDK and IcedTea) * Added a check to prevent people from installing JDK on macOS * Set CLASSPATH for packages depending on Java (JDK and IcedTea) * Add TODO for extending virtual packages (not currently possible) * Add TODO for adding Java dependents to views * Add TODO for packages which extend multiple packages (e.g. Java and Python)

This PR includes the following changes: * Added JDK 10 * Changed the JDK version numbers according to the consensus reached in spack#2284 * Added spec['java'].home and spec['java'].libs, similar to spack#3367 (JDK and IcedTea) * Added a check to prevent people from installing JDK on macOS * Set CLASSPATH for packages depending on Java (JDK and IcedTea) * Add TODO for extending virtual packages (not currently possible) * Add TODO for adding Java dependents to views * Add TODO for packages which extend multiple packages (e.g. Java and Python)

adamjstewart added dependencies external-packages python WIP labels Mar 4, 2017

adamjstewart commented Mar 4, 2017

View reviewed changes

davydden approved these changes Mar 4, 2017

View reviewed changes

citibeth suggested changes Mar 5, 2017

View reviewed changes

BarrySmith reviewed Mar 5, 2017

View reviewed changes

alalazo reviewed Mar 5, 2017

View reviewed changes

mdavezac reviewed Mar 9, 2017

View reviewed changes

adamjstewart commented Mar 9, 2017

View reviewed changes

adamjstewart force-pushed the features/python-headerlist branch from dd139be to c71d276 Compare March 11, 2017 15:11

adamjstewart force-pushed the features/python-headerlist branch from c71d276 to cd6b18e Compare March 15, 2017 16:08

adamjstewart force-pushed the features/python-headerlist branch from ecc96ba to 490893f Compare March 17, 2017 16:52

iteritems doesn't work in Python 3, use items instead

fd00548

adamjstewart mentioned this pull request Apr 29, 2017

depends_on(python) broken on python3 #4051

Closed

tgamblin merged commit ce3ab50 into spack:develop Apr 30, 2017

adamjstewart deleted the features/python-headerlist branch April 30, 2017 01:38

alalazo reviewed May 5, 2017

View reviewed changes

alalazo mentioned this pull request May 7, 2017

LibraryList and HeaderList not ordered properly #4156

Merged

alalazo mentioned this pull request May 7, 2017

filesystem.py: fixed bug introduced in #3367 (scrambled order in output) epfl-scitas/spack#72

Merged

adamjstewart mentioned this pull request Jun 7, 2017

Speeding up Modules Tutorial #4448

Closed

tgamblin mentioned this pull request Jun 12, 2017

Adding a RubyPackage to enable ruby extensions. #3127

Closed

tgamblin added this to the v0.11.0 milestone Nov 12, 2017

adamjstewart mentioned this pull request Jun 29, 2018

Major overhaul of Java packages #8613

Merged

6 tasks


		.. code-block:: python

		>>> find('/usr/local/bin', 'python', recurse=False)


		cmd = 'from distutils.sysconfig import get_config_var; '
		cmd += self.print_string("get_config_var('{0}')".format(key))

Conversation

adamjstewart commented Mar 4, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Prefix

Command

Libraries

Headers

Reviewing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamjstewart Mar 4, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davydden left a comment

Choose a reason for hiding this comment

Uh oh!

citibeth commented Mar 5, 2017

Uh oh!

citibeth left a comment

Choose a reason for hiding this comment

Uh oh!

adamjstewart commented Mar 5, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wscullin commented Mar 9, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tgamblin Apr 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamjstewart commented Mar 9, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamjstewart commented Mar 4, 2017 •

edited

Loading

adamjstewart Mar 4, 2017 •

edited

Loading

tgamblin Apr 2, 2017 •

edited

Loading

tgamblin commented Apr 30, 2017 •

edited

Loading