Python libraries incorrectly determined in boost#3140
Closed
mdavezac wants to merge 2 commits intospack:developfrom
Closed
Python libraries incorrectly determined in boost#3140mdavezac wants to merge 2 commits intospack:developfrom
mdavezac wants to merge 2 commits intospack:developfrom
Conversation
Member
|
I noticed this same bug when I was working on #3293. I'll test this out tomorrow. |
Member
|
This does indeed solve the problem for me! Testing now on Python 3. Update: Works for Python 3! |
Member
|
Although things work, I think we can do a lot more to simplify def bjam_python_line(self, spec):
all_libs = glob(spec['python'].python_lib_dir)
libs = set([realpath(u) for u in all_libs
if splitext(u)[1] == '.' + dso_suffix])
if len(libs) == 0:
libs = set([realpath(u) for u in all_libs if splitext(u)[1] == '.a'])
# bjam only accepts one library. So if more than one is found, use the first
if len(libs) > 0:
libs = libs[0]
else:
libs = ''
return 'using python : {0} : {1} : {2} : {3} ;\n'.format(
spec['python'].version.up_to(2),
python_exe,
spec['python'].python_include_dir,
libs)I think things can be further simplified with the use of |
Contributor
Author
|
Fair enough. It'll probably be a week or two until I can get back to it though. |
Member
|
@mdavezac I don't want to steal credit for your hard work, but if you want I can open up my own PR with these changes and do some testing. |
Contributor
Author
|
That would be great! Frankly, as long as I can do |
Member
|
I'm going to close this as #3367 should be a much more robust solution. |
tgamblin
pushed a commit
that referenced
this pull request
Apr 30, 2017
## Motivation Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically. I originally got into this mess with #3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on #3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers. Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends. ## Prefix For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through #2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`. Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in #2173 anyway. If we change our minds, I would be more than happy to simplify things. To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable. ## Command In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking: If the spec is for Python 3, try searching for the `python3` command. If the spec is for Python 2, try searching for the `python2` command. If neither are found, try searching for the `python` command. ## Libraries Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see #2295 and #2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work. The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing: ``` lib/libpython2.7.dylib ``` For Python 3.6 on CentOS 6, I'm seeing: ``` lib/libpython3.so lib/libpython3.6m.so.1.0 lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0 ``` Notice the `m` after the version number. Yeah, that's a thing. ## Headers In Python 2.7, I'm seeing: ``` include/python2.7/pyconfig.h ``` In Python 3.6, I'm seeing: ``` include/python3.6m/pyconfig.h ``` It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6 Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.
diaena
pushed a commit
to diaena/spack
that referenced
this pull request
May 26, 2017
## Motivation Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically. I originally got into this mess with spack#3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on spack#3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers. Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends. ## Prefix For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through spack#2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`. Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in spack#2173 anyway. If we change our minds, I would be more than happy to simplify things. To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable. ## Command In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking: If the spec is for Python 3, try searching for the `python3` command. If the spec is for Python 2, try searching for the `python2` command. If neither are found, try searching for the `python` command. ## Libraries Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see spack#2295 and spack#2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work. The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing: ``` lib/libpython2.7.dylib ``` For Python 3.6 on CentOS 6, I'm seeing: ``` lib/libpython3.so lib/libpython3.6m.so.1.0 lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0 ``` Notice the `m` after the version number. Yeah, that's a thing. ## Headers In Python 2.7, I'm seeing: ``` include/python2.7/pyconfig.h ``` In Python 3.6, I'm seeing: ``` include/python3.6m/pyconfig.h ``` It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6 Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.
xavierandrade
pushed a commit
to xavierandrade/spack
that referenced
this pull request
Jun 16, 2017
## Motivation Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically. I originally got into this mess with spack#3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on spack#3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers. Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends. ## Prefix For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through spack#2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`. Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in spack#2173 anyway. If we change our minds, I would be more than happy to simplify things. To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable. ## Command In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking: If the spec is for Python 3, try searching for the `python3` command. If the spec is for Python 2, try searching for the `python2` command. If neither are found, try searching for the `python` command. ## Libraries Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see spack#2295 and spack#2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work. The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing: ``` lib/libpython2.7.dylib ``` For Python 3.6 on CentOS 6, I'm seeing: ``` lib/libpython3.so lib/libpython3.6m.so.1.0 lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0 ``` Notice the `m` after the version number. Yeah, that's a thing. ## Headers In Python 2.7, I'm seeing: ``` include/python2.7/pyconfig.h ``` In Python 3.6, I'm seeing: ``` include/python3.6m/pyconfig.h ``` It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6 Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.
EmreAtes
pushed a commit
to EmreAtes/spack
that referenced
this pull request
Jul 10, 2017
## Motivation Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically. I originally got into this mess with spack#3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on spack#3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers. Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends. ## Prefix For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through spack#2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`. Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in spack#2173 anyway. If we change our minds, I would be more than happy to simplify things. To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable. ## Command In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking: If the spec is for Python 3, try searching for the `python3` command. If the spec is for Python 2, try searching for the `python2` command. If neither are found, try searching for the `python` command. ## Libraries Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see spack#2295 and spack#2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work. The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing: ``` lib/libpython2.7.dylib ``` For Python 3.6 on CentOS 6, I'm seeing: ``` lib/libpython3.so lib/libpython3.6m.so.1.0 lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0 ``` Notice the `m` after the version number. Yeah, that's a thing. ## Headers In Python 2.7, I'm seeing: ``` include/python2.7/pyconfig.h ``` In Python 3.6, I'm seeing: ``` include/python3.6m/pyconfig.h ``` It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6 Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.