python: improved external detection for obscure variants#20116
python: improved external detection for obscure variants#20116
Conversation
| # Get config args for shared, debug, and optimizations variant checks | ||
| sysconf_imp = 'import sysconfig' | ||
| if version < Version('2.7'): | ||
| sysconf_imp = 'import distutils.sysconfig as sysconfig' | ||
| conf_args = python( | ||
| '-c', | ||
| '%s; print(sysconfig.get_config_var("CONFIG_ARGS"))' % sysconf_imp, | ||
| output=str, error=str) |
There was a problem hiding this comment.
See self.get_config_var. It's hard-coded to use self.command, but maybe we could modify it so there is less overlap. At the very least, this print syntax won't work for older versions of Python. That's why we use self.print_string elsewhere.
There was a problem hiding this comment.
I thought as long as you were printing a single value, print(foo) and print foo were the same at least as far back as 2.6. I only have 2.7 around to confirm on though. Do you have easy access to 2.6 or should I build it to test?
There was a problem hiding this comment.
I guess they are the same, as (foo) will be treated as foo.
| output=str, error=str) | ||
| variants += '~ucs4' if maxunicode == 0xFFFF else '+ucs4' | ||
|
|
||
| # +pythoncommand does not apply to python@2 |
There was a problem hiding this comment.
Personally, I don't think we should set this variant for Python 2 at all.
There was a problem hiding this comment.
We have to set the variant, we don't (yet) have a mechanism to have the concretizer omit a variant for certain versions. ucs4 has a similar issue, where in that case we raise exceptions if we try to build with it at certain versions.
We can either set it to the default in the detection logic, or we can let the concretizer fill it in. The latter would be appealing except for the fact that it looks like we're going to need to swap the python that Spack is run with into an already concrete spec as part of our bootstrapping process (to know which binary to download for clingo/libstdc++). That means we need to have all the variants filled out by the detection logic.
There was a problem hiding this comment.
Hmm, in that case maybe just enable it for Python 2 since it's on by default and Python 2 installs a python executable.
| output=str, error=str) | ||
|
|
||
| # check for shared variant | ||
| # defualt is on, so we check for the string to turn it off |
There was a problem hiding this comment.
The default for this variant in this package is on, but ./configure --help in the cpython repo produces:
--enable-shared enable building a shared Python library (default is
no)
I think the right way to do these checks is to comb through all the variables and avoid using CONFIG_ARGS. For this one, it would be sysconfig.get_config_var('Py_ENABLE_SHARED'), which returns 1 or 0.
There was a problem hiding this comment.
It looks like RUNSHARED is stable across py2 and py3, and sysconfig.get_config_var('RUNSHARED') will return a non-empty string if the python executable has a shared library. Note that the error message from the clingo cmake build if you provide a python executable without a shared library is very difficult to parse -- it just says Python_LIBRARIES was not found, even if you define it on the same command line.
There was a problem hiding this comment.
@cosmicexplorer I've pushed an improved version of this, let me know what you think.
| output=str, error=str) | ||
|
|
||
| libdir = python( | ||
| '-c', '%s; print(sysconfig.get_config_var("LIBPL"))' % sysconf_imp, |
There was a problem hiding this comment.
I think we would want to get the directory containing LIBPL?
There was a problem hiding this comment.
Using os.path.join(sysconfig.get_config_var('LIBPL'), '../..') to get the location of libdir is working on my osx and linux machines.
There was a problem hiding this comment.
I don't think so. LIBPL is the directory containing libraries, config headers, etc. It's described in the code comment as "Library and miscellaneous stuff needed for extending/embedding".
There was a problem hiding this comment.
Ah. Well it does not appear to contain libpython3.so, just libpythonX.Y.a (on python 2 and 3). Are you also seeing this behavior?
There was a problem hiding this comment.
buuuuuuuuut, it looks like gcc -shared -o libpythonX.Y.so libpythonX.Y.a does work, so we could perhaps replace it with that? Do libs in spack refer to both static and shared libraries?
There was a problem hiding this comment.
It looks like libpythonX.y.a will work as well to build the clingo python module (just checked)! Wonderful.
There was a problem hiding this comment.
I think we can also change the libs() method:
@property
def libs(self):
return find_libraries('libpython*', root=self.get_config_var('LIBPL'), shared=False)
alalazo
left a comment
There was a problem hiding this comment.
LGTM, modulo the typo in the comment.
| '-c', '%s; print(sysconfig.get_config_var("LIBPL"))' % sysconf_imp, | ||
| output=str, error=str).strip() | ||
| # Search for shared libraries in the prefix and set shared variant | ||
| matches = fs.find_libraries('libpython*', prefix=libdir, shared=True) |
There was a problem hiding this comment.
| matches = fs.find_libraries('libpython*', prefix=libdir, shared=True) | |
| matches = fs.find_libraries('libpython*', libdir, shared=True) |
|
@becker33 I'm removing this from the point release, since it's not crucial for bootstrapping. I'm willing to review it again though and merge it to develop if you have any chance to update the PR. |
|
@becker33 did you get a chance to take a look at the review feedback. Is this PR still something you'd like to pursue? |
|
Closing this PR -- I will probably reopen at some point, but the contents will be substantially different since we no longer support python2 |
@adamjstewart @alalazo @tgamblin @cosmicexplorer
Depending on our eventual strategy to bootstrap clingo, we may need to be able to detect a fully concrete spec for python without using the concretizer. This is a step in that direction.
As the comment says, the
libxml2variant for python should go away as we add cycle-detection into the new concretizer and deprecate the old.