Skip to content

python: improved external detection for obscure variants#20116

Closed
becker33 wants to merge 5 commits intodevelopfrom
features/improved-python-detection
Closed

python: improved external detection for obscure variants#20116
becker33 wants to merge 5 commits intodevelopfrom
features/improved-python-detection

Conversation

@becker33
Copy link
Copy Markdown
Member

@adamjstewart @alalazo @tgamblin @cosmicexplorer

Depending on our eventual strategy to bootstrap clingo, we may need to be able to detect a fully concrete spec for python without using the concretizer. This is a step in that direction.

As the comment says, the libxml2 variant for python should go away as we add cycle-detection into the new concretizer and deprecate the old.

Copy link
Copy Markdown
Member

@adamjstewart adamjstewart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +308 to +315
# Get config args for shared, debug, and optimizations variant checks
sysconf_imp = 'import sysconfig'
if version < Version('2.7'):
sysconf_imp = 'import distutils.sysconfig as sysconfig'
conf_args = python(
'-c',
'%s; print(sysconfig.get_config_var("CONFIG_ARGS"))' % sysconf_imp,
output=str, error=str)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See self.get_config_var. It's hard-coded to use self.command, but maybe we could modify it so there is less overlap. At the very least, this print syntax won't work for older versions of Python. That's why we use self.print_string elsewhere.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought as long as you were printing a single value, print(foo) and print foo were the same at least as far back as 2.6. I only have 2.7 around to confirm on though. Do you have easy access to 2.6 or should I build it to test?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess they are the same, as (foo) will be treated as foo.

output=str, error=str)
variants += '~ucs4' if maxunicode == 0xFFFF else '+ucs4'

# +pythoncommand does not apply to python@2
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally, I don't think we should set this variant for Python 2 at all.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have to set the variant, we don't (yet) have a mechanism to have the concretizer omit a variant for certain versions. ucs4 has a similar issue, where in that case we raise exceptions if we try to build with it at certain versions.

We can either set it to the default in the detection logic, or we can let the concretizer fill it in. The latter would be appealing except for the fact that it looks like we're going to need to swap the python that Spack is run with into an already concrete spec as part of our bootstrapping process (to know which binary to download for clingo/libstdc++). That means we need to have all the variants filled out by the detection logic.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, in that case maybe just enable it for Python 2 since it's on by default and Python 2 installs a python executable.

output=str, error=str)

# check for shared variant
# defualt is on, so we check for the string to turn it off
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default for this variant in this package is on, but ./configure --help in the cpython repo produces:

  --enable-shared         enable building a shared Python library (default is
                          no)

I think the right way to do these checks is to comb through all the variables and avoid using CONFIG_ARGS. For this one, it would be sysconfig.get_config_var('Py_ENABLE_SHARED'), which returns 1 or 0.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like RUNSHARED is stable across py2 and py3, and sysconfig.get_config_var('RUNSHARED') will return a non-empty string if the python executable has a shared library. Note that the error message from the clingo cmake build if you provide a python executable without a shared library is very difficult to parse -- it just says Python_LIBRARIES was not found, even if you define it on the same command line.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cosmicexplorer I've pushed an improved version of this, let me know what you think.

output=str, error=str)

libdir = python(
'-c', '%s; print(sysconfig.get_config_var("LIBPL"))' % sysconf_imp,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we would want to get the directory containing LIBPL?

Copy link
Copy Markdown
Contributor

@cosmicexplorer cosmicexplorer Dec 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using os.path.join(sysconfig.get_config_var('LIBPL'), '../..') to get the location of libdir is working on my osx and linux machines.

Copy link
Copy Markdown
Member Author

@becker33 becker33 Dec 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so. LIBPL is the directory containing libraries, config headers, etc. It's described in the code comment as "Library and miscellaneous stuff needed for extending/embedding".

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. Well it does not appear to contain libpython3.so, just libpythonX.Y.a (on python 2 and 3). Are you also seeing this behavior?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

buuuuuuuuut, it looks like gcc -shared -o libpythonX.Y.so libpythonX.Y.a does work, so we could perhaps replace it with that? Do libs in spack refer to both static and shared libraries?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like libpythonX.y.a will work as well to build the clingo python module (just checked)! Wonderful.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can also change the libs() method:

    @property
    def libs(self):
        return find_libraries('libpython*', root=self.get_config_var('LIBPL'), shared=False)

Copy link
Copy Markdown
Member

@alalazo alalazo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, modulo the typo in the comment.

'-c', '%s; print(sysconfig.get_config_var("LIBPL"))' % sysconf_imp,
output=str, error=str).strip()
# Search for shared libraries in the prefix and set shared variant
matches = fs.find_libraries('libpython*', prefix=libdir, shared=True)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
matches = fs.find_libraries('libpython*', prefix=libdir, shared=True)
matches = fs.find_libraries('libpython*', libdir, shared=True)

@alalazo
Copy link
Copy Markdown
Member

alalazo commented Dec 15, 2021

@becker33 I'm removing this from the point release, since it's not crucial for bootstrapping. I'm willing to review it again though and merge it to develop if you have any chance to update the PR.

@alecbcs
Copy link
Copy Markdown
Member

alecbcs commented Nov 19, 2023

@becker33 did you get a chance to take a look at the review feedback. Is this PR still something you'd like to pursue?

@becker33
Copy link
Copy Markdown
Member Author

becker33 commented Mar 6, 2024

Closing this PR -- I will probably reopen at some point, but the contents will be substantially different since we no longer support python2

@becker33 becker33 closed this Mar 6, 2024
@haampie haampie deleted the features/improved-python-detection branch July 9, 2025 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants