Skip to content

PEP 632: Proposed deprectation of distutils #7702

Closed

Description

@FRidh

There is a PEP proposed for deprecating distutils. Meson uses distutils in the python module for, if I am correct, building extension modules, so this PEP may be of interest to you.

Activity

  1. lazka commented on Sep 6, 2020

    @lazka
    Contributor

    distutils is only used to decide if extensions should link against libpython to follow the platform defaults (Debian patched to not link, default was to link). With Python 3.8 upstream Python default changed to not link to libpython by default, so if distutils ever gets removed, meson can just default to not linking always: https://docs.python.org/3/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

  2. eli-schwartz commented on Oct 3, 2021

    @eli-schwartz
    Member

    Note that in #9288 we added an additional reliance on distutils (in the same exact introspection command that already imports distutils). In addition to detecting whether to link against libpython, we now also use distutils to detect whether sysconfig is completely crippled by the vendor and unable to tell the truth about install paths.

    Handling libpython linking could be done by a try/except that reacts to an ImportError by saying "clearly this must be sys.version_info >= (3, 8) so the answer is don't link".

    Handling debian paths cannot be resolved other than by waiting on debian to fix things.

    As we get closer to the final removal of distutils in 3.12 we will need to both consider that try/except for libpython, and keep an eye on the debian situation.

  3. tacaswell commented on Nov 14, 2022

    @tacaswell

    distutils has now been pulled out of the main branch of CPython and meson is failing to detect it as a valid Python:

    $ meson --version
    0.64.0
    $ python --version
    Python 3.12.0a1+
    

    adapted from numpy's meson branch:

    project(
      'test',
      'c',
      version: '0',
      license: 'BSD-3',
      meson_version: '>= 0.64.0',
    )
    
    cc = meson.get_compiler('c')
    
    # https://mesonbuild.com/Python-module.html
    py_mod = import('python')
    py = py_mod.find_installation(pure: false)
    py_dep = py.dependency()
    
    
    $ meson setup build --prefix=/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages
    The Meson build system
    Version: 0.64.0
    Source dir: /home/tcaswell/source/p/numpy/numpy
    Build dir: /home/tcaswell/source/p/numpy/numpy/build
    Build type: native build
    Project name: test
    Project version: 0
    C compiler for the host machine: ccache cc (gcc 12.2.0 "cc (GCC) 12.2.0")
    C linker for the host machine: cc ld.bfd 2.39.0
    Host machine cpu family: x86_64
    Host machine cpu: x86_64
    
    meson.build:13:0: ERROR: <PythonExternalProgram 'python3' -> ['/home/tcaswell/.virtualenvs/bleeding/bin/python3']> is not a valid python or it is missing distutils
    
    A full log can be found at /home/tcaswell/source/p/numpy/numpy/build/meson-logs/meson-log.txt
    
  4. added this to the 1.0.0 milestone on Nov 14, 2022
  5. CAM-Gerlach commented on Nov 15, 2022

    @CAM-Gerlach

    FYI, there's been a migration among packaging tools to use the standalone, modern sysconfig stdlib module over distutils' equivalent, which mesonpy's @FFY00 has led the charge on. Perhaps he has some insightful comments on that here?

  6. eli-schwartz commented on Nov 15, 2022

    @eli-schwartz
    Member

    We already use sysconfig for most things. There is:

    • one thing that sysconfig doesn't currently provide -- I've been meaning to open a CPython ticket to have it added (I mentioned it in passing in a meson-python ticket somewhere) which I will try doing soon
    • one thing that sysconfig does provide, but Debian has frustratingly broken, which I need to investigate whether it works now.

    I'll try to see if I can get news on both counts in the near future. Because one of them is totally dependent on Debian's timeframe and they already know about the problem but have been flatly ignoring it for years, I've been letting the general issue slide, hoping that if I check in on it closer to the cutoff point it will finally resolve itself. But I didn't think about people using the alphas for 3.12 a year before the cutoff point. :)

  7. CAM-Gerlach commented on Nov 16, 2022

    @CAM-Gerlach

    I see, thanks. In that case, I'd suggest opening a CPython issue for the former as soon as possible (and the latter, if there's anything needed on the upstream end), to allow plenty of time before the beta freeze in less than 6 months for discussion, implementation, review and merging, and preferably allow you, distros and others to test with it.

  8. tacaswell commented on Nov 16, 2022

    @tacaswell

    The suggestion of installing a setuptools 60+ worked for getting meson to work with py312.

  9. dnicolodi commented on Nov 16, 2022

    @dnicolodi
    Member

    It seems that recent Debian patches sysconfig too now https://salsa.debian.org/cpython-team/python3/-/blob/master/debian/patches/sysconfig-debian-schemes.diff I'll have a look at when this was introduced. If it is old enough (how old?) we can probably rely on it. Otherwise we will need to make the checks conditional. I'll see what can be done.

    Edit: it looks like the sysconfig patching goes way back to Python 3.7 https://salsa.debian.org/cpython-team/python3/-/blob/python3.7/debian/patches/sysconfig-debian-schemes.diff I think we can probably safely rely on it. I'll prepare a PR and see what the CI tests think about it.

  10. dnicolodi commented on Nov 16, 2022

    @dnicolodi
    Member

    It would be great to have some input from the Debian Python maintainers on this.

  11. dnicolodi commented on Nov 16, 2022

    @dnicolodi
    Member

    I've done some more archeology. The sysconfig patch is in the repository for the Debian's Python package since a long time, but it is actually installed only starting with Python 3.10, in Debian testing. The only thing that can be done is to check the Python version ad fall back to distutils for Python versions earlier than 3.10.

    The other distutils is used for is this:

    def links_against_libpython():
    from distutils.core import Distribution, Extension
    cmd = Distribution().get_command_obj('build_ext')
    cmd.ensure_finalized()
    return bool(cmd.get_libraries(Extension('dummy', [])))
    for which there is no replacement AFAICT. @eli-schwartz Do you want me to open an issue with CPython for this?

  12. FFY00 commented on Nov 17, 2022

    @FFY00
    Member

    I've done some more archeology. The sysconfig patch is in the repository for the Debian's Python package since a long time, but it is actually installed only starting with Python 3.10, in Debian testing. The only thing that can be done is to check the Python version ad fall back to distutils for Python versions earlier than 3.10.

    See https://ffy00.github.io/blog/02-python-debian-and-the-install-locations/ if you want some more archeology 😛

    Do you want me to open an issue with CPython for this?

    python/cpython#88611 is the proposed fix.

  13. dnicolodi commented on Nov 17, 2022

    @dnicolodi
    Member

    IIUC, the linked issue is about where to find the headers, not about whether to link or not with libpython. Aren't these separate issues?

  14. FFY00 commented on Nov 17, 2022

    @FFY00
    Member

    Yes, sorry, I misread it.

  15. 22 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions