handle distutils deprecation by dimbleby · Pull Request #7766 · python-poetry/poetry

dimbleby · 2023-04-04T19:57:07Z

distutils is going away in python 3.12.

I don't find this comment very clear

# We can't use sysconfig.get_paths() because
# on some distributions it does not return the proper paths
# (those used by pip for instance).

but it was written three years ago now, we can always hope that things have moved on far enough to make it no longer a thing that needs to be worried about.

Well anyway let's start by seeing how the pipelines like it if we don't use distutils...

dimbleby · 2023-04-04T20:14:54Z

probably the comment is a reference to debian cf #6459 (comment)

on the other hand that open issue seems to say that things already don't work on such distributions unless you create a virtual environment - in which case this wouldn't make things worse anyway?

radoering · 2023-04-25T16:03:16Z

I did some basic testing on Ubuntu 20.04:

Test script

import site
import sysconfig
import warnings

from pprint import pprint

orig_paths = sysconfig.get_paths().copy()
if site.check_enableusersite():
  orig_paths["usersite"] = site.getusersitepackages()
  orig_paths["userbase"] = site.getuserbase()
print("sys_config.get_paths():")
pprint(orig_paths)
paths = orig_paths.copy()

from distutils.core import Distribution
from distutils.command.install import SCHEME_KEYS

d = Distribution()
d.parse_config_files()
with warnings.catch_warnings():
  warnings.filterwarnings("ignore", "setup.py install is deprecated")
  obj = d.get_command_obj("install", create=True)
obj.finalize_options()
for key in SCHEME_KEYS:
  if key == "headers":
      # headers is not a path returned by sysconfig.get_paths()
      continue

  paths[key] = getattr(obj, f"install_{key}")

if site.check_enableusersite() and hasattr(obj, "install_usersite"):
  paths["usersite"] = getattr(obj, "install_usersite")
  paths["userbase"] = getattr(obj, "install_userbase")

print("distutils:")
pprint(paths)

print(f"Equal? {paths == orig_paths}")

Python 3.8: ❌ (not equal)

sys_config.get_paths():
{'data': '/usr',ers for dbus (1.12.16-2ubuntu2.3) ...
'include': '/usr/include/python3.8',ls (0.136ubuntu6.7) ...
'platinclude': '/usr/include/python3.8',
'platlib': '/usr/lib/python3.8/site-packages',
'platstdlib': '/usr/lib/python3.8',
'purelib': '/usr/lib/python3.8/site-packages',
'scripts': '/usr/bin',
'stdlib': '/usr/lib/python3.8',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.8/site-packages'}
distutils:
{'data': '/usr/local',
'include': '/usr/include/python3.8',
'platinclude': '/usr/include/python3.8',
'platlib': '/usr/local/lib/python3.8/dist-packages',
'platstdlib': '/usr/lib/python3.8',
'purelib': '/usr/local/lib/python3.8/dist-packages',
'scripts': '/usr/local/bin',
'stdlib': '/usr/lib/python3.8',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.8/site-packages'}
Equal? False

Python 3.9: ❌ (not equal)

sys_config.get_paths():
{'data': '/usr',
'include': '/usr/include/python3.9',
'platinclude': '/usr/include/python3.9',
'platlib': '/usr/lib/python3.9/site-packages',
'platstdlib': '/usr/lib/python3.9',
'purelib': '/usr/lib/python3.9/site-packages',
'scripts': '/usr/bin',
'stdlib': '/usr/lib/python3.9',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.9/site-packages'}
distutils:
{'data': '/usr/local',
'include': '/usr/include/python3.9',
'platinclude': '/usr/include/python3.9',
'platlib': '/usr/local/lib/python3.9/dist-packages',
'platstdlib': '/usr/lib/python3.9',
'purelib': '/usr/local/lib/python3.9/dist-packages',
'scripts': '/usr/local/bin',
'stdlib': '/usr/lib/python3.9',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.9/site-packages'}
Equal? False

Python 3.10: ✔ (equal)

sys_config.get_paths():
{'data': '/usr/local',
'include': '/usr/include/python3.10',
'platinclude': '/usr/include/python3.10',
'platlib': '/usr/local/lib/python3.10/dist-packages',
'platstdlib': '/usr/lib/python3.10',
'purelib': '/usr/local/lib/python3.10/dist-packages',
'scripts': '/usr/local/bin',
'stdlib': '/usr/lib/python3.10',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.10/site-packages'}
/mnt/c/dev/poetry/sysconfig_paths.py:15: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils.core import Distribution
distutils:
{'data': '/usr/local',
'include': '/usr/include/python3.10',
'platinclude': '/usr/include/python3.10',
'platlib': '/usr/local/lib/python3.10/dist-packages',
'platstdlib': '/usr/lib/python3.10',
'purelib': '/usr/local/lib/python3.10/dist-packages',
'scripts': '/usr/local/bin',
'stdlib': '/usr/lib/python3.10',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.10/site-packages'}
Equal? True

Python 3.11: ✔ (equal)

sys_config.get_paths():
{'data': '/usr/local',
'include': '/usr/include/python3.11',
'platinclude': '/usr/include/python3.11',
'platlib': '/usr/local/lib/python3.11/dist-packages',
'platstdlib': '/usr/lib/python3.11',
'purelib': '/usr/local/lib/python3.11/dist-packages',
'scripts': '/usr/local/bin',
'stdlib': '/usr/lib/python3.11',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.11/site-packages'}
/mnt/c/dev/poetry/sysconfig_paths.py:15: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils.core import Distribution
distutils:
{'data': '/usr/local',
'include': '/usr/include/python3.11',
'platinclude': '/usr/include/python3.11',
'platlib': '/usr/local/lib/python3.11/dist-packages',
'platstdlib': '/usr/lib/python3.11',
'purelib': '/usr/local/lib/python3.11/dist-packages',
'scripts': '/usr/local/bin',
'stdlib': '/usr/lib/python3.11',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.11/site-packages'}
Equal? True

Python 3.9 venv: ✔ (equal)

sys_config.get_paths():
{'data': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9',
'include': '/usr/include/python3.9',
'platinclude': '/usr/include/python3.9',
'platlib': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/lib/python3.9/site-packages',
'platstdlib': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/lib/python3.9',
'purelib': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/lib/python3.9/site-packages',
'scripts': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/bin',
'stdlib': '/usr/lib/python3.9',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.9/site-packages'}
distutils:
{'data': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9',
'include': '/usr/include/python3.9',
'platinclude': '/usr/include/python3.9',
'platlib': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/lib/python3.9/site-packages',
'platstdlib': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/lib/python3.9',
'purelib': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/lib/python3.9/site-packages',
'scripts': '/home/randy/.cache/pypoetry/virtualenvs/poetry-gXMMKm0L-py3.9/bin',
'stdlib': '/usr/lib/python3.9',
'userbase': '/home/randy/.local',
'usersite': '/home/randy/.local/lib/python3.9/site-packages'}
Equal? True

on the other hand that open issue seems to say that things already don't work on such distributions unless you create a virtual environment - in which case this wouldn't make things worse anyway?

I agree and even if it worked I'd say it's a Debian/Ubuntu bug and the workaround is to create a virtual environment. I think I will check in the next maintainer meeting if there are other opinions and if we want to merge it now or later.

dimbleby · 2023-04-25T18:46:14Z

Indeed this has long been understood as a debian bug.

The muddiness, as I understand it, is that at some point the workarounds which I am removing in this MR did actually work. And then at some point they stopped working (which is where #6459 has got to).

So unconditionally removing them here is abandoning any pretence that poetry intends to repair those workarounds.

It would I suppose be possible, though awfully ugly, to (i) fix the workarounds and (ii) 'improve' them with "if python version less than 3.12" checks, or something like that. I don't intend to do any such thing myself, though of course if y'all think that's worthwhile I wouldn't - couldn't! - stand in your way.

Secrus · 2023-04-26T08:50:35Z

In my opinion, we should drop the special cases and just point people to 3.9-venv or 3.10+ solution.

github-actions · 2024-03-03T22:05:38Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

dimbleby mentioned this pull request Apr 16, 2023

python 3.12 in pipelines #7803

Merged

dimbleby force-pushed the distutils-deprecation branch 2 times, most recently from 101ea14 to c85a9bc Compare April 16, 2023 12:22

radoering force-pushed the distutils-deprecation branch 2 times, most recently from 250718b to 988410f Compare April 30, 2023 12:32

dimbleby added 2 commits April 30, 2023 14:32

handle distutils deprecation

495a595

more distutils deprecation

988410f

radoering approved these changes Apr 30, 2023

View reviewed changes

radoering enabled auto-merge (squash) April 30, 2023 12:32

radoering merged commit 3804a13 into python-poetry:master Apr 30, 2023

dimbleby deleted the distutils-deprecation branch April 30, 2023 13:24

dimbleby mentioned this pull request Apr 30, 2023

Poetry with virtualenvs.create false installs packages into the wrong place on Ubuntu #6459

Open

3 tasks

gridhead mentioned this pull request Sep 13, 2023

mdapi build in fedora prod openshift takes up lots of memory and never finishes fedora-infra/mdapi#212

Closed

github-actions bot locked as resolved and limited conversation to collaborators Mar 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

handle distutils deprecation#7766

handle distutils deprecation#7766
radoering merged 2 commits intopython-poetry:masterfrom
dimbleby:distutils-deprecation

dimbleby commented Apr 4, 2023

Uh oh!

dimbleby commented Apr 4, 2023 •

edited

Loading

Uh oh!

radoering commented Apr 25, 2023

Uh oh!

dimbleby commented Apr 25, 2023

Uh oh!

Secrus commented Apr 26, 2023

Uh oh!

github-actions bot commented Mar 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

dimbleby commented Apr 4, 2023

Uh oh!

dimbleby commented Apr 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

radoering commented Apr 25, 2023

Uh oh!

dimbleby commented Apr 25, 2023

Uh oh!

Secrus commented Apr 26, 2023

Uh oh!

github-actions bot commented Mar 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dimbleby commented Apr 4, 2023 •

edited

Loading