Drop Python 2 support #31824
Replies: 6 comments 22 replies
-
|
My 2 cents are: do it. Running a yum command to get a newer version of Python is not that painful. And, as long as you have a mechanism for users to use the older version, they can. Unless there are users who are supporting the development burdens of EoL Python development with huge sums of money/resources dedicated to the project that justify the pains of compatibility with legacy software. IIUC, the only folks still on RHEL 7 are institutional users who have a support contract with RedHat, either directly or through a intermediary. I’ve said this in various places, but OSS maintainers (including packaging tooling) shouldn’t feel obliged to maintain support beyond EoL if they’re not getting paid sufficiently for supporting it. And, most aren’t. Even for those that are, it still rarely makes sense to try to match RHEL’s support cycle — RedHat has both the scale and scope to employ a small full-time team for this (it’s small teams, even for foundational pieces like Python or gcc!) and, even then, they only have to support one Python version or two per RHEL version which they directly control and patch. And… a huge chunk of the Python ecosystem moved in sync with CPython on dropping Python 3.6 support, when it hit EoL within CPython’s workflow. I won’t be surprised if the same happens for 3.7. |
Beta Was this translation helpful? Give feedback.
-
|
FWIW, the bring your own python thing is what homebrew does with Ruby in many cases already. Also, thought on the packages split thing... I ran into this the other day and started thinking how convenient it might be for things like spack, or raja-framework: https://github.com/josh-project/josh It's a proxy that provides a standard git endpoint that lets you subset bits of a monorepo as though they were completely separate repositories, completely transparent to git clients in both directions. Keep everything in one repo, but say just clone spack core as a whole, and packages as a shallow depth 1, for example. Maybe not worth it, but got the gears spinning for sure. |
Beta Was this translation helpful? Give feedback.
-
|
Not a new idea for me! https://syspack.github.io/pakages/ |
Beta Was this translation helpful? Give feedback.
-
|
Talking about 1. (which version we should use to run Spack), here's a PR that makes me want to have support for Python 3.6+: Why do we need to vendor old backports to import pathlib? |
Beta Was this translation helpful? Give feedback.
-
|
Another kind of issues that will disappear when we'll move to Python 3 only: CI failures on Python 2.7 due to UTF-8 characters that are not decoded correctly. See #32608, which caused one job in CI to fail consistently. |
Beta Was this translation helpful? Give feedback.
-
|
Another commit that caused issues on Python 2.7 due to UTF-8 characters 71434d8 |
Beta Was this translation helpful? Give feedback.
-
|
Update for 1: Spack no longer supports running Spack with Python 3.5: #31908 |
Beta Was this translation helpful? Give feedback.
-
|
Update for 1 and 2: Python 2 support will be officially deprecated once #32615 is merged. |
Beta Was this translation helpful? Give feedback.
-
|
Regarding (2) I think installing Python 2.7 with Spack 0.18/0.19 (according to #32615 (comment)) is a good solution. I want to consider whether there is an easy path for users that need to install a mixture of packages needing Python 2.7 and newer packages that may only be available in a later Spack version (i.e. as new packages are added to Spack > 0.19). To do that, a user could install Python 2.7 with Spack 0.19 and then create a manifest .json file (or externals in Another option is to improve support for switching Spack versions within the same Spack instance like: (right now, we cache repo details that generally require running |
Beta Was this translation helpful? Give feedback.
-
Another possible idea, if we have requests of that kind, would be to maintain a separate repository for Python 2.7 packages under the Spack organization. For an approach like this to be feasible we need at least:
Not sure this is the best approach, but if we have some strong case that comes up after deprecation this can be a way to mitigate the impact. |
Beta Was this translation helpful? Give feedback.
-
|
There was some discussion on this at today's meeting: https://github.com/spack/spack/wiki/Telcon%3A-2022-09-14. The main question I want to bring up here: Is the plan to also remove any packages that only build with Python 2.7? I think it would make sense to contact the maintainers of those packages (not as a prerequisite for making this change, but as part of it) |
Beta Was this translation helpful? Give feedback.
-
|
Yes, that's the plan. Since Python 2 is deprecated, they won't be able to install those packages without noticing that it's deprecated. When I deprecated Python 2 last time, I pinged all of the maintainers of those packages and everyone seemed fine with it. |
Beta Was this translation helpful? Give feedback.
-
|
Circling back on this 1 year later. The policy that was adopted after this discussion was: Regarding (1): Follow distro support cycle (currently Python 3.6+) However, the fact that (1) and (2) aren't in sync makes it difficult to support bootstrapping Spack (both clingo binaries and style dependencies). I would like to reopen this discussion to find a solution to this issue. ProposalI propose following the PSF support cycle (currently Python 3.8+) for all of Spack. Benefits of this:
For users on systems where Python 3.8+ is not installed, they can choose one of the following solutions:
If we release Spack on PyPI, it becomes easier for package managers like pip to only install versions of Spack that are compatible with the Python version being used. Alternatives
Feel free to suggest other alternatives! |
Beta Was this translation helpful? Give feedback.
-
What I fail to see is how removing support for (installing) older Python 3.x is reducing the maintenance burden for Python package maintainers. You'll get exactly as many review requests as before, and I just cannot imagine more than 1% of the pull requests have to do with support for Python 3.7. You never have to modify rules like depends_on("[email protected]:", when="@x:")they're frozen.
GHA is certainly not the bottleneck. What I'd really wanna avoid is Spack turning into Nix where you almost need a meta-package manager to figure out what version of Spack has what version of particular software. Spack really has this advantage over Nix, so let's be happy with that. I also really dread the idea of I would really want to be able to say to people: with spack you can just login to your 6 year old HPC cluster and run The suggestion of being on top of deprecating critical packages as soon as possible is not really what I imagine is good for Spack. |
Beta Was this translation helpful? Give feedback.
-
|
To take one example that really shows why I am not in favor of this "let's deprecate python 3.7 everywhere" sentiment (from the relevant PR):
I see absolutely no benefit in these types of changes. No "maintenance burden" type of argument applies here. It's just "let me remove any mention of python 3.7 for consistency". It makes no sense... |
Beta Was this translation helpful? Give feedback.
-
|
A more constructive alternative: Identify packages you find a pain to maintain for Python 3.7 (e.g. branching on 3.7 for specific build instructions, flags, giant patches, ...) and replace the ugliness locally with |
Beta Was this translation helpful? Give feedback.
-
*Python package maintainer It's not frozen rules I'm worried about, it's ensuring that every new package that gets added still supports Python versions released 7+ years ago. If your concern is solely about bootstrapping clingo, why don't we add a GHA test that ensures that Spack's dependencies can be bootstrapped? @alalazo tried adding one but it seems to only fail on develop, not on PRs that break bootstrapping. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
For anyone new to Spack development, you may be surprised to learn that Spack still supports Python 2 (both for running Spack, which is written in Python, and for installing Python libraries). However, Python 2 has reached EOL and is no longer supported by the PSF. This introduces both risks (Spack will happily install a version of Python that may contain known security vulnerabilities) and significant maintainer burden. I would like to once again open the can of worms that is dropping support for Python 2 (and EOL Python versions in general).
I would like to split this discussion into two directions:
These do not necessarily need to happen in tandem. When expressing opinions in favor of or against changes in Python version support, please make it clear which you mean.
1. Versions of Python that can be used to run Spack
At the moment, Spack supports Python 2.7 and 3.5–3.10. We only recently dropped Python 2.6 support in Spack 0.18 (#27256). I think we should accelerate this timeline and require at least Python 3.6+.
Many Spack developers (myself, @alalazo, @trws) have expressed interest in using Python language features introduced in modern Python. These include things like type hints (introduced in Python 3.5) and f-strings (introduced in Python 3.6). Most of our vendored dependencies can no longer be updated because they dropped Python 2 support years ago.
Why do we still support Python 2.7 for running Spack? Many HPC centers run clusters that use RHEL 7 which comes with Python 2.7 by default. Unfortunately, RHEL 7 is still supported until 2026. I don't think it's realistic for Spack to also continue to support Python 2.7 for that long.
What about users stuck on RHEL 7? There are two easy solutions:
yum install python3or use Spack 0.18. Users can even use Spack 0.18 to install Python 3, then upgrade to newer versions of Spack. I personally don't believe that this is a significant hurdle to Spack usage, as Spack already has a number of system prerequisites in addition to Python.2. Versions of Python that Spack can install
At the moment, Spack supports installing Python 2.7 and 3.1–3.10. I think we should follow PSF and only support Python 3.7+.
While working on #27798 and #31823, I've found it quite difficult to support both Python 2 as well as modern Python. I already tried deprecating EOL Python versions in #28003 but received pushback from a user (singular) who still relies on Python 2-only libraries.
There will always be a tradeoff between maintainability and reproducibility. The Python ecosystem is evolving so rapidly that it is becoming difficult to keep up while still also supporting Python 2.7 (released 12 years ago). I understand concerns about reproducibility, but there is an easy solution for this: use Spack 0.18. If users want to be able to install ancient versions of software, they can use ancient versions of Spack (okay, not yet ancient, but will be a decade from now when people are still requesting Python 2 support). Unless we allow deprecation and removal of ancient versions of Python, one of 2 things will happen: Spack will start to fall behind other package managers in terms of Python support, or I will go crazy trying to support everything.
In #31823, I introduce support for the
--config-settingsflag for pip. However, support for this flag was added in pip 22.1, which only supports Python 3.7+. This means that any package we addconfig_settingsto will only support Python 3.7+. Forpy-scipy, we will only add this flag to scipy 1.9+, which already only supports Python 3.7+, so there's no issue there. But for other packages which specify abuild-backendinpyproject.toml, this raises the question of which Python versions we are willing to support.This is not the only feature reliant on removing support for EOL Python versions. We currently use pip to install all Python libraries. However, pip is an entire package manager with support for many things we don't need or use in Spack. Instead of our current "package manager (pip) inside of a package manager (spack)" approach, I would love to use build and installer instead. However, these libraries also require Python 3.7+.
Comparison with other package managers
It's always good to look outside of our own bubble and see what other package managers are doing:
I dream of the day that Spack can join this list of popular package managers brave enough to abandon Python 2 support.
Relevant parties: @varioustoxins @scheibelp @tgamblin @skosukhin @pradyunsg
Beta Was this translation helpful? Give feedback.
All reactions