Deprecate Python 2 support by adamjstewart · Pull Request #32615 · spack/spack

adamjstewart · 2022-09-12T17:50:20Z

This PR deprecates using Spack to install EOL Python versions, including Python 2.7, 3.1–3.6. It also deprecates running Spack with Python 2.7. Going forward, we expect Spack to have the following Python support timeline.

Running Spack

Spack 0.18 (spring 2022): Python 2.7, 3.5–3.10
Spack 0.19 (fall 2022): Python 2.7 (deprecated), 3.6–3.11
Spack 0.20 (spring 2023): Python 3.6–3.11

Building with Spack

Spack 0.18 (spring 2022): Python 2.7, 3.1–3.10
Spack 0.19 (fall 2022): Python 2.7, 3.1–3.6 (deprecated), 3.7–3.11
Spack 0.20 (spring 2023): Python 3.7–3.11

This is a reboot of #28003. See #31824 for a detailed discussion of the motivation for this PR. If you have concerns about this change, please comment on #31824.

adamjstewart · 2022-09-12T17:50:29Z

@spackbot fix style

spackbot-app · 2022-09-12T17:50:32Z

Let me see if I can fix that for you!

adamjstewart · 2022-09-12T17:51:41Z

var/spack/repos/builtin/packages/python/package.py

    list_depth = 1

-    maintainers = ["adamjstewart", "skosukhin", "scheibelp", "varioustoxins", "pradyunsg"]
+    maintainers = ["adamjstewart", "skosukhin", "scheibelp", "pradyunsg"]


@varioustoxins if I remember correctly, your willingness to help maintain this build recipe was contingent on Spack having Python 2 support. I removed you since we're planning on removing Python 2 support, but let me know if you still want to be listed as a maintainer.

spackbot-app · 2022-09-12T17:52:21Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, mypy, black, flake8
==> Modified files
  var/spack/repos/builtin/packages/python/package.py
==> Running isort checks
  isort checks were clean
==> Running mypy checks
Success: no issues found in 558 source files
  mypy checks were clean
==> Running black checks
reformatted var/spack/repos/builtin/packages/python/package.py
All done! ✨ 🍰 ✨
1 file reformatted.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> spack style checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with isort fixes.

tldahlgren · 2022-09-12T18:10:00Z

@tgamblin FYI. Since you mentioned a deprecation plan in a meeting earlier today.

adamjstewart · 2022-09-18T15:07:46Z

@scottwittenburg how can I tell why e4s-pr-generate is failing?

scottwittenburg · 2022-09-19T15:53:32Z

@scottwittenburg how can I tell why e4s-pr-generate is failing?

Reading the trace of that job, it seems there's a known broken spec on develop:

==> Error: Pipeline generation failed due to the presence of the following specs that are known to be broken in develop:
* [email protected]%[email protected]...

I think @zackgalbreath has been tracking down a fix for that spec to get develop pipelines passing again.

zackgalbreath · 2022-09-19T16:53:03Z

@johnwparent deserves the credit here. The fix has been merged and develop pipelines are once again passing on GitLab CI.

adamjstewart · 2022-09-19T18:09:10Z

@spackbot run pipeline

spackbot-app · 2022-09-19T18:09:13Z

I've started that pipeline for you!

adamjstewart · 2022-09-19T20:31:49Z

Yay, passing now!

alalazo · 2022-09-26T08:17:51Z

This is one of those cases where rebuilding all pipelines would have helped. At least this pipeline: https://gitlab.spack.io/spack/spack/-/pipelines/172567 is broken (we'll see the breakage on first rebuild) because it needs [email protected]. I noticed that just because of an unrelated PR: https://gitlab.spack.io/spack/spack/-/jobs/3370750

alalazo · 2022-09-26T08:47:27Z

It seems the issue is due to the Waf (?) build-system. We have 3 packages using that, out of 6k. One of them already moved to CMake and was not updated.

I'll be removing that build-system from the ones checked in pipelines. Would people agree on removing Waf entirely as a build-system (e.g. by updating or removing packages that rely on it)?

PR spack#32615 deprecated Python versions up to 3.6.X. Since the "build-systems" pipeline requires Python 3.6.15 to build "tut", it will fail on the first rebuild that involves Python. The "tut" package is meant to perform an end-to-end test of the "Waf" build-system, which is scarcely used. The fix therefore is just to remove it from the pipeline.

PR #32615 deprecated Python versions up to 3.6.X. Since the "build-systems" pipeline requires Python 3.6.15 to build "tut", it will fail on the first rebuild that involves Python. The "tut" package is meant to perform an end-to-end test of the "Waf" build-system, which is scarcely used. The fix therefore is just to remove it from the pipeline.

adamjstewart · 2022-09-26T13:30:12Z

I don't see any reason to remove it entirely. We could test a different package if you're concerned about untested code.

alalazo · 2022-09-26T13:50:44Z

We could test a different package if you're concerned about untested code.

I'm slightly more concerned about maintaining a build-system for just 3 packages. Wondering if we can use another build-system for them. The three packages are:

ns3dev (https://gitlab.com/nsnam/ns-3-dev already using CMake)
py-py2cairo (https://www.cairographics.org/pycairo/ legacy resource for py-pycairo on Python 2)
tut (https://github.com/mrzechonek/tut-framework/ unit-test framework for C++, I guess not widely used)

I think the first can move to CMake. The second will disappear when we remove Python 2 packages. For the third we are shipping a version from 2016, so I guess not many people are using it.

Not a high-priority thing, but if people agree, it wouldn't take a lot of time to remove the build-system and convert the packages.

adamjstewart · 2022-09-26T14:48:27Z

n3-3-dev: maybe @yee29 can comment, but I imagine we can update this to use CMake
py-py2cairo: didn't even realize this was for Python 2, I'll mark it as such
tut: requires Python 3.6 and older, so this will also be removed in the next release

So yeah, I'm fine with dropping WafPackage once ns-3-dev is converted and py-py2cairo and tut are removed. See https://waf.io/projects_using_waf.html for a list of other projects using Waf, not sure if we ever plan on adding any of these.

This PR removes [end of life](https://endoflife.date/python) versions of Python from Spack. Specifically, this includes all versions of Python older than 3.7. See #31824 for rationale. Deprecated in #32615. And #28003. For anyone using software that relies on Python 2, you have a few options: * Upgrade the software to support Python 3. The `3to2` tool may get you most of the way there, although more complex libraries may need manual tweaking. * Add Python 2 as an [external package](https://spack.readthedocs.io/en/latest/build_settings.html#external-packages). Many Python libraries do not support Python 2, but you may be able to add older versions that did once upon a time. * Use Spack 0.19. Spack 0.19 is the last release to officially support Python 3.6 and older * Create and maintain your own [custom repository](https://spack.readthedocs.io/en/latest/repositories.html). Basically, you would need a package for Python 2 and any other Python 2-specific libraries you need.

This PR removes [end of life](https://endoflife.date/python) versions of Python from Spack. Specifically, this includes all versions of Python older than 3.7. See spack#31824 for rationale. Deprecated in spack#32615. And spack#28003. For anyone using software that relies on Python 2, you have a few options: * Upgrade the software to support Python 3. The `3to2` tool may get you most of the way there, although more complex libraries may need manual tweaking. * Add Python 2 as an [external package](https://spack.readthedocs.io/en/latest/build_settings.html#external-packages). Many Python libraries do not support Python 2, but you may be able to add older versions that did once upon a time. * Use Spack 0.19. Spack 0.19 is the last release to officially support Python 3.6 and older * Create and maintain your own [custom repository](https://spack.readthedocs.io/en/latest/repositories.html). Basically, you would need a package for Python 2 and any other Python 2-specific libraries you need.

spackbot-app bot added maintainers new-version python update-package labels Sep 12, 2022

spackbot-app bot requested review from pradyunsg, scheibelp, skosukhin and varioustoxins September 12, 2022 17:50

adamjstewart commented Sep 12, 2022

View reviewed changes

adamjstewart force-pushed the packages/python2 branch 3 times, most recently from 1a527b7 to 4c5e6f3 Compare September 17, 2022 16:09

adamjstewart and others added 2 commits September 18, 2022 11:19

Deprecate Python 2 support

370708b

[@spackbot] updating style on behalf of adamjstewart

2ec1e1e

adamjstewart force-pushed the packages/python2 branch from 4c5e6f3 to 2ec1e1e Compare September 18, 2022 16:19

tgamblin approved these changes Sep 19, 2022

View reviewed changes

tgamblin merged commit 83e66ce into spack:develop Sep 19, 2022

adamjstewart deleted the packages/python2 branch September 19, 2022 20:35

wyphan mentioned this pull request Sep 21, 2022

Do not export SPACK_PYTHON when sourcing setup_env.sh #32747

Closed

2 tasks

alalazo mentioned this pull request Sep 26, 2022

Remove "tut" from the "build-systems" pipeline #32808

Merged

2 tasks

adamjstewart mentioned this pull request Nov 14, 2022

Python: drop EOL versions #33898

Merged

Conversation

adamjstewart commented Sep 12, 2022

Running Spack

Building with Spack

Uh oh!

adamjstewart commented Sep 12, 2022

Uh oh!

spackbot-app bot commented Sep 12, 2022

Uh oh!

adamjstewart Sep 12, 2022

Choose a reason for hiding this comment

Uh oh!

spackbot-app bot commented Sep 12, 2022

Uh oh!

tldahlgren commented Sep 12, 2022

Uh oh!

adamjstewart commented Sep 18, 2022

Uh oh!

scottwittenburg commented Sep 19, 2022

Uh oh!

zackgalbreath commented Sep 19, 2022

Uh oh!

adamjstewart commented Sep 19, 2022

Uh oh!

spackbot-app bot commented Sep 19, 2022

Uh oh!

adamjstewart commented Sep 19, 2022

Uh oh!

alalazo commented Sep 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alalazo commented Sep 26, 2022

Uh oh!

adamjstewart commented Sep 26, 2022

Uh oh!

alalazo commented Sep 26, 2022

Uh oh!

adamjstewart commented Sep 26, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

alalazo commented Sep 26, 2022 •

edited

Loading