PythonPackage: install packages with pip by adamjstewart · Pull Request #27798 · spack/spack

adamjstewart · 2021-12-04T19:34:56Z

Motivation

Python's build system has rapidly changed over the years. Previously, all Python packages were built using:

$ python setup.py build
$ python setup.py install --root=...

However, there are several issues with this:

Direct invocation of setup.py is now deprecated
The introduction of pyproject.toml and setup.cfg means that some packages may not come with a setup.py script at all
We don't yet have good support for alternatives to setuptools like poetry/flit
Setuptools will automatically download missing dependencies, leading to packages that may not work on air-gapped systems
Does not support pre-compiled wheels (needed for proprietary software)

Implementation

This PR updates our PythonPackage build system to use pip to install packages instead of setup.py. This is non-trivial, since packages like pip/wheel/setuptools require pip/wheel/setuptools to build and install. Specifically, this PR includes:

External references

https://discuss.python.org/t/bootstrapping-a-specific-version-of-pip/12306
https://discuss.python.org/t/list-dependencies-of-a-package/12341
https://discuss.python.org/t/understanding-site-packages-directories/12959
pypa/setuptools#2813

adamjstewart · 2021-12-05T04:13:34Z

Design question: which dependencies should PythonPackage add by default?

Every package will require pip to install, but 90% of packages will also require:

setuptools to build the library, except for packages that use other build tools (poetry, flit, etc.)
wheel to provide the bdist_wheel command, except for packages that are already distributed as wheels

Note that pip silently replaces distutils with setuptools, so all distutils-based packages will need a new dep on setuptools. I don't think it would cause any problems to just always add these deps, but there are other options as well. We could make wheel a run-time dep of pip and build pip with wheel instead of building wheel with pip.

adamjstewart · 2021-12-05T18:57:15Z

Another option is to use build/installer instead of pip to install all of our Python packages. However, these libraries seem less stable and have significantly more dependencies, making bootstrapping more difficult.

adamjstewart · 2021-12-05T19:06:35Z

Isort builds fine for me, no idea why it would be failing to bootstrap in CI.

alalazo · 2021-12-06T15:27:48Z

Added myself as an assignee to keep track of the progress here 🙂

var/spack/repos/builtin/packages/python/package.py

scottwittenburg

This is looking good from the gitlab perspective, which is the only aspect I'm considering, but needs a couple changes to start building everything. Also, as you're likely aware, the generate job for this stack is failing to concretize, which is probably the first thing to address. Thanks for adding a stack @adamjstewart! 🚀

share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml

share/spack/gitlab/cloud_pipelines/stacks/python/spack.yaml

share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml

share/spack/gitlab/cloud_pipelines/stacks/python/spack.yaml

* Use pip to bootstrap pip * Bootstrap wheel from source * Update PythonPackage to install using pip * Update several packages * Add wheel as base class dep * Build phase no longer exists * Add py-poetry package, fix py-flit-core bootstrapping * Fix isort build * Clean up many more packages * Remove unused import * Fix unit tests * Don't directly run setup.py * Typo fix * Remove unused imports * Fix issues caught by CI * Remove custom setup.py file handling * Use PythonPackage for installing wheels * Remove custom phases in PythonPackages * Remove <phase>_args methods * Remove unused import * Fix various packages * Try to test Python packages directly in CI * Actually run the pipeline * Fix more packages * Fix mappings, fix packages * Fix dep version * Work around bug in concretizer * Various concretization fixes * Fix gitlab yaml, packages * Fix typo in gitlab yaml * Skip more packages that fail to concretize * Fix? jupyter ecosystem concretization issues * Solve Jupyter concretization issues * Prevent duplicate entries in PYTHONPATH * Skip fenics-dolfinx * Build fewer Python packages * Fix missing npm dep * Specify image * More package fixes * Add backends for every from-source package * Fix version arg * Remove GitLab CI stuff, add py-installer package * Remove test deps, re-add install_options * Function declaration syntax fix * More build fixes * Update spack create template * Update PythonPackage documentation * Fix documentation build * Fix unit tests * Remove pip flag added only in newer pip * flux: add explicit dependency on jsonschema * Update packages that have been added since this was branched off of develop * Move Python 2 deprecation to a separate PR * py-neurolab: add build dep on py-setuptools * Use wheels for pip/wheel * Allow use of pre-installed pip for external Python * pip -> python -m pip * Use python -m pip for all packages * Fix py-wrapt * Add both platlib and purelib to PYTHONPATH * py-pyyaml: setuptools is needed for all versions * py-pyyaml: link flags aren't needed * Appease spack audit packages * Some build backend is required for all versions, distutils -> setuptools * Correctly handle different setup.py filename * Use wheels for py-tomli to avoid circular dep on py-flit-core * Fix busco installation procedure * Clarify things in spack create template * Test other Python build backends * Undo changes to busco * Various fixes * Don't test other backends

flxmr · 2022-02-26T11:58:15Z

@adamjstewart @alalazo

So this makes pip not working in environments anymore: #29237 (technically on vacation now, so my response time might vary)

EDIT: my other problem is also solved that way. I don't know if it's this thing specifically, but as I didn't find any other changes, it seems sensible to blame.

* Use pip to bootstrap pip * Bootstrap wheel from source * Update PythonPackage to install using pip * Update several packages * Add wheel as base class dep * Build phase no longer exists * Add py-poetry package, fix py-flit-core bootstrapping * Fix isort build * Clean up many more packages * Remove unused import * Fix unit tests * Don't directly run setup.py * Typo fix * Remove unused imports * Fix issues caught by CI * Remove custom setup.py file handling * Use PythonPackage for installing wheels * Remove custom phases in PythonPackages * Remove <phase>_args methods * Remove unused import * Fix various packages * Try to test Python packages directly in CI * Actually run the pipeline * Fix more packages * Fix mappings, fix packages * Fix dep version * Work around bug in concretizer * Various concretization fixes * Fix gitlab yaml, packages * Fix typo in gitlab yaml * Skip more packages that fail to concretize * Fix? jupyter ecosystem concretization issues * Solve Jupyter concretization issues * Prevent duplicate entries in PYTHONPATH * Skip fenics-dolfinx * Build fewer Python packages * Fix missing npm dep * Specify image * More package fixes * Add backends for every from-source package * Fix version arg * Remove GitLab CI stuff, add py-installer package * Remove test deps, re-add install_options * Function declaration syntax fix * More build fixes * Update spack create template * Update PythonPackage documentation * Fix documentation build * Fix unit tests * Remove pip flag added only in newer pip * flux: add explicit dependency on jsonschema * Update packages that have been added since this was branched off of develop * Move Python 2 deprecation to a separate PR * py-neurolab: add build dep on py-setuptools * Use wheels for pip/wheel * Allow use of pre-installed pip for external Python * pip -> python -m pip * Use python -m pip for all packages * Fix py-wrapt * Add both platlib and purelib to PYTHONPATH * py-pyyaml: setuptools is needed for all versions * py-pyyaml: link flags aren't needed * Appease spack audit packages * Some build backend is required for all versions, distutils -> setuptools * Correctly handle different setup.py filename * Use wheels for py-tomli to avoid circular dep on py-flit-core * Fix busco installation procedure * Clarify things in spack create template * Test other Python build backends * Undo changes to busco * Various fixes * Don't test other backends

adamjstewart requested a review from tgamblin December 4, 2021 19:34

spackbot-app bot added conflicts dependencies extends new-version python resources update-package labels Dec 4, 2021

spackbot-app bot added the build-systems label Dec 5, 2021

alalazo self-assigned this Dec 6, 2021

adamjstewart commented Dec 6, 2021

View reviewed changes

var/spack/repos/builtin/packages/python/package.py Outdated Show resolved Hide resolved

adamjstewart mentioned this pull request Dec 7, 2021

Concretizer chooses build dep that doesn't satisfy all packages #27835

Closed

3 tasks

scottwittenburg previously requested changes Dec 7, 2021

View reviewed changes

scottwittenburg reviewed Dec 7, 2021

View reviewed changes

share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

scottwittenburg reviewed Dec 9, 2021

View reviewed changes

share/spack/gitlab/cloud_pipelines/stacks/python/spack.yaml Outdated Show resolved Hide resolved

jbigot mentioned this pull request Feb 3, 2022

Installation issue: py-mpi4py #28748

Closed

4 tasks

adamjstewart mentioned this pull request Feb 3, 2022

py-mpi4py: fix install_options #28759

Merged

This was referenced Feb 26, 2022

Installation issue: py-pip in environments not working... #29237

Closed

CUDA not found/linked building pytorch inside a spack environment #29206

Closed

climbfuji mentioned this pull request Mar 2, 2022

Installation issue: py-pkgconfig needs py-poetry, not py-poetry-core? #29308

Closed

4 tasks

iarspider mentioned this pull request Mar 21, 2022

Fix py-onnx-runtime recipe #29646

Merged

adamjstewart mentioned this pull request Jul 15, 2022

Add CI stack for ML packages #31592

Merged

tkameyama mentioned this pull request Aug 15, 2022

py-scipy: Fortran compiler specify code is change to setup.cfg for Fujitsu compiler #32142

Merged

snehring mentioned this pull request Aug 17, 2022

Update busco to 5.4.3 #32221

Merged

adamjstewart mentioned this pull request Nov 25, 2022

Fix recipe for py-onnx-runtime #34130

Merged

G-071 mentioned this pull request Jul 25, 2023

sgpp: fix build issues #39086

Merged

manuelakuhn mentioned this pull request Sep 28, 2023

py-isort: needs setuptools build dep before v5 #40234

Merged

fsimonis mentioned this pull request Sep 4, 2024

Spack testing obsolete precice/python-bindings#220

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PythonPackage: install packages with pip#27798

PythonPackage: install packages with pip#27798
adamjstewart merged 72 commits intospack:developfrom
adamjstewart:packages/pip-bootstrap

adamjstewart commented Dec 4, 2021 •

edited

Loading

Uh oh!

adamjstewart commented Dec 5, 2021 •

edited

Loading

Uh oh!

adamjstewart commented Dec 5, 2021

Uh oh!

adamjstewart commented Dec 5, 2021

Uh oh!

alalazo commented Dec 6, 2021

Uh oh!

Uh oh!

scottwittenburg left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

flxmr commented Feb 26, 2022 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

adamjstewart commented Dec 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Implementation

External references

Uh oh!

adamjstewart commented Dec 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamjstewart commented Dec 5, 2021

Uh oh!

adamjstewart commented Dec 5, 2021

Uh oh!

alalazo commented Dec 6, 2021

Uh oh!

Uh oh!

scottwittenburg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

flxmr commented Feb 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

adamjstewart commented Dec 4, 2021 •

edited

Loading

adamjstewart commented Dec 5, 2021 •

edited

Loading

flxmr commented Feb 26, 2022 •

edited

Loading