Skip to content

Make spack pip installable#50862

Draft
adamjstewart wants to merge 7 commits intospack:developfrom
adamjstewart:pip/install
Draft

Make spack pip installable#50862
adamjstewart wants to merge 7 commits intospack:developfrom
adamjstewart:pip/install

Conversation

@adamjstewart
Copy link
Copy Markdown
Member

@adamjstewart adamjstewart commented Jun 9, 2025

Follow-up to #32616 with newer package metadata
Closes #50238 without the custom shell scripts to maintain
Closes #1385 after a nice long wait

Pitch

One of our goals for Spack 1.0 (or shortly thereafter) is to make Spack as easy to install as possible. This PR does two things (which could be separated into separate PRs if desired):

  • Support pip install . for installation (by modifying pyproject.toml, may require additional changes)
  • Support pip install spack for installation (by adding continuous deployment to PyPI)

Usage

To test this PR, simply clone this branch and run:

python -m build

You can then see what gets placed in the sdist and bdist using:

tar tvzf dist/*.tar.gz
unzip -l dist/*.whl 

The package can be installed using:

pip install .

You can then run spack help and other commands to test things.

sdist

At the moment, every file that is tracked by git ends up in the sdist. We can and should minimize this to remove random CI files, metadata, and tests.

bdist

At the moment, the package will be installed with the following hierarchy:

<prefix>/bin/
    spack
<prefix>/lib/pythonX.Y/site-packages/
    spack/
        etc/

We need to decide what layout we plan to support, as this would currently require changing a lot of import paths to support.

Resources

For those unfamiliar with Python packaging, the following resources are useful:

@adamjstewart adamjstewart requested review from psakievich and trws June 9, 2025 12:28
@spackbot-app spackbot-app bot added core PR affects Spack core functionality workflow labels Jun 9, 2025
on:
release:
types:
- published
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a release is published on GitHub, the sdist and bdist will automatically be uploaded to PyPI.

TODO: set up trusted publishing: https://docs.pypi.org/trusted-publishers/

Comment on lines +9 to +15
authors = [
{name="Todd Gamblin", email="[email protected]"},
]
maintainers = [
{name="Todd Gamblin", email="[email protected]"},
{name="Adam J. Stewart", email="[email protected]"},
]
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can add more people here, just let me know who we want. Maybe authors come from the first Spack paper and maintainers are a subset of people with merge privs?

"hpsf"
]
classifiers = [
# https://pypi.org/classifiers/
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are just the first ones that came to mind, can refine

"pytest",
"pytest-xdist",
"setuptools",
"click",
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see anywhere in Spack where we use setuptools or click?


[build-system]
requires = ["hatchling"]
requires = ["hatchling>=1.26"]
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1.26+ needed for license_files support

@psakievich
Copy link
Copy Markdown
Contributor

@matthewcurry please review this. @tgamblin I can't add @matthewcurry as a reviewer

@matthewcurry
Copy link
Copy Markdown
Contributor

At first blush, this looks pretty clean. It doesn't work yet as Adam has alluded to, as the import paths don't match the current organization.

I'm not sure about the specifics of how this will work without wrenching around import paths in the main repo, since llnl, external, and spack are installed as peers under site-packages.

@adamjstewart
Copy link
Copy Markdown
Member Author

Personally, I would move all of llnl into spack. For external, we should try to directly use pip-installed dependencies. If there is anything in external that isn't on PyPI, I would also move it into spack. Of course, we would need to discuss this since it doesn't preserve the current directory structure. But that would solve all of our problems.

@adamjstewart adamjstewart mentioned this pull request Jun 28, 2025
4 tasks
@adamjstewart adamjstewart force-pushed the pip/install branch 2 times, most recently from dd7e666 to b8ad37f Compare July 19, 2025 10:41
@adamjstewart adamjstewart added this to the v1.0.1 milestone Jul 20, 2025
Signed-off-by: Adam J. Stewart <[email protected]>
Signed-off-by: Adam J. Stewart <[email protected]>
Signed-off-by: Adam J. Stewart <[email protected]>
Signed-off-by: Adam J. Stewart <[email protected]>
Signed-off-by: Adam J. Stewart <[email protected]>
Signed-off-by: Adam J. Stewart <[email protected]>
Signed-off-by: Adam J. Stewart <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core PR affects Spack core functionality workflow

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Release Spack on PyPI

4 participants