Skip to content

consider using the pex library to allow spack users to configure where their python code resolves from #20504

@cosmicexplorer

Description

@cosmicexplorer

I don't know if this would have even been feasible before #17587 is merged, which introduces a much more natural and unified way to refer to python packages off of pypi.

I'm coming from using tools which very kindly allowed interchanging different sources of python repos: see pants configuration at https://www.pantsbuild.org/docs/python-third-party-dependencies#using-custom-repositories:

[python-repos]
indexes.add = ["https://custom-cheeseshop.net/simple"]

Closer to the point, see pex, which is both a CLI tool and a python library: https://pex.readthedocs.io/en/latest/buildingpex.html#tailoring-requirement-resolution

In general, pex honors the same options as pip when it comes to resolving packages. Like pip, by default pex fetches artifacts from PyPI. This can be disabled with --no-index.
If PyPI fetching is disabled, you will need to specify a search repository via -f/--find-links. This may be a directory on disk or a remote simple http server.
For example, you can delegate artifact fetching and resolution to pip wheel for whatever reason – perhaps you’re running a firewalled mirror – but continue to package with pex:

pip wheel -w /tmp/wheelhouse sphinx sphinx_rtd_theme
pex -f /tmp/wheelhouse --no-index -e sphinx:main -o sphinx.pex sphinx sphinx_rtd_theme

Does Spack currently have the ability to address any of these use cases? The reason I ask is because pex has this incredibly nice python library API for things that would normally involve manually scraping webpages:

from pex.resolver import download

download(['my-package==3.0'],
         indexes=['https://my-corporate-pypi.repo'],
         find_links=['/my/local/dir', 'https://my-corporate-flat.repo']) 

Pex also staunchly remains compatible with python 2.6 (pip install pex works).

Figuring out the appropriate way to pipe this sort of configuration into the user is definitely its own question, but I would be interested in knowing if it seems at all useful to look into how to depend on the pex library to expose this repo location choice via settings.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions