@spackbot fix style

Let me see if I can fix that for you!

I was able to run spack style --fix for you!

==> Running style checks on spack
  selected: isort, black, flake8, mypy
==> Modified files
  var/spack/repos/builtin/packages/py-pip/package.py
  var/spack/repos/builtin/packages/python/package.py
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted var/spack/repos/builtin/packages/python/package.py
All done! ✨ 🍰 ✨
1 file reformatted, 1 file left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
Success: no issues found in 577 source files
  mypy checks were clean
==> spack style checks were clean

I've updated the branch with style fixes.

Really like the idea, but don't want to deprecate any versions (I'm going to deprecate 3.7 tomorrow though).

Is there any reason why not? I don't think there are any packages that would break or fail to concretize by updating a patch version...

@tgamblin is concerned about environment reproducibility. We want people to be able to reproduce an installation years later. Of course, we're still deprecating and removing EOL versions, but that's only after 5 years since the release date, so that's not as bad.

I could add a few choice words for @tgamblin about Spack and (lack of) reproducibility, but I'll hold off here to stay on topic.

My understanding from the official packaging guide is that marking version(s) as deprecated=True expresses that:

1. Said version(s) are discouraged in new concretizations,
2. You get a confirmation prompt when installing (i.e. heavily discouraged to install said version(s)),
3. The Spack team reserves the right to not give it full support (i.e. patches may not be added, "extras" such as --test may not work, etc.), and
4. Said version(s) may be removed in a later release of Spack.

I'm not suggesting that these versions be removed in later versions and break reproducibility (4), but they should be discouraged in any new work due to the risks (1,2) and --test no longer passes due to the new check for PEP 668 (3). It seems like deprecated=True is exactly the flag to use in this case... am I missing something?

I could add a few choice words for @tgamblin about Spack and (lack of) reproducibility, but I'll hold off here to stay on topic

I mean, you didn’t — what are the choice words? Maybe start a separate issue?

Happy to add more things to enable reproducibility, but as I think you know, we have many pans in the fire. You are always welcome to help improve stuff and we appreciate PRs like this one.

FWIW, I’m fine with deprecating versions here, the real issue is whether this conflicts with @adamjstewart’s work. I leave that part up to him.

Spack environments do not provide the same experience as a virtualenv or venv, the environment is managed by program(s) (i.e. Spack) external to the Python packaging infrastructure (i.e. pip). The primary effect is that pip-installed packages are (silently!!!) uninstalled whenever the view regenerates, for any reason:

# spack env activate --temp && spack install --add python
# python -m ensurepip && pip3 install pre-commit
# pre-commit --version
pre-commit 3.3.3
# spack install --add cmake
# pre-commit --version
bash: /tmp/spack-d977jsnc/.spack-env/view/bin/pre-commit: No such file or directory
# python -m ensurepip && pip3 install pre-commit                        
# spack env view disable && spack env view enable
# pre-commit --version
bash: /tmp/spack-d977jsnc/.spack-env/view/bin/pre-commit: No such file or directory

Because of this external and significant influence, Spack environments also need to be marked as EXTERNALLY-MANAGED. I can however extend the warning to mention that --break-system-packages is "less bad" when done inside a Spack environment (and to be careful of view regeneration in that case). The preferred solution will remain a venv.

I'm not super keen on +externally-managed, it's pretty much essential to maintain sanity when both package managers are present and active ("+dont-let-pip-break-my-spack-install-please"). If any users are relying on a silently working pip, they are playing with fire and/or are very lucky (IMHO there is no "I know what I'm doing" category here, if you knew what you were doing you'd use a venv). PEP 668 is the official "slap on the wrist" for those users. I don't think it's a good idea, but I can add the variant if it's insisted upon.

I will extend the warning by referring to a more verbose description stored in a separate file, I'll rip off "take inspiration" from a very nice Markdown document shipped in Debian (/usr/share/doc/python3.*/README.venv) that discusses these options and more. (I'll also throw together a PR adding a recipe for pipx, since that's recommended for Python applications and isn't in Spack yet.)

Sounds mostly good, but:

Because of this external and significant influence, Spack environments also need to be marked as EXTERNALLY-MANAGED.

They should not be marked this way. We can fix the issue but currently I even occasionally use pip in Spack environments despite the limitation. This breaks that. I still think we should make it so that Spack environments are not marked externally managed.

I don't think it's a good idea, but I can add the variant if it's insisted upon.

Please add the variant and default to True -- which basically means anyone who doesn't know about this is going to get an externally managed Spack python.

Because of this external and significant influence, Spack environments also need to be marked as EXTERNALLY-MANAGED.

They should not be marked this way. We can fix the issue but currently I even occasionally use pip in Spack environments despite the limitation. This breaks that. I still think we should make it so that Spack environments are not marked externally managed.

@pradyunsg would be curious to hear what your thoughts are on this part -- what's the "right" way to implement PEP 668 for environments that do support pip installs?

Note that the --break-system-packages option is going to be confusing to users here, since they're not system packages -- they're packages in a Spack environment, likely in some user install of Spack. I wish the arg were more obviously related to EXTERNALLY-MANAGED.

While we're on the topic, PEP 668 defines a way to prevent pip from installing packages. But is there a way to prevent pip from uninstalling packages? Oftentimes when using pip I find that it uninstalls a Spack-installed package. The resulting environment works, but other Spack installs now break. This came up recently in #28282.

Working backwards:

While we're on the topic, PEP 668 defines a way to prevent pip from installing packages. But is there a way to prevent pip from uninstalling packages?

PEP 668 handles both, pip will refuse to alter an installation marked as EXTERNALLY-MANAGED (without --break-system-packages). This idea is repeated throughout the text of the PEP:

[Abstract:] This PEP proposes a mechanism for a Python installation to communicate to tools like pip that its global package installation context is managed by some means external to Python, such as an OS package manager. It specifies that Python-specific package management tools should neither install nor remove packages into the interpreter’s global context, by default, and should instead guide the end user towards using a virtual environment.

[Motivation:] First, it proposes a way for distributors of a Python interpreter to mark that interpreter as having its packages managed by means external to Python, such that Python-specific tools like pip should not change the installed packages in the interpreter’s global sys.path in any way (add, upgrade/downgrade, or remove) unless specifically overridden. It also provides a means for the distributor to indicate how to use a virtual environment as an alternative.

[Use case 8:] As a result of this PEP, pip will no longer be able to remove packages already on the system. However, this behavior change is fine because a package build process should not (and generally cannot) include instructions to delete some other files on the system; it can only package up its own files.

Note that the --break-system-packages option is going to be confusing to users here, since they're not system packages -- they're packages in a Spack environment, likely in some user install of Spack. I wish the arg were more obviously related to EXTERNALLY-MANAGED.

There's not much helping that one. From Python's perspective the Spack install or environment is the system installation. So the name accurate in principle but not in Spack's larger context. 😞

The silver lining is that the pip install --help output and online docs describe the option as:

  --break-system-packages     Allow pip to modify an EXTERNALLY-MANAGED Python installation

what's the "right" way to implement PEP 668 for environments that do support pip installs?

I'm pretty certain the answer here is "you don't." PEP 668 is for installations/environments that pip should not touch. If pip is allowed to touch an installation/environment, then you don't implement PEP 668 and don't have an EXTERNALLY-MANAGED file.

Please add the variant and default to True -- which basically means anyone who doesn't know about this is going to get an externally managed Spack python.

👍

Because of this external and significant influence, Spack environments also need to be marked as EXTERNALLY-MANAGED.

They should not be marked this way. We can fix the issue but currently I even occasionally use pip in Spack environments despite the limitation. This breaks that. I still think we should make it so that Spack environments are not marked externally managed.

For full clarity, this change will make it so you you have to explicitly --break-system-packages to do what you (shouldn't but) do now. This flag roughly expresses that "I, Human, have RTFM'd, am consciously aware of the limitations and caveats of performing this action (install/upgrade/remove) and will take full responsibility when my experience is degraded as a consequence."

Spack does not interoperate with pip (#28282). Until it does, users need to be aware of the caveats if they wish to combine spack and pip commands in any meaningful way. An explicit --break-system-packages option is a simple, convenient and intuitive way to provide that awareness.

I believe breaking the current "silently works" semantics of pip install is a necessary evil here. If you don't agree, then at least patch py-pip to not be silent in this case, and extend the documentation to be excessively clear that this behavior has significant caveats (worse than Conda's) and that there are better solutions users should be using.

PEP 668 defines a way to prevent pip from installing packages. But is there a way to prevent pip from uninstalling packages?

EXTERNALLY-MANAGED protections in pip extend to pip uninstall as well.

@pradyunsg would be curious to hear what your thoughts are on this part -- what's the "right" way to implement PEP 668 for environments that do support pip installs?

Having an EXTERNALLY-MANAGED file means that you don't want pip to be installing in the environment. If you want to block pip [un]installs, then you should add the EXTERNALLY-MANAGED file. If you don't want to block that, do not add the file.

Honestly, I don't quite understand what the nuances/intent/goals are with the interoperability needs of Spack/Conda etc with pip, and what exactly the intended behaviours are. 😅

If you don't want people to use pip to uninstall files installed by Spack, you can remove the RECORD files in the .dist-info. If you don't want pip to be able to install packages into the Python environment created by Spack, add the externally-managed file. If you want some behaviour other than that, let me know! :)

If I'm following along, it sounds like we want to add EXTERNALLY-MANAGED to both python and all py-* packages installed by Spack, but we don't want to symlink this file to any Spack environments we create. This will prevent pip from installing to the python package prefix and from uninstalling any py-* packages, but will still allow pip to integrate nicely on top of an environment. Is this correct?

This pull request has been automatically marked as stale because it has not had
any activity in the last 6 months. It will be closed if there is no further activity.
Thank you for your contributions!

This pull request was closed because it had no activity for 30 days after being marked stale.