For context of this PR see the last bits of this message, this is the 3rd (?) attempt at fixing issues with Python build isolation and extension install directory layout.

This PR adds a layer of indirection to improve build isolation with and without external Python, as well as usability of environment views.

It adds python-venv as a dependency to all packages that extends("python"), which has the following advantages:

1. Build isolation: only PYTHONPATH is considered in builds, not user / system packages
2. Stable install layout: fixes the problem on Debian, RHEL and Fedora where external / system python produces bin/local subdirs in Spack install prefixes. Currently bootstrapping of black, isort, mypy and flake8 is broken on Debian because of this.
3. Environment views are Python virtual environments (and if you add py-pip things like pip list work)

For a Python package py-app, the structure is as follows:

py-app depends both on python-venv and python, and python-venv depends on python.

On the file system it looks like this:

------------------------------------[python package]-------------------------------------

/opt/py-app/
├── bin
│   └── app                                         # shebang to python-venv python
└── lib
    └── python3.11
        └── site-packages
            └── app
                └── app.py

----------------------------------[python-venv package]----------------------------------

/opt/python-venv/ 
├── bin/
│   ├── activate                                    # venv activate script
│   └── python3.11 -> /opt/python/bin/python3.11    # symlink to underlying Python
└── pyvenv.cfg                                      # venv config file
  
-------------------------------------[python itself]-------------------------------------

/opt/python/
├── bin/
│   └── python3.11
├── include                                         # libraries and headers only in underlying python
├── lib/
│   ├── libpython3.11.so                            # libraries and headers only in underlying python
│   └── python3.11/
│       └── abc.py                                  # Python standard library
└── share

Views with python-venv + python

Views work whether they're symlink, hardlink or copy type (thanks to this PR). They simply flatten the above directory structure:

/view/
├── bin
│   ├── activate                                    # copy of activate script with $VIRTUAL_ENV path rewritten
│   ├── app                                         # shebang rewritten to python in view
│   └── python3.11 -> /opt/python/bin/python3.11    # symlink to underlying Python
├── include
├── lib
│   ├── libpython3.11.so -> /opt/python/lib/libpython3.11.so
│   └── python3.11
│       ├── abc.py -> /opt/python/lib/python3.11/abc.py
│       └── site-packages
│           └── app
│               └── app.py -> /opt/py-app/lib/python3.11/site-packages/app/app.py
├── pyvenv.cfg -> /opt/python-venv/pyvenv.cfg
└── share

which constitutes a Python venv. In case Python is external, you conveniently still get a bin/python3.11 pointing to system Python, which before this PR was not the case.

spec["python"].command

This PR additionally makes spec["python"].command return spec["python-venv"].command. The rationale is that packages in repos we do not own do not pass the underlying python to the build system, which could still result in incorrectly computed install layouts.

Other attributes like libs, headers should be on python anyways and need no change. In the future we can think to make python-venv a "provider wrapper" for a virtual python, and cpython another provider for python (or something along those lines). That would ultimately just clean up internals.

Spack environment views <=> Python venvs

After this PR, the following is a venv equivalent to python3 -m venv ./view:

spack:
  specs: [py-pip]
  view: ./view

The next example is a venv equivalent to python3 -m venv --without-pip ./view:

spack:
  specs: [python-venv]
  view: ./view

However, an environment that just contains python and nothing else is not a venv:

spack:
  specs: [python]
  view: ./view

because there would be no ./view/pyvenv.cfg file. This should in general not be problematic, cause the view also does not contain any Python package that need to be located relative to the view.

Bootstrap environment

Bootstrap logic is simplified with this PR. For convenience, Spack automatically re-concretizes and installs bootstrap environments that do not contain python-venv. That means spack style should just work for users on Debian, no need to spack clean -b.

Possible pitfalls

1. System Python + system Python packages

   The pyvenv.cfg file contains the following line:

   include-system-site-packages = false

   which means that you cannot import system Python packages with if you marked python external.

   I think this is new behavior. If necessary, we could put a variant on python-venv +system_packages to indicate whether it should include system site packages or not. Such a variant would then also disable build isolation. This PR does not introduce such a variant, I'm just stating this as a workaround if necessary.

Context / Alternatives considered

This all started fixing the underlying problem of #31131 properly s.t. we could install setuptools from sources instead of wheel, so that it would be easier to add patches for its vendored distutils, to fix incorrect rpath flags (we currently apply those patches in python itself, but python 3.12 dropped distutils and recommends distutils from setuptools)

1. Previously (Improve build isolation in PythonPipBuilder #40224) I tried a temporary virtual environment for the build, that ensured pip could not execute hooks from unrelated user and system packages. This had two problems:
   1. shebangs of generated executables pointed to the venv and had to be patched
   2. the install layout in a venv differs from the install layout of underlying python (e.g. Debian system Python uses bin/local vs bin when run from underlying executable vs venv resp.) since the venv is no longer around, we computed incorrect PYTHONPATHs
2. Before that I've tried passing -S flags to pip to ensure it doesn't see hooks from user / system packages, but that doesn't really work because pip runs sub-processes that do not inherit these flags.

Closes #42445
Closes #40601
Closes #37632
Closes #35214
Closes #33923
Closes #32769
Closes #32662
Closes #42445
Fixes #31939
Fixes #31131