closes #49097

Currently, there are scenarios and use cases that are difficult to express with a single spack.yaml file. For instance:

1. Start with an old system compiler to bootstrap a new toolchain, and then software on top of that
2. Deploy a highly heterogeneous stack (different compilers, different options for the same set of specs) while ensuring a fine control over dependencies

These use cases are usually dealt with multiple environments¹ or by "abusing" certain concretization strategies². Both methods are sub-optimal and error prone.

In this PR we solve these kind of issues by allowing named group of specs, as shown in #49097. Each named group of specs:

• Can have dependencies on other groups. If that happens, the dependency groups are concretized before the current group, and their roots specs are always available for reuse in the current concretization.
• Can override configuration scopes with details that matter only for the current group, since the override is then used only for the concretization of the current group

An example environment using these features is:

# Bootstrap [email protected], then use it to build the same
# set of specs for x86_64_v3 and x86_64_v4
spack:
  definitions:
  - apps: [hdf5, mpileaks]

  packages:
    all: {prefer: [target=x86_64_v3]}

  specs:
  - group: compiler
    specs:
    - [email protected]

  - group: apps-x86_64_v4
    needs: [compiler]
    specs:
    - $apps
    override:
      packages:
        all: {prefer: [target=x86_64_v4]}
        c: {prefer: [[email protected]]}
        cxx: {prefer: [[email protected]]}
        fortran: {prefer: [[email protected]]}

  - group: apps-x86_64_v3
    needs: [compiler]
    specs:
    - $apps
    override:
      packages:
        c: {prefer: [[email protected]]}
        cxx: {prefer: [[email protected]]}
        fortran: {prefer: [[email protected]]}

The groups can also be used to create tailored views. For instance, an entry like:

views:
  x86_64_v3:
    root: ...
    group: apps-x86_64_v3

will just put specs from the apps-x86_64_v3 group into the view.

Benchmark

Trying this PR in spack/spack-packages#3175 already shows:

• A drastic reduction in the specs to be built for the ml-*-x86_64 stacks, when they are unified
• The correct concretization of the tutorial stack using unify:false, which is currently broken on develop when using the unify:when_possible hack

Notes

This PR requires a bump of the lockfile for environments, from v6 to v7. Nonetheless, to preserve backward compatbility, if the feature is not used Spack will continue to write lockfiles in v6 format, which are undersatood by previous version of Spack.

To do in following PRs

• Hook "groups" into commands other than concretization e.g. modules etc. (can be split in other PRs)
• Add a behavior similar to concrete included environments between groups (can also be in a following PR)
• Parallelize concretization among groups