Summary

This proposal aims at extending the current syntax for specs in environments, by allowing:

1. A specific order of concretization among matrices
2. The possibility of overriding configuration options per-matrix

Together with #45189, these two extensions should make it easier to bootstrap an entire stack in a single environment, starting from minimal requirements on the system.

Rationale

This extension should simplify a number of use cases, for instance:

1. aws-pcluster pipelines, that need to bootstrap compilers in N steps
2. e4s can use different matrixes for different compilers / options in a single pipeline. This means we can potentially lump e4s-rocm, e4s-oneapi etc. in a single pipeline

Ideally, many workflows that are now using multiple environments, and manual boilerplate operations to connect them, can be unified in a single environment.

Description

From a user perspective, this should just add a few optional fields to matrices:

• The fields id and needs can be used to specify dependencies among parts of the same environment.
• The configuration field can be used to override config option for that matrix only

An example spack.yaml might look like:

spack:
  specs:
  - group: gcc12
    spec: gcc@12+binutils

  - group: oneapi2024
    needs:
    - gcc12
    spec: [email protected] %gcc@12

  - group stack
    needs:
    - gcc12
    - oneapi2024
    matrix:
    - [ gromacs, lammps]
    - ["%oneapi"]
    configuration:
      packages:
        fortran:
          require: [email protected]

  view: true
  concretizer:
    unify: true

Roughly, the rules for concretizing it would be as follows:

• If a matrix id is needed by another matrix, then it will be concretized before that other matrix, and the concretized root specs will be available for "reuse" (regardless of filters that are applied to reuse). In the example above gcc12 is concretized before oneapi2024 and the concrete gcc@12+binutils spec is available when concretizing [email protected] %gcc@12.
• Matrixes / specs without an id are considered to be part of a default id that has no dependencies
• Configuration overrides per matrix are used only when concretizing that matrix

The syntax used is taken from Github Actions syntax, so users would hopefully be familiar with it.

Additional information

For debugging purposes, we should be able to easily reproduce intermediate steps of concretization, e.g. try to concretize the oneapi2024 matrix and be able to inspect its input, output, ASP encoding etc.

General information

• I have searched the issues of this repo and believe this is not a duplicate