Fixes #954.
Fixes #5655.

Summary

In this branch, compilers stop being a node attribute, and become a build-only dependency.

Packages may declare a dependency on the c, cxx, or fortran languages, which are now treated as virtuals, and compilers would be providers for one or more of those languages. Compilers can also inject runtime dependency, on the node being compiled. An example graph for something as simple as zlib-ng is the following:

Here gcc is used for both the c, and cxx languages. Edges are annotated with the virtuals they satisfy (c, cxx, libc). gcc injects gcc-runtime on the nodes being compiled. glibc is also injected for packages that require c. The compiler-wrapper is explicitly represented as a node in the DAG, and is included in the hash.

This change in the model has implications on the semantics of the % sigil, as discussed in #44379, and requires a version bump for our Specfile, Database, and Lockfile formats.

Breaking changes

Breaking changes below may impact users of this branch.

1. Custom, non-numeric version of compilers are not supported

Currently, users can assign to compilers any custom version they want, and Spack will try to recover the "real version" whenever the custom version fails some operation. To deduce the "real version" Spack must run the compiler, which can add needless overhead to common operations.

Since any information that a version like gcc@foo might give to the user, can also be suffixed while retaining the correct numeric version, e.g. [email protected], Spack will not try anymore to deduce real versions for compilers.

Said otherwise, users should have no expectation that gcc@foo behaves as [email protected] internally.

2. The % sigil in the spec syntax means "direct build dependency"

The % sigil in the spec syntax means "direct build dependency", and is not a node attribute anymore. This means that:

node.satisfies("%gcc")

is true only if gcc is a direct build dependency of the node. Nodes without a compiler dependency are allowed.

parent["child"], and node in spec, will now only inspect the link/run sub-DAG and direct build dependencies

The subscript notation for Spec:

parent["child"]

will look for a child node only in the link/run transitive graph of parent, and in its direct build dependencies. This means that to reach a transitive build dependency, we must first pass through the node it is associated with.

Assuming parent does not depend on cmake, but depends on a CMakePackage e.g. hdf5, then we have the following situation:

# This one raises an Exception, since "parent" does not depend on cmake
parent["cmake"]
# This one is ok
cmake = parent["hdf5"]["cmake"]

3. Externals differing by just the compiler attribute

Externals are nodes where dependencies are trimmed, and that is not planned to change in this branch. Currently, on develop it is ok to write:

packages:
  hdf5:
    externals:
    - spec: [email protected] %gcc
      prefix: /prefix/gcc
    - spec: [email protected] %clang
      prefix: /prefix/clang

and Spack will account for the compiler node attribute when computing the optimal spec. In this branch, using externals with a compiler specified is allowed only if any compiler in the dag matches the constraints specified on the external. The external will be still represented as a single node without dependencies.

4. Spec matrices enforcing a compiler

Currently we can have matrices of the form:

matrix:
- [x, y, z]
- [%gcc, %clang]

to get the cross-product of specs and compilers. We can disregard the nature of the packages in the first row, since the compiler is a node attribute required on each node.

In this branch, instead, we require a spec to depend on c, cxx, or fortran for the % to have any meaning. If any of the specs in the first row doesn't depend on these languages, there will be a concretization error.

Deprecations

• The entire compilers section in the configuration (i.e., compilers.yaml) has been deprecated, and current entries will be removed in v1.2.0. For the time being, if Spack finds any compilers configuration, it will try to convert it automatically to a set of external packages.
• The packages:compiler soft-preference has been deprecated. It will be removed in v1.1.0.

Other notable changes

• The tokens {compiler}, {compiler.version}, and {compiler.name} in Spec.format expand to "none" if a Spec does not depend on C, C++, or Fortran.
• The default install tree layout is now "{architecture.platform}-{architecture.target}/{name}-{version}-{hash}"

Known limitations

The major known limitations of this branch that we intend to fix before v1.0 is that compilers cannot be bootstrapped directly.

In this branch we can build a new compiler using an existing external compiler, for instance:

$ spack install gcc@14 %[email protected]

where [email protected] is external, and gcc@14 is to be built.

What we can't do at the moment is use a yet to be built compiler, and expect it will be bootstrapped, e.g. :

spack install hdf5 %gcc@14

We plan to tackle this issue in a following PR.

Important Review comments

• Turn compilers into nodes #45189 (comment)
• Turn compilers into nodes #45189 (review)
• Turn compilers into nodes #45189 (review)
• Turn compilers into nodes #45189 (review)
• Turn compilers into nodes #45189 (comment) (spack.compilers)
• Turn compilers into nodes #45189 (review)
• Turn compilers into nodes #45189 (review)
• Turn compilers into nodes #45189 (review) (mostly done, a few open discussions)
• Turn compilers into nodes #45189 (review)
• Turn compilers into nodes #45189 (review) (mostly done, a few open discussions)
• Turn compilers into nodes #45189 (comment)
• Turn compilers into nodes #45189 (review) (done, needs to be checked and resolved)
• Turn compilers into nodes #45189 (review) (done, needs to be checked and resolved)
• Turn compilers into nodes #45189 (review) (question on gcc from user)
• Turn compilers into nodes #45189 (review) (done, needs to be checked and resolved)