Summary

This PR removes unused options for the "spack stack" extension in an order to clean up the amount of code and facilitate adding the new capability described below.

The new capability requires the user to define a compiler --compiler when running spack stack create env. It further updates the logic for creating common and site environment spack config files for modules and packages. For both common and site configs (in configs/common/ and configs/sites/tier?/*/), the following is true:

1. For packages: If both packages.yaml and packages_${COMPILER}.yaml exist in the same directory, merge the two (packages_${COMPILER}.yaml overrides packages.yaml if necessary) and write to the environment's common or site config as packages.yaml. See for example the updated blackpearl site config in this PR. If only one of them exists, copy that file to the destination.
2. For modules: If both modules.yaml and packages_${LUA_OR_LMOD}.yaml exist in the same directory, merge the two (packages_${LUA_OR_LMOD}.yaml overrides modules.yaml if necessary) and write to the environment's common or site config as modules.yaml. This is currently not the case, but maybe we can consolidate some of the contents into one default modules.yaml file?

In the environment spack.yaml, update the compiler definition line for the matrix (if applicable) and set packages:all:require:['%COMPILER']. This requires that packages that must be built with other compilers (e.g. gcc-runtime) have their own require: '%COMPILER' setting to overwrite this. This is already the case for gcc-runtime in configs/common/packages.yaml. The same may be needed for other runtime packages for each of the compilers in the future, and for packages like bison that don't build with oneapi for example. I am basically following what E4S does.

Testing

• Created unified-env and empty environments with both gcc and oneapi on blackpearl, concretized and verified that the blas/lapack/fftw-api providers are correct, the correct compilers are being used, and that no conflicts are reported (except the known/greenlighted ones).
• CI (this represents testing new site configs for different compilers and OSs)
• Built neptune-env on Nautilus with [email protected] using the existing site config (i.e. no changes necessary). Then built NEPTUNE with the same compiler and ran basic tests.
• Built intel-impi container with jedi-ci specs

Applications affected

None

Systems affected

None

Dependencies

• waiting on Updates of hdf-eos2 for Intel oneAPI spack#452

Issue(s) addressed

Resolves #1167
Resolves #1126

Checklist

• This PR addresses one issue/problem/enhancement, or has a very good reason for not doing so.
• These changes have been tested on the affected systems and applications.
• All dependency PRs/issues have been resolved and this PR can be merged.