This PR updates the spack submodule to version 1.0.1 with a few extra commits (see JCSDA/spack#563). The two most important changes are the separation of the spack packages (builtin repo) into its own repository https://github.com/spack/spack-packages, and that compilers are now "nodes" (regular dependencies with a few extra attributes). Please refer to the updated spack documentation and release pages for more details, I cannot cover this here.

The separation of the repository is handled as follows: Our fork of the spack-packages repo https://github.com/jcsda/spack-packages is cloned as a submodule into repos/builtin (and spack is modified to find it there). The spack-stack repo lives under repos/spack_stack as part of the spack-stack GitHub repository. The spack_stack packages were migrated to the new API.

The handling of compilers as nodes means that compilers.yaml no longer exists; instead, compilers are configured under their new names in packages.yaml. The GitHub CI runs on the self-hosted runner illustrate how to do this for a new site config.

The spack-generated modules differ slightly from beforehand:

- For lmod, modules for packages that do not depend on a compiler are now in modulefiles/Core
- For tcl/ modules for packages are now in modulefiles/none/none and modulefiles/mpi-name/mpi-version/none/none
- The module hierarchy names use the new compiler names instead of the legacy names (see spack.aliases)
- The compiler changes and the associated updates to the spack engine provided an opportunity to update and simplify the spack stack setup-meta-modules extension significantly. With this, the 7-digit hash for the mpi module names in lmod are removed, and there is no longer a stack-python module. The spack python module is loaded as part of the module dependencies. We still have stack-compiler and stack-mpi.

This PR updates only the Ubuntu self-hosted runner and a few selected site configs (see below, systems affected). I would like to merge this PR without waiting for all other site configs and container builds being updated to make this more manageable. We should create an issue to document that spack-stack develop will be broken for the sites that are not yet updated.

Similarly, only selected templates were updated, all others moved to configs/templates_NOT_YET_CONVERTED_TO_SPACK_v1.

In configs/common, the following compiler packages.yaml are updated: clang, gcc, oneapi. I suggest removing aocc, apple-clang, intel if nobody else is willing to fix them in a follow-up PR.

Site config updates: I encourage spack-stack developers to use the tier1 atlantis or tier2 bounty/blackpearl site configs as blueprints for updating your site configs. Specifically, note one important change for the --compiler= argument for spack stack create env:

spack stack create env --name=ne-oneapi-2025.1.0 --site=blackpearl --template=neptune-dev --compiler=oneapi-2025.1.0 2>&1 | tee log.create.ne-oneapi-2025.1.0.001
if the site contains configs/sites/tierX/sitename/packages_oneapi-2025.1.0.yaml. If the site contains configs/sites/tierX/sitename/packages_oneapi.yaml instead (the versioned and non-versioned files are mutually exclusive), then the command is

spack stack create env --name=ne-oneapi-2025.1.0 --site=blackpearl --template=neptune-dev --compiler=oneapi 2>&1 | tee log.create.ne-oneapi-2025.1.0.001
This is to support multiple versions of the same compiler for a given site. (Side note: Because filter_compilers is no longer needed, it is removed.) Note that spack stack create env will not fail if the file packages_compiler.yaml or packages_compiler-version.yaml doesn't exist - it will print a big and fat warning, though.

MISSING DOCUMENTATION UPDATES - REFER TO A FOLLOW UP PR BY SOMEONE ELSE

…s, MPI meta-modules updates, NRL site config updates round 2 of X (#1782)

This PR is for branch feature/update_to_spack_v1.

spack-packages submodule update for Intel oneAPI FPP bug affecting NASA packages, to allow for external Intel oneAPI runtime libraries, for ESMF bug (linking to MPI C++ libraries)
Numerous additional spack-packages updates from the authoritative repo (contributed by @AlexanderRichert-NOAA)
Return control over zlib libraries to site configs (can be external or spack-built)
Return control over runtime libraries to site configs (can be external or spack-built)
Updates to MPI meta-modules: set correct compiler wrapper and backend environment variables, set additional environment variables that define the MPI install prefix
NRL site config updates, second round (more to come in a follow-up PR)

…H so it can find rustc

…lua (#1793)

Acorn spack-stack v2 updates

Update to the latest versions of scotch, g2tmpl, and prod-util. Update the spack-packages submodule pointer.

This PR ensures that CARGO_HOME is used (rather than the default of ~/.cargo) when fetching cargo deps, and also gets rid of the llnl.* deprecation warning for fetch_cargo_deps.py.

…site configs; add flag to 'spack stack create env' to promote warnings to errors; updates from develop and release/1.9.0 branches (#1798)

- Add argument --treat-warnings-as-errors for spack stack create env so that warnings for non-existent packages_*.yaml are promoted to errors. The default remains unchanged - they remain warnings.
- Update NRL batch install script for spack v1 updates (mostly due to site config updates).
- By default, do not create spack compiler modules (not to be confused with meta modules, we definitely do want those by default), use the native compiler modules on HPCs. For user site configs and CI, explicitly add compiler modules to module includelist in configs/sites/**/modules.yaml.
- Bring in updates from spack-stack develop (which are updates from the latest release 1.9.3): site configs and wgrib2 variants. All site config updates except NRL sites are from this step.
- Remove explicit variants for ewok-env in configs/templates/skylab-dev/spack.yaml and configs/templates/unified-dev/spack.yaml, since they are the default values. This allows us to turn off the ecflow variant in the site configs (c.f. Atlantis).
- Add missing export of environment variables from MPI packages in MPI meta modules setup-meta-modules).
- [email protected]

…, bug fix for Intel oneAPI MPI in setup-meta-modules (#1807)

1. New package adp-preprocessors, enabled as a variant for jedi-neptune-env and off by default so that the unified environment builds out of the box on other systems.
2. Use w3emc version 2.10.0 instead of the newer 2.11.0 with LLVM native, because 2.11.0 doesn't build. We only need 2.11.0 with Intel LLVM for now, because the older versions don't support the Intel LLVM compilers. In a future update, when grib-utils are updated to no longer require the deprecated extradeps variant from w3emc, we can hopefully update to 2.12.0 or a newer version that builds with all compilers.
3. Update submodule pointer for spack-packages for the recently merged PR Use spack v1 syntax in MCT package to set -fPIC spack-packages#17 (use spack v1 syntax in MCT package to set -fPIC).
4. Bug fix in spack-ext/lib/jcsda-emc/spack-stack/stack/meta_modules.py: set correct path to Intel oneAPI MPI compiler wrappers (this can be wrong, depending on how the external Intel oneAPI MPI was installed and is used (prefix vs modules).

…eview and testing

…eview and testing

…bfuji/spack-stack into feature/shorter_modulepaths

This PR shortens the module paths for the modules generated by spack module [tcl|lmod] refresh and spack stack setup-meta-modules. It does so by implementing a compiler- and MPI-name translation and shortening the path where the modules are generated. This is done in multiple places:

configs/common/modules_lmod.yaml
(configs/common/modules_tcl.yaml)
spack-ext/lib/jcsda-emc/spack-stack/stack/meta_modules.py
While this solution comes with a certain amount of code duplication, I consider it as the best option because any other solution would have required changes to the spack code itself. Also, we rarely make changes to these names or add new compilers and MPI providers, thus the maintenance effort is low.

While working on this, I realized that the ecmwf-atlas --> atlas translation never worked as intended, because it was missing the MPI module dependency. I checked and the spack-generated modules were named ecmwf-atlas before this PR was made (at least for tcl modules). I therefore removed this mapping, since the module name ecmwf-atlas apparently worked fine until now.

* Build FMS in two different ways

* Update NAS configs

…x setup-meta-modules, Nautilus gcc stack update (#1817)

Update configs/sites/tier1/nautilus/packages_gcc-13.3.1.yaml

Bug fix in spack-ext/lib/jcsda-emc/spack-stack/stack/meta_modules.py: when creating MPI meta module, set environment variables from MPI spec (not compiler spec), and use correct compiler/mpi aliases. Check that meta modules are created.

Update submodule pointer for repos/builtin for netcdf-c+mpi bug fix and [email protected] addition: Add [email protected] + bug fix for netcdf-c to find MPI functions when build system is cmake (replaces #18) spack-packages#21

Update configs/** for [email protected]

This PR makes the following general changes:

- blacklist compiler-wrapper and intel-oneapi-compilers-classic from modules
- adds a uwtools variant to global-workflow-env; on by default but disable-able for stacks/sites where it's not needed
- use 'require' to set virtual package providers in common/packages*.yaml so that we don't have to configure/override these separately for various dependencies (and to avoid possible concretization quirks because of "providers" being a soft preference).
- move show_duplicate_packages output logic outside the show_duplicate_packages() function; also cleaned up some defunct logic left over from the previous iteration of this script

and the following Acorn/WCOSS2/NCO-specific changes:

- overhaul acorn and wcoss2 site configs, including reconciling them so they're basically the same apart from some NCO-specific package configuration
- overhaul of deploy.py: better logging; better integration with Spack Python API; move deployments.yaml under site config dir; validate env's by ensuring only user-specified packages can be built with GCC; add a "modules_flat/" directory that tests having all modulefiles in a flat structure (no compiler or mpi subdirectories)
- resurrect 'nco' template and add it to acorn deployments
- no more intel@2022 (per NCO & EMC agreement)

* Update ECMWF libs ecbuild, eckit, fckit, atlas, fiat, ectrans
* Update spack-packages submodule version

…with package variant `+openmp`; add variants `ncview` and `openmp` to `neptune-env` (#1829)

- Add suffix -noopenmp to package modules for esmf, ip, neptune-env when variant ~openmp is used. This includes the special ESMF suffices for MAPL.
- Remove unused ESMF suffices for MAPL modules.
- Add variants ncview and openmp to the neptune-env virtual package; update the neptune-dev and neptune-ops templates accordingly.
- Update submodule pointer for repos/builtin (misc package updates spack-packages#24 and ESMF: Add option to compile without OpenMP spack-packages#25),
- Update submodule pointer for spack (install_test.py: append to install-time test log for multiple test phases (#50477) spack#569),

…eature/update_to_spack_v1