- pkg-config

yum install -y finger

yum install -y finger

Refer to https://github.com/JCSDA-internal/jedi-tools/tree/develop/crontabs/noaa-gcloud

The scripts are all set up in the /contrib space and should work after a restart of the cluster. However, any updates to R2D2 that require changes to the scrubber scripts need to be made!

spack:

concretizer:

unify: when_possible

view: false

include:

- site

- common

specs:

# Externals or gcc-built packages

- cmake

- git

- git-lfs

- wget

- curl

- pkg-config

- python

# Several packages are commented out and not removed from the list;

# this is intentional since they may be needed for running ctest etc.

# Packages built with nvhpc

- zlib-api %nvhpc

- hdf5 %nvhpc

- netcdf-c %nvhpc ~blosc ~dap ~zstd

- netcdf-fortran %nvhpc

- parallel-netcdf %nvhpc

- parallelio %nvhpc

#- nccmp

- blas

- boost %nvhpc

#- bufr

- ecbuild %nvhpc

#- eccodes

- eckit %nvhpc

- ecmwf-atlas %nvhpc

- fckit %nvhpc

# Currently using openblas, would be nice if we could use the nvhpc package/provider for this

- fftw-api

# Doesn't build with nvhpc:

#- gsibec

- gsl-lite %nvhpc

- jedi-cmake %nvhpc

#- nlohmann-json

#- nlohmann-json-schema-validator

#- odc

- sp %nvhpc

- udunits %nvhpc

- jasper %nvhpc

| Intel classic (icc, icpc, ifort) | 2021.3.0 to the latest available version in oneAPI 2023.2.3 [#fn1]_ | ``intel@`` |

| Nvidia HPC SDK (nvcc, nvc++, nvfortran) | 12.3 (Nvidia HPC SDK 24.3) [#fn3]_ | ``nvhpc@`` |

.. [#fn3]

.. _NewSiteConfigs_Linux_Ubuntu_Prerequisites:

.. _NewSiteConfigs_Linux_CreateEnv:

.. _NewSiteConfigs_Linux_CreateEnv_Nvidia:

Creating a new environment with Nvidia compilers

.. warning::

Support for Nvidia compilers is experimental and limited to a small subset of packages of the unified environment. The Nvidia compilers are known for their bugs and flaws, and many packages simply don't build. The strategy for building environments with Nvidia is therefore the opposite of what it is with other supported compilers.

In order to build environments with the Nvidia compilers, a different approach is needed than for our main compilers (GNU, Intel). Since many packages do not build with the Nvidia compilers, the idea is to provide as many packages as possible as external packages or build them with ``gcc``. Because our spack extension ``spack stack setup-meta-modules`` does not support combiniations of modules built with different compilers, packages not being built with the Nvidia compilers need to fulfil the two following criteria:

1. The package is used as a utility to build or run the code, but not linked into the application (this may be overly restrictive, but it ensures that the application will be able to leverage all of Nvidia's features, for example run on GPUs).

2. One of the following applies:

a. The package is installed outside of the spack-stack environment and made available as an external package. A typical use case is a package that is installed using the OS package manager.

b. The package is built with another compiler (typically ``gcc``) within the same environment, and no modulefile is generated for the package. The spack modulefile generator in this case ensures that other packages that depend on this particular package have the necessary paths in their own modules. If the ``gcc`` compiler itself requires additional ``PATH``, ``LD_LIBRARY_PATH``, ... variables to be set, then these can be set in the spack compiler config for the Nvidia compiler (similar to how we configure the ``gcc`` backend for the Intel compiler).

With all of that in mind, the following instructions were used on an Amazon Web Services EC2 instance running Ubuntu 22.04 to build an environment based on template ``jedi-mpas-nvidia-dev``. These instructions follow the one-off setup instructions in :numref:`Section %s <NewSiteConfigs_Linux_Ubuntu_Prerequisites>` and replace the instructions in Section :numref:`Section %s <NewSiteConfigs_Linux_CreateEnv>`.

1. Follow the instructions in :numref:`Section %s <NewSiteConfigs_Linux_Ubuntu_Prerequisites>` to install the basic packages. In addition, install the following packages using `apt`:

.. code-block:: console

2. Download the latest version of the Nvidia HPC SDK following the instructions on the Nvidia website. For ``[email protected]``:

.. code-block:: console

3. Load the correct module shipped with ``nvhpc-24-3``. Note that this is only required for ``spack`` to detect the compiler and ``openmpi`` library during the environment configuration below. It is not required when using the new environment to compile code.

.. code-block:: console

4. Clone spack-stack and its dependencies and activate the spack-stack tool.

.. code-block:: console

5. Create a pre-configured environment with the default (nearly empty) site config for Linux and activate it (optional: decorate bash prompt with environment name). At this point, only the ``jedi-mpas-nvidia-dev`` template is supported.

.. code-block:: console

6. Temporarily set environment variable ``SPACK_SYSTEM_CONFIG_PATH`` to modify site config files in ``envs/jedi-mpas-nvidia-env/site``

.. code-block:: console

7. Find external packages, add to site config's ``packages.yaml``. If an external's bin directory hasn't been added to ``$PATH``, need to prefix command.

.. code-block:: console

8. Find compilers, add to site config's ``compilers.yaml``

.. code-block:: console

9. Unset the ``SPACK_SYSTEM_CONFIG_PATH`` environment variable

.. code-block:: console

10. Add the following block to ``envs/jedi-mpas-nvidia-env/spack.yaml`` (pay attention to the correct indendation, it should be at the same level as ``specs:``):

.. code-block:: console

11. If you have manually installed lmod, you will need to update the site module configuration to use lmod instead of tcl. Skip this step if you followed the Ubuntu instructions above.

.. code-block:: console

12. Process the specs and install

It is recommended to save the output of concretize in a log file and inspect that log file using the :ref:`show_duplicate_packages.py <Duplicate_Checker>` utility.

This is done to find and eliminate duplicate package specifications which can cause issues at the module creation step below. Specifically for this environment, the

concretizer log must be inspected to ensure that all packages being built are built with the Nvidia compiler (``%nvhpc``) except for those described at the beginning of this section.

.. code-block:: console

13. Create tcl module files (replace ``tcl`` with ``lmod`` if you have manually installed lmod)

.. code-block:: console

14. Create meta-modules for compiler, mpi, python

.. code-block:: console

15. You now have a spack-stack environment that can be accessed by running ``module use ${SPACK_STACK_DIR}/envs/jedi-mpas-nvidia-env/install/modulefiles/Core``. The modules defined here can be loaded to build and run code as described in :numref:`Section %s <UsingSpackEnvironments>`.

.. _Preconfigured_Sites_Tier2:

Pre-configured sites (tier 2)

Tier 2 preconfigured site are not officially supported by spack-stack. As such, instructions for these systems are provided in form of a `README.md` in the site directory or may not be available. Also, these site configs are not updated on the same regular basis as those of the tier 1 systems and therefore may be out of date and/or not working.

The following sites have site configurations in directory `configs/sites/`:

- TACC Frontera (`configs/sites/frontera/`)

- AWS Single Node with Nvidia (NVHPC) compilers (`configs/sites/aws-nvidia/`)