Overview

The goal of this PR is to make gitlab pipeline builds (especially build failures) more reproducible outside of the pipeline environment. The two key changes here which aim to improve reproducibility are:

1. Produce a spack.lock during pipeline generation which is passed to child jobs via artifacts. This concretized environment is used both by generated child jobs as well as uploaded as an artifact to be used when reproducing the build locally.
2. In the spack ci rebuild command, if a spec needs to be rebuilt from source, do this by generating and running an install.sh shell script which is then also uploaded as a job artifact to be run during local reproduction.

To make it easier to take advantage of improved build reproducibility, this PR also adds a new subcommand, spack ci reproduce-build, which, given a url to job artifacts:

• fetches and unzips the job artifacts to a local directory
• looks for the generated pipeline yaml and parses it to find details about the job to reproduce
• attempts to provide a copy of the same version of spack used in the ci build
• if the ci build used a docker image, the command prints a docker run command you can run to get an interactive shell for reproducing the build

Some highlights

One consequence of this change will be much smaller pipeline yaml files. By encoding the concrete environment in a spack.lock and passing to child jobs via artifacts, we will no longer need to encode the concrete root of each spec and write it into the job variables, greatly reducing the size of the generated pipeline yaml.

Additionally spack ci rebuild output (stdout/stderr) is no longer internally redirected to a log file, so job output will appear directly in the gitlab job trace. With debug logging turned on, this often results in log files getting truncated because they exceed the maximum amount of log output gitlab allows. If this is a problem, you still have the option to tee command output to a file in the within the artifacts directory, as now each generated job exposes a user_data directory as an artifact, which you can fill with whatever you want in your custom job scripts.

There are some changes to be aware of in how pipelines should be set up after this PR:

Pipeline generation

Because the pipeline generation job now writes a spack.lock artifact to be consumed by generated downstream jobs, spack ci generate takes a new option --artifacts-root, inside which it creates a concrete_env directory to place the lockfile. This artifacts root directory is also where the user_data directory will live, in case you want to generate any custom artifacts. If you do not provide --artifacts-root, the default is for it to create a jobs_scratch_dir within your CI_PROJECT_DIR (a gitlab predefined environment variable) or whatever is your current working directory if that variable isn't set. Here's the diff of the PR testing .gitlab-ci.yml taking advantage of the new option:

$ git diff develop..pipelines-reproducible-builds share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml
diff --git a/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml b/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml
index 579d7b56f3..0247803a30 100644
--- a/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml
+++ b/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml
@@ -28,10 +28,11 @@ default:
     - cd share/spack/gitlab/cloud_pipelines/stacks/${SPACK_CI_STACK_NAME}
     - spack env activate --without-view .
     - spack ci generate --check-index-only
+      --artifacts-root "${CI_PROJECT_DIR}/jobs_scratch_dir"
       --output-file "${CI_PROJECT_DIR}/jobs_scratch_dir/cloud-ci-pipeline.yml"
   artifacts:
     paths:
-      - "${CI_PROJECT_DIR}/jobs_scratch_dir/cloud-ci-pipeline.yml"
+      - "${CI_PROJECT_DIR}/jobs_scratch_dir"
   tags: ["spack", "public", "medium", "x86_64"]
   interruptible: true

Notice how we replaced the specific pointer to the generated pipeline file with its containing folder, the same folder we passed as --artifacts-root. This way anything in that directory (the generated pipeline yaml, as well as the concrete environment directory containing the spack.lock) will be uploaded as an artifact and available to the downstream jobs.

Rebuild jobs

Rebuild jobs now must activate the concrete environment created by spack ci generate and provided via artifacts. When the pipeline is generated, a directory called concrete_environment is created within the artifacts root directory, and this is where the spack.lock file is written to be passed to the generated rebuild jobs. The artifacts root directory can be specified using the --artifacts-root option to spack ci generate, otherwise, it is assumed to be $CI_PROJECT_DIR. The directory containing the concrete environment files (spack.yaml and spack.lock) is then passed to generated child jobs via the SPACK_CONCRETE_ENV_DIR variable in the generated pipeline yaml file.

When you don't provide custom script sections in your mappings within the gitlab-ci section of your spack.yaml, the default behavior of rebuild jobs is now to change into SPACK_CONCRETE_ENV_DIR and activate that environment. If you do provide custom rebuild scripts in your spack.yaml, be aware those scripts should do the same thing: assume SPACK_CONCRETE_ENV_DIR contains the concretized environment to activate. No other changes to existing custom rebuild scripts should be required as a result of this PR.

As mentioned above, one key change made in this PR is the generation of the install.sh script by the rebuild jobs, as that same script is both run by the CI rebuild job as well as exported as an artifact to aid in subsequent attempts to reproduce the build outside of CI. The generated install.sh script contains only a single spack install command with arguments computed by spack ci rebuild. If the install fails, the job trace in gitlab will contain instructions on how to reproduce the build locally:

To reproduce this build locally, run:
  spack ci reproduce-build https://gitlab.next.spack.io/api/v4/projects/7/jobs/240607/artifacts [--working-dir <dir>]
If this project does not have public pipelines, you will need to first:
  export GITLAB_PRIVATE_TOKEN=<generated_token>
... then follow the printed instructions.

When run locally, the spack ci reproduce-build command shown above will download and process the job artifacts from gitlab, then print out instructions you can copy-paste to run a local reproducer of the CI job.

This PR includes a few other changes to the way pipelines work, see the documentation on pipelines for more details.

This PR erelies on
- [ ] #23194 to be able to refer to uninstalled specs by DAG hash
EDIT: that is going to take longer to come to fruition, so for now, we will continue to install specs represented by a concrete spec.yaml file on disk.

• --no-add installs only specified specs and only if already present in… #22657 to support install a single spec already present in the active, concrete environment