CI: Concretization step silently fails #27865
Description
Steps to reproduce
Using the following spack generate job, no child pipelines are actually generated. The gitlab job "passes" but no pipelines are uploaded as artifacts, and the child pipeline croaks. This occurs with commit 2d45a9d617 (and all commits since), which introduced the concretization pool. This does not occur with commit 64a323b, which is the parent of 2d45a9d617.
spack-generate-job:
<<: *spack-variables
<<: *spack-rules
stage: spack-generate
image: spack/ubuntu-bionic
tags: [k8s, ikp, exasgd, basic]
before_script:
- git clone ${SPACK_REPO}
- pushd spack && git checkout ${SPACK_REF} && popd
- . "./spack/share/spack/setup-env.sh"
script:
- cp buildsystem/container/spack.yaml . && spack env activate --without-view .
- spack -d ci generate
--artifacts-root "${CI_PROJECT_DIR}/jobs_scratch_dir"
--output-file "${CI_PROJECT_DIR}/jobs_scratch_dir/pipeline.yml"
artifacts:
paths:
- "${CI_PROJECT_DIR}/jobs_scratch_dir"Expected Output
I expect the usual concretization output, along with all jobs for specs that need to be rebuilt uploaded as artifacts:
==> [2021-12-08-18:37:29.824885] Pruning spec that does not need to be rebuilt.
==> [2021-12-08-18:37:29.880549] 17 build jobs generated in 11 stages
==> [2021-12-08-18:37:29.880590] The max_needs_job is (specs) exago/v5c656i develop [email protected] linux-ubuntu18.04-x86_64, with 5 needs
==> [2021-12-08-18:37:29.880630] Warning: Unable to populate buildgroup without CDash credentials
==> [2021-12-08-18:37:29.882520] '/usr/bin/git' 'describe' '--tags' '--match' 'v*'
Uploading artifacts...
00:01
/builds/exasgd/frameworks/exago/jobs_scratch_dir: found 5 matching files
Uploading artifacts to coordinator... ok id=86300 responseStatus=201 Created token=2G4bF89dActual Output
The concretization output seems to be omitted and the pipeline appears to pass even though the child pipelines are not uploaded. This takes a very long time to occur.
$ spack -d ci generate --artifacts-root "${CI_PROJECT_DIR}/jobs_scratch_dir" --output-file "${CI_PROJECT_DIR}/jobs_scratch_dir/pipeline.yml"
... (snipped output)
==> [2021-12-06-22:14:15.564685] Starting concretization pool with 8 processes
Uploading artifacts...
00:01
/builds/exasgd/frameworks/exago/jobs_scratch_dir: found 1 matching files
Uploading artifacts to coordinator... ok id=86080 responseStatus=201 Created token=pQsj7GWQError message
The child gitlab ci pipeline fails with:
Found errors in your .gitlab-ci.yml:
Path `jobs_scratch_dir/pipeline.yml` does not exist inside the `spack-generate-job` artifacts archive!
You can also test your .gitlab-ci.yml in CI Lint
Information on your system
Relevant snippet of .gitlab-ci.yml:
stages:
- spack-generate
- spack-build
.spack-variables: &spack-variables
variables:
# We usually use develop branch or tagged version here, but using commits to identify the issue
# 64a323b, parent commit of commit that introduced concretization pools
# 2d45a9d617, commit that introduced conc pools
SPACK_REF: 2d45a9d617
SPACK_REPO: https://github.com/LLNL/spack.git
S3_ENDPOINT_URL: http://cache.exasgd.pnl.gov
.spack-rules: &spack-rules
rules:
- if: '$CI_COMMIT_REF_NAME == "develop"'
- if: '$CI_COMMIT_REF_NAME == "master"'
- if: '$CI_COMMIT_MESSAGE =~ /\[matrix\]/'
- if: $CI_MERGE_REQUEST_ID
when: never
- *pnnl-rule # specific to our site's gitlab
- when: manual
spack-generate-job:
<<: *spack-variables
<<: *spack-rules
stage: spack-generate
image: spack/ubuntu-bionic
tags: [k8s, ikp, exasgd, basic]
before_script:
- git clone ${SPACK_REPO}
- pushd spack && git checkout ${SPACK_REF} && popd
- . "./spack/share/spack/setup-env.sh"
script:
- cp buildsystem/container/spack.yaml . && spack env activate --without-view .
- spack -d ci generate
--artifacts-root "${CI_PROJECT_DIR}/jobs_scratch_dir"
--output-file "${CI_PROJECT_DIR}/jobs_scratch_dir/pipeline.yml"
artifacts:
paths:
- "${CI_PROJECT_DIR}/jobs_scratch_dir"
spack-build-job:
<<: *spack-rules
<<: *spack-variables
stage: spack-build
trigger:
include:
- artifact: "jobs_scratch_dir/pipeline.yml"
job: spack-generate-job
strategy: depend
needs:
- artifacts: True
job: spack-generate-jobSpack environment used for spack ci
spack:
definitions:
- pkgs:
- exago@develop+hiop
- rajas: ['+raja', '~raja']
- ipopts: ['+ipopt', '~ipopt']
- mpis: ['+mpi', '~mpi']
- arch:
- '%[email protected] arch=linux-ubuntu18.04-x86_64'
specs:
- matrix:
- [$pkgs]
- [$arch]
- [$rajas]
- [$ipopts]
- [$mpis]
mirrors:
pnnl: s3://spack-ci
config:
install_tree:
padded_length: 128
packages:
all:
target: [x86_64]
providers:
mpi: [openmpi, mpich]
petsc:
version: [3.14.6]
variants: ~hypre~hdf5~metis~superlu-dist
ipopt:
variants: +coinhsl~mumps
version: [3.12.10]
hiop:
version: [0.5.1]
coinhsl:
variants: +blas
gitlab-ci:
# the mc business is to pull proprietary tarballs so spack can build coinhsl in CI without violating licenses
before_script:
- unset SPACK_SIGNING_KEY
- curl -L -s -o mc 'https://dl.min.io/client/mc/release/linux-amd64/mc'
&& chmod +x ./mc
&& ./mc alias set minio $S3_ENDPOINT_URL $AWS_ACCESS_KEY_ID $AWS_SECRET_ACCESS_KEY
&& ./mc cp -r minio/private/ .
- git clone ${SPACK_REPO}
- pushd spack && git checkout ${SPACK_CHECKOUT_VERSION} && popd
- . "./spack/share/spack/setup-env.sh"
script:
- unset SPACK_SIGNING_KEY
- pushd ${SPACK_CONCRETE_ENV_DIR} && spack env activate --without-view . && popd
- spack -d ci rebuild
rebuild-index: true
service-job-attributes:
before_script:
- git clone ${SPACK_REPO}
- pushd spack && git checkout ${SPACK_CHECKOUT_VERSION} && popd
- . "./spack/share/spack/setup-env.sh"
tags:
- k8s
- ikp
- exasgd
- basic
image:
name: spack/ubuntu-bionic
entrypoint: [""]
mappings:
- match:
- arch=linux-ubuntu18.04-x86_64
runner-attributes:
tags:
- k8s
- ikp
- exasgd
- basic
image:
name: spack/ubuntu-bionic
entrypoint: [""]