Major slowdown when installing binaries from binary cache due to #18482 #19539
Description
Bisected to #18482 bugfix: fix config merge order for OrderdDicts
My CI script that ran spack install [spec] using a binary cache slowed down after #18482 to the point where building from sources is almost equally fast. Previously the CI job ran in 11 minutes (installing ~100 packages), now it's taking > 90 minutes and it is not even finished yet pulling binaries from the binary cache.
Ping @tgamblin.
The CI script builds a docker image as follows (adding a mirror / binary cache, copying an environment file, and then installing):
# "Install" compilers
COPY "$COMPILER_CONFIG" /opt/spack/etc/spack/compilers.yaml
# Add our custom spack repo from here
COPY ./spack /user_repo
RUN spack repo add --scope site /user_repo
# Set up the binary cache and trust the public part of our signing key
COPY ./ci/spack/public_key.asc ./public_key.asc
RUN spack mirror add --scope site minio [some url] && \
spack gpg trust ./public_key.asc
# Copy over the environment file
COPY $SPACK_ENVIRONMENT /spack_environment/spack.yaml
# Build dependencies
# 1. Create a spack environment named `ci` from the input spack.yaml file
# 2. Install only the dependencies of this (top level is our package)
RUN spack --color=always env create --without-view ci /spack_environment/spack.yaml
RUN spack --color=always -e ci install --only=dependencies