This is an expansion of the final checkbox here, which reads:

 ensure that binaries are rebuilt when changes are made to package.py files on a release branch.

I want to summarize here that entire process as I imagine it. This will support two goals:

1. Help me ensure that I actually understand the whole process
2. Allow us to discuss which specific subset of the whole process is required to satisfy the final checkbox in package.py hashing #7119.

So I imagine the entire process as follows:

1. gitlab-ci detects merge from release to master and kicks off the entire process by running some script from within the spack repository. For the purposes of this discussion I'll call that script/functionality binary-check.

2. 'binary-check' starts by doing some git-fu (similar to lib/spack/spack/cmd/flake8.py) to determine if any package.py files have changed. If not, the job of binary-check is finished. If so, it proceeds to the next step.

3. For each changed package file (call it changed-package in the steps below), binary-check should iterate over the configured mirrors (using spack mirror list?) to see if any binaries should be rebuilt and pushed to those mirrors. The specifics of this for a single remote binary mirror are expanded in the nested items below.

   1. Since a binary mirror might contain built images of many specs for a single package.py file, we need to fetch the hash for changed-package from the mirror and compare it to the local hash. I assume that a single hash on the mirror is sufficient to represent all the built specs there. I.e. if the hashes don't match, then all specs on the remote binary mirror (which are based on changed-package) need to be rebuilt. I.e. There will be one hash per package, not one hash per built spec.

   2. To do the hashing, we may need the functionality from lib/spack/spack/util/package_hash.py exposed as a command-line utility, unless perhaps binary-check is written in python and has access to the full set of spack functionality. Even in that case, it might make sense to expose the package hashing functionality on the command line, but then it could be a separate task/issue.

   3. For some changed-package discovered in step (3), if we determine that a binary mirror needs a rebuild, we need to fetch from the mirror a list of all the built packages there (likely, in the form of a list of specs, but also including which os/compiler to use), and then run those builds locally. This seems to imply we'll need to have some mapping that will tell us which Docker container to use in order to achieve a particular build (I imagine there should be a different container for each combination of OS/compiler?). Then we map our list specs to build over the appropriate set of Docker containers in order to generate all the new binaries.

   4. Once the needed binaries have been built for a given changed-package + remote binary mirror, we push those binaries into place on the remote.