Depends on #7119.

Spack needs build testing for releases. The plan is to do this with buildbot (or similar) and CDash.

We have the rudiments of nightly testing with buildbot implemented in #2917 and spack/spack-buildbot-config. This is not enough to test full releases or to understand where build errors are coming from.

Ideally we would like to support this or a similar release workflow:

1. create a release candidate branch (releases/vX.Y.Z)
2. A project is created at spack.io/cdash for release
3. Build system reports status for "all" (see below) package builds to this new CDash project
4. Iterate on release branch until builds are all green
5. Release is merged to master and tagged

Each release would be tested with a set of OS/compiler combinations, and we should be able to specify that set in a YAML file on the release branch. We will base this format off of the test-suite YAML format introduced in #2917. It allows someone to specify a cartesian product of package specs.

We would like to get to where we are testing on multiple OS's and compilers, e.g.:

• Compilers gcc, clang, intel
• OS's: ubuntu (latest?), centos (6 and 7?), something else?

We'll start with one OS and gcc + clang for simplicity

The plan:

• Stand up a build bot (or similar) instance in LLNL AWS using Kubernetes. We want to use Kubernetes so that the CI system is easily to replacate at sites that want to contribute. This also allows people to replicate the CI system across different clouds.

• Based on the old spack/spack-buildbot-config, create a new buildbot config that works with the latest version of buildbot (that one is based on a really old version that is used by the ZFSonlinux team). Or blow that repo away and make a new config based on the implementor's CI system of choice (something we can maintain and replicate, though).

• Make our buildbot understand the test-suite.yaml format. Have it farm out jobs to EC2 instances based on a release.yaml in the root of the release branch.

• When a release branch is pushed, A new CDash project is created, and buildbot posts build results there based on release.yaml.

• Builds in the cloud should are stored as signed binary packages on completion.

  • storage for this is up to implementation, but these should not go in the public S3 repo.

• As we iterate on the release branch, package builds should install by reusing binaries from prior builds if possible (this will reduce build time).