Spack has self tests, and these are fine, and I'm sure these are useful. But what I miss are full system tests, where a new version of Spack is checked out, a non-trivial set of packages are built, and various Spack commands are run on the install. I've implemented a Bash script that does this https://raw.githubusercontent.com/eschnett/spack-test/master/spacktest-ubuntu16.sh, along the lines of:

• clone Spack
• build GCC
• point Spack to this GCC
• build a set of "important" HPC packages (HDF5, MPI, OpenBLAS, Python)
• run some simple tests on these packages
• build a set of tools (git, lmod, rsync)
• run some simple tests with these tools

My plan is to submit issues (and/or put work-arounds into my script) until things work. Then we can run the script once a day, and every time it succeeds, we tag the respective Spack checkout as "good". Every time it doesn't, there's a bug report.