@hegner @tgamblin In the future we may specialize even further for CDash (if it is the tool we are going to use for the whole project) and fill the Configure and Build columns appropriately.

@alalazo - yes. That would be great. In that case we would need to define what we want to have in the fields and what the granularity should be (single packages or given subsets?)

@tgamblin - so much about WIP. That was quick :-)

@hegner: ack didn't notice that! Was it not done?

@tgamblin - it is functional as merged, but some cleanups should go in still. E.g. I see that cdash has a size limit in the build name. I will just open another PR once that is there.

I'm concerned about how we can identify Spack package builds for CDash in a way that will let us navigate/view/query them. We can always use a hash for that, but that makes it hard to see what's going on at the top level. But I think we have to try it first to see what'll work.

Yes, that is a problem indeed. I could very well imagine the following:

• Canonical sets with expressive names going into a nightly 'validation' slot. And then we can distribute to load to volunteers.
• Individual packages going into another slot which may be less organized

And then we take it from there...

@tgamblin - here you see the test results of our poor man's package testing:
https://cdash.cern.ch/index.php?project=Spack

not putting any logic into it yet it is a flat list of whatever we build. I'd propose conventions/ etc we can discuss during the next telcon.

And - you are free using the same instance.

@hegner: Thanks! This looks awesome -- it looks like we need to tweak the format to get the configure/build output.

I'm curious whether if we had a dedicated Spack CDash, whether we could use the projects to divide things up into canonical sets/stacks. One project could be tip of develop, one could be intel stack, gcc stack, etc., and those could be "stable", bleeding edge, particular spack versions, etc.

Just having a straw man to play with is really helpful. Thanks again!

@tgamblin - it depends on how many axes you want to have. What we do w/ our spack-predecessor is this:
https://cdash.cern.ch/index.php?project=LCGSoft

We have slots for different configurations of the build (package versions+branch of recipes) and within each slot the different compiler/platform combinations. Given that testing coverage is usually a very sparse matrix, it worked well so far.

Just dreaming a bit. A feature like
spack test-release --set [python3, R, ... ] --upload_url <instance>
would allow any user to participate in validation

Concerning the empty columns - yes, the junit support is bad. If we want to follow this direction, we should implement the proper cdash format

@hegner: That actually looks very good.

I think CDash is a really good way to go. I like the test-release command - how do you want the user to specify combinations/variants? More specs on the command line? I think we could provide a default upload URL to a central CDash instance, so the user doesn't have to provide the URL.

@tgamblin - variants is a good question. Probably set%compiler+variants would be a good way. For a common URL - yes, I agree. You want to make it as simple as possible to contribute.

@hegner: We've discussed in #2014 the idea of having a common format to specify combinatorial sets -- sort of how Travis allows it. I think we could implement the simple command you suggest then add more capabilities as needed.

@tgamblin - great. will follow up there

@hegner I am using your cdash logging format. I am uploading them to spack.io cdash but I am not able to see the results. In your example you provide a link to your cdash. I am not able to produce the same overview output on cdash. Do you have an example of the xml files you uploaded to your cdash? I feel like I am missing a flag. My xml output is below:

<Site BuildName="[email protected]%[email protected] arch=darwin-elcapitan-x86_64-gh6bktr" BuildStamp="20171001-14:45:08-Experimental" Name="Computer1" Type="Experimental">
	<Configure>
		<StartDateTime>Tue Jan 10 14:45:08 PST</StartDateTime>
		<EndDateTime>Tue Jan 10 14:45:08 PST</EndDateTime>
		<ConfigureCommand>spack install</ConfigureCommand>
		<ConfigureStatus>0</ConfigureStatus>
		<ElapsedMinutes>0.0</ElapsedMinutes>
		<Log>[email protected]%[email protected] arch=darwin-elcapitan-x86_64</Log>
	</Configure>
</Site>