spack install : added --log-format option (incorporates test-install command) by alalazo · Pull Request #2112 · spack/spack

alalazo · 2016-10-25T08:28:17Z

Modifications:

fixes Merge commands with overlapping responsibilities #1907
adds two options to install : --log-format and --log-file
default log file is written in join_path(spack.var_path, 'junit-report') instead of join_path(os.getcwd(), 'junit-report')

…command) fixes spack#1907

alalazo · 2016-10-25T08:46:33Z

@scheibelp I was wondering if you could review the changes here and see if anything looks wrong.

@adamjstewart @tgamblin @davydden @citibeth Is there any other information that you would like to see in the JunitXML report and is not there yet ?

davydden

looks good, maybe you could post here an example of test output to see how it all looks for a simple package with say 1-2 dependencies, with one of them already installed.

davydden · 2016-10-25T08:39:00Z

lib/spack/spack/cmd/install.py

+                test_case.set_result(TestResult.PASSED)
+            except InstallError:
+                # Check if the package relies on dependencies that
+                #  did not install


extra whitespace

alalazo · 2016-10-25T09:10:18Z

Ok, giving the command:

spack install --log-format=junit  netcdf

with only the root that needs to be built produces the following report in var/spack/junit-report:

<?xml version="1.0" ?>
<testsuite errors="0" failures="0" tests="11">
        <testcase classname="zlib" name="[email protected]%[email protected] arch=linux-Ubuntu14-x86_64-asydrba" time="0.0">
                <skipped message="Skipped [already installed]" type="already_installed"/>
        </testcase>
        <testcase classname="openssl" name="[email protected]%[email protected] arch=linux-Ubuntu14-x86_64-hibnfda" time="0.0">
                <skipped message="Skipped [already installed]" type="already_installed"/>
        </testcase>
        <testcase classname="curl" name="[email protected]%[email protected] arch=linux-Ubuntu14-x86_64-2ffacqm" time="0.0">
                <skipped message="Skipped [already installed]" type="already_installed"/>
        </testcase>
        <testcase classname="libsigsegv" name="[email protected]%[email protected] arch=linux-Ubuntu14-x86_64-g3qpmbi" time="0.0">
                <skipped message="Skipped [already installed]" type="already_installed"/>
        </testcase>
        <testcase classname="m4" name="[email protected]%[email protected]+sigsegv arch=linux-Ubuntu14-x86_64-lhgqa6s" time="0.0">
                <skipped message="Skipped [already installed]" type="already_installed"/>
        </testcase>
        <testcase classname="pkg-config" name="[email protected]%[email protected] arch=linux-Ubuntu14-x86_64-n6bkle3" time="0.0">
                <skipped message="Skipped [already installed]" type="already_installed"/>
        </testcase>
        <testcase classname="libpciaccess" name="[email protected]%[email protected] arch=linux-Ubuntu14-x86_64-eo2siet" time="0.0">
                <skipped message="Skipped [already installed]" type="already_installed"/>
        </testcase>
        <testcase classname="hwloc" name="[email protected]%[email protected] arch=linux-Ubuntu14-x86_64-3ostwel" time="0.0">
                <skipped message="Skipped [already installed]" type="already_installed"/>
        </testcase>
        <testcase classname="openmpi" name="[email protected]%[email protected]~mxm~pmi~psm~psm2~slurm~sqlite3~thread_multiple~tm~verbs+vt arch=linux-Ubuntu14-x86_64-s3qbtby" time="0.0">
                <skipped message="Skipped [already installed]" type="already_installed"/>
        </testcase>
        <testcase classname="hdf5" name="[email protected]%[email protected]+cxx~debug+fortran+mpi+shared~szip~threadsafe arch=linux-Ubuntu14-x86_64-3a22tnq" time="0.0">
                <skipped message="Skipped [already installed]" type="already_installed"/>
        </testcase>
        <testcase classname="netcdf" name="[email protected]%[email protected]~hdf4 maxdims=1024 maxvars=8192+mpi arch=linux-Ubuntu14-x86_64-x7rkaqv" time="36.8761630058"/>
</testsuite>

davydden · 2016-10-25T09:14:54Z

@alalazo thanks, looks good. I don't have in mind any further information we would need.

Supposedly, there could be a script somewhere which can parse numerous XML (from different users who contribute test results) into a single output attached to a release? Maybe a document, html page or something that could be easy to navigate for a user to see which packages are tested on which architectures and which compilers for that particular release.

davydden · 2016-10-25T09:16:27Z

p.s. I do not suggest to do this within the current PR, just throwing ideas on the table.

alalazo · 2016-10-25T10:19:43Z

@davydden To be fair @scheibelp implemented JUnitXML in the first place. I am just refactoring the stuff to reduce the number of similar commands in Spack without loosing the functionality. That said I think it will be an essential building block for a release installation test.

What we are currently missing for that (and won't likely happen before SC) is imo:

a file format to describe a matrix of things to be try-installed for the release
the logic to trigger all those tests and create a report
a dashboard online that can display JUnitXML submitted from the various groups

I know there are some discussions that touched these topics in latest issues, but couldn't find them right now...

davydden · 2016-10-25T10:24:33Z

To be fair @scheibelp implemented JUnitXML in the first place. I am just refactoring the stuff to reduce the number of similar commands in Spack without loosing the functionality

I understand that, i was trying to see how would one later use this functionality for testing.

a dashboard online that can display JUnitXML submitted from the various groups

essentially i was speaking about this. Something like CDash would be good.

I know there are some discussions that touched these topics in latest issues, but couldn't find them right now...

One of them is #2014 , i suppose...

davydden · 2016-10-25T12:19:25Z

@alalazo : I think there is one more thing we have to log -- hash of the Spack and maybe the most recent tag. Maybe within testsuite errors="0" failures="0" tests="11" hash="xyz" tag="abc".

tgamblin · 2016-10-25T15:08:46Z

So, I could set up a CDash instance in EC2 so we could at least start reporting test results. It apparently has some support NUnit format so we could start by adding a few fields to the current test format. What do people think of that?

I am worried that the matrix display stuff is going to require more infrastructure than we have. I think Spack ultimately needs its own dashboard of some sort, where we could do spack find-ish queries on test results and query a server-side database of spec.yamls. But that's not happening before SC.

citibeth · 2016-10-25T18:15:14Z

What do I do with this xml file? It looks interesting but is not really
readable without further processing.
On Oct 25, 2016 5:10 AM, "Massimiliano Culpo" [email protected]
wrote:

Ok, giving the command:

spack install --log-format=junit netcdf

with only the root that needs to be built produces the following report in
var/spack/junit-report:

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2112 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB1cd0dGzlHqEAbzvQmT2cC9gIDmHgE8ks5q3cd9gaJpZM4KfseA
.

tgamblin · 2016-10-25T18:18:52Z

@citibeth: you submit it to Bamboo, Jenkins, or CDash.

alalazo · 2016-10-25T18:22:30Z

@citibeth All major CI framework supports that to display results in various way.

davydden · 2016-10-25T18:29:20Z

@alalazo : you probably missed my comment above, but do we need to encode Spack's hash used to create tests? I would say yes.

alalazo · 2016-10-25T18:53:16Z

@davydden I didn't miss the message. I just need to have a look at the NUnit format to see how they support writing metadata (if they do at all).

davydden · 2016-10-25T18:54:31Z

i see, no worries.

citibeth · 2016-10-25T19:02:03Z

This will be pretty awesome.

On Tue, Oct 25, 2016 at 2:22 PM, Massimiliano Culpo <
[email protected]> wrote:

@citibeth https://github.com/citibeth All major CI framework supports
that to display results in various way.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2112 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB1cd9FSugqz7wf1wzedlTIMyU0XxUeoks5q3kjrgaJpZM4KfseA
.

tgamblin · 2016-10-26T21:16:30Z

@alalazo: do you want this merged now to update NUnit later, or do you want to keep it open?

alalazo · 2016-10-26T21:19:34Z

@tgamblin Fine with me to merge this and update later

hegner · 2016-11-29T14:01:37Z

With #2433 the minimal requirements for uploading to CDash are now there. See https://cdash.cern.ch/index.php?project=Spack

For now, that dashboard is free for submission via e.g.
curl -k --upload-file var/spack/junit-report/test-zlib-1.2.8-pfy5ppjohs2f7cmt3br4bozg7rd4zack.xml https://cdash.cern.ch/submit.php?project=Spack

spack install : added --log-format option (incorporates test-install …

007c1bc

…command) fixes spack#1907

davydden reviewed Oct 25, 2016

View reviewed changes

qa : removed extra whitespace

5fe68c1

tgamblin added the tests General test capability(ies) label Oct 26, 2016

tgamblin merged commit e73ab84 into spack:develop Oct 26, 2016

alalazo deleted the refactoring/test_install_merged_with_install branch October 26, 2016 21:28

davydden mentioned this pull request Nov 8, 2016

spack install --log-format=junit : record if --run-tests was used #2277

Closed

davydden mentioned this pull request Nov 29, 2016

add NUnit support to --log-format #2432

Closed

Conversation

alalazo commented Oct 25, 2016

Modifications:

Uh oh!

alalazo commented Oct 25, 2016

Uh oh!

davydden left a comment

Choose a reason for hiding this comment

Uh oh!

davydden Oct 25, 2016

Choose a reason for hiding this comment

Uh oh!

alalazo commented Oct 25, 2016

Uh oh!

davydden commented Oct 25, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davydden commented Oct 25, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alalazo commented Oct 25, 2016

Uh oh!

davydden commented Oct 25, 2016

Uh oh!

davydden commented Oct 25, 2016

Uh oh!

tgamblin commented Oct 25, 2016

Uh oh!

citibeth commented Oct 25, 2016

Uh oh!

tgamblin commented Oct 25, 2016

Uh oh!

alalazo commented Oct 25, 2016

Uh oh!

davydden commented Oct 25, 2016

Uh oh!

alalazo commented Oct 25, 2016

Uh oh!

davydden commented Oct 25, 2016

Uh oh!

citibeth commented Oct 25, 2016

Uh oh!

tgamblin commented Oct 26, 2016

Uh oh!

alalazo commented Oct 26, 2016

Uh oh!

hegner commented Nov 29, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

davydden commented Oct 25, 2016 •

edited

Loading

davydden commented Oct 25, 2016 •

edited

Loading