Add spack test-suite command for posting combinatorial tests to CDash by kielfriedt · Pull Request #2917 · spack/spack

kielfriedt · 2017-01-24T00:39:44Z

Added testsuite which reads in a yaml file. Example yaml file is located in lib/spack/docs/tutorial/examples/. Changed dashboard to simple and complete versions. Current requirements are for build only, but in the future a complete configure, build and test would be needed. Testsuite is documented and tested.

kielfriedt · 2017-01-24T00:47:56Z

I thought the changes from @hegner for cdash were merged. I guess I have a conflict on install.py

tgamblin · 2017-01-24T03:45:32Z

@kielfriedt: nope -- but they're in your branch, so they'll be included when we merge this.

tgamblin

@kielfriedt: looks pretty good! Can you address the various inline comments and the following requests?

Can you rename spack testsuite to spack test-suite -- to match the formatting of the other commands?
Can you add docs on how to use the command and on the YAML format to the testing guide?
flake8 is going to complain about formatting in cmd/teststuite.py so you'll need to fix the style errors. You can run spack flake8 locally to fix that. See the contributor guide

tgamblin · 2017-01-24T03:51:21Z

lib/spack/spack/cmd/testsuite.py

+Get the list of compilers from spack found on the system.
+Compares the spack list to test file.
+If a compiler is found on both lists its returned.
+'''


Needs to be proper docstring, which goes inside the function after the def line.

See PEP257 or this.

tgamblin · 2017-01-24T03:51:37Z

lib/spack/spack/cmd/testsuite.py

+
+
+'''
+fucntion Name: removeTests


moved to doc strings

tgamblin · 2017-01-24T04:00:53Z

lib/spack/spack/cmd/testsuite.py

+    pkg = ""
+    compiler=""
+    teststoRemove = []
+    if "%" in exclusion:


Parse this stuff using the Spec constructor? It should make your life easier and you don't have to do all the string analysis.

You can just pass it a string, e.g.:

from spack.spec import Spec spec = Spec('foo @1.0 %[email protected]')

Then you can do things like this:

print spec.satisfies('%clang@3') >>> True

Im not sure this is what I am doing. This is for exclusion, by doing the string analysis so I can determine how to properly test for the exclusion in the set of tests.

tgamblin · 2017-01-24T04:03:18Z

lib/spack/spack/cmd/testsuite.py

+    path = ""
+    dashboards = []
+    #designed to use a single file and modify the enabled tests, thus requiring a single file modification.
+    for files in args.yamlFile: #read yaml files which contains description of tests


use a singular variable name -- each element in args.yamlFile is a file, not a "files". 😄

tgamblin · 2017-01-24T04:05:06Z

lib/spack/spack/cmd/testsuite.py

+    for files in args.yamlFile: #read yaml files which contains description of tests
+        with open(files, 'r') as stream:
+            try:
+                yamlFile= yaml.load(stream)


Convert this to use spack.util.spack_yaml, e.g.:

import spack.util.spack_yaml as syaml with open(file) as stream: syaml.load(stream)

Spack's YAML adds support for a few nice things like line numbers (which you can use in error messages) and ordered dictionaries.

tgamblin · 2017-01-24T04:28:13Z

lib/spack/spack/cmd/testsuite.py

+            args = parser.parse_args([cdash]) #use cdash-complete if you want configure, build and test output.
+            args.package = test
+            install.install(parser, args)
+        except Exception as ex:


Whatever error occurs should probably go in the test log and get displayed as part of CDash output.

tgamblin · 2017-01-24T04:29:10Z

lib/spack/spack/cmd/testsuite.py

+        except Exception as ex:
+            template = "An exception of type {0} occured. Arguments:\n{1!r}"
+            message = template.format(type(ex).__name__, ex.args)
+            tty.msg(message)


Instead of printing, put errors in the XML.

I will have to look into this. cdash files are produced by logging. I am not sure I can put it into the xml file but ill figure it out.

tgamblin · 2017-01-24T04:31:46Z

lib/spack/spack/hooks/dashboards.py

+        buildName = buildName.split('=')
+        template.set('BuildName',str(buildName[0]) + " " + str(buildName[1]))
+        template.set('BuildStamp', self.buildstamp)
+        tempcompiler = self.spec.short_spec.split('%')[1].split(' ')[0]


avoid using split and use the actual fields on . e.g. self.spec.compiler.name, self.spec.compiler.version, etc.

done. Wasn't aware of that option.

tgamblin · 2017-01-24T04:34:58Z

lib/spack/spack/hooks/dashboards.py

+
+
+    def prepare_build_report(self):
+        report = self.create_template()


I'm curious whether the XML parts of this would be more maintainable if you just make an XML template with """, e.g.:

template = """\ <Build> <StartDateTime>{date}</StartDateTime> <StartDateTime>{log_text}</StartDateTime> </Build> """ output = template.format(date=date, log_text=captured_output)

etc. You can get pretty far with Python's str.format() function.

What do you think? I think manually constructing the ElementTree is kind of tedious and hard to read or change.

This is done by @hegner I did not redo it. I just adjusted the format to get it to output a file that would display on cdash. I will talk with hegner for advice.

tgamblin · 2017-01-24T04:37:15Z

lib/spack/spack/hooks/dashboards.py

+    return decorator
+
+
+


too much space.

kielfriedt · 2017-02-16T01:41:52Z

Add done with changes as requested and tested on my end.

tgamblin · 2017-03-20T22:41:45Z

@kielfriedt: can you rebase this and/or merge current develop?

kielfriedt · 2017-03-21T21:14:30Z

@tgamblin I am now using testing/wip_xsdk, I got rid of the wip branch. Should I make a new pull request?

tgamblin · 2017-03-21T21:21:47Z

@kielfriedt: Well, you got rid of your local WIP branch, not the one on GitHub.

We want to keep these features separate from the xsdk ones, so that if we merge your feature first, it doesn't conflict with Barry's xsdk branch, or vice versa. When integrating stuff like this you should keep your changes on one branch, then make a new integration branch where you keep merging your feature branch and the one you want to integrate with.

It sounds like you should cherry-pick your work on the testing/wip_xsdk branch onto this one, and push to the wip branch on the server. Then the new stuff will show up on the PR.

tgamblin

@kielfriedt: This is looking better! I added a bunch of comments Can you:

Address the comments
Fix the PEP8 and doc warnings

Nice work!

tgamblin · 2017-03-22T00:04:17Z

lib/spack/spack/cmd/test_suite.py

+        return spec, "PackageStillNeededError"
+    return spec, ""
+
+def installSpec(spec,cdash,test):


use_this_format notThisOne

tgamblin · 2017-03-22T00:04:42Z

lib/spack/spack/cmd/test_suite.py

+        cdash = '--log-format=cdash-simple'
+
+    cdash_root = "/var/spack/cdash/"
+    enabledTests = []


enabled_tests

tgamblin · 2017-03-22T00:37:42Z

lib/spack/spack/cmd/test_suite.py

+                compilers = data['compilers']
+                exclusions = data['exclusions']
+            except:
+                tty.msg("Testing yaml files must contain atleast enable, packages, exclusions and compilers to produce results. Exclusions can be empty, IE []")


Isn't this warning unnecessary? The schema validation should ensure this never happens.

tgamblin · 2017-03-22T00:38:24Z

lib/spack/spack/schema/test.py

+                },
+            },
+        },
+        'exclusions': {


Exclusions should be optional (might not want to exclude anything, and requiring [] is tedious for the user)

tgamblin · 2017-03-22T00:46:41Z

lib/spack/spack/schema/test.py

+            'default': [],
+            'items': {'type': 'string'}
+        },
+        'dashboard': {


I think the dashboard ought to be optional, or maybe only specified at the command line.

Also, can you change this to use include and exclude instead of enable and exclusions? Each of those should take specs. Example:

spack-tests: dashboard: # optional string include: # use this instead of "enable" # specs to include (any others are assumed excluded. Default to all included) exclude: # specs to exclude (from set of included specs) packages: # ... compilers: # ... dependencies: # TBD -- we need this, but ignore for now (we'll figure out how to factor in later) variants: # TBD -- need this too, but ignore for now. not clear if it is global or if it's a per-package setting or both.

tgamblin · 2017-03-22T01:06:28Z

lib/spack/spack/cmd/test_suite.py

+        help='using this option switches from simple cdash output to compelet: simple is only build, complete is configure, build and test xml output.')
+    subparser.add_argument(
+        'yamlFile', nargs=argparse.REMAINDER,
+        help="yaml file that contains a list of tests, example yaml file can be found in /lib/spack/docs/tutorial/examples/test.yaml")


The file's really long and it's buried in the source -- I think we shouldn't assume that users can dig through the source directories. Can you remove the reference to the file and just say "see docs on test suites" for now?

Since you're adding a new command, can you add tab completion as well? It would look like:

function _spack_test-suite { if $list_options then compgen -W "-c --complete" -- "$cur" fi }

tgamblin · 2017-03-22T01:08:45Z

lib/spack/docs/tutorial/examples/test.yaml

@@ -0,0 +1,2231 @@
+---


The file is too long -- if it's this long it should really just be generated. Can you just remove this and link to the docs in the command help where this is referenced?

Also, lib/spack/docs/tutorial/examples is really just for our SC'16 tutorial. Examples like this should probably get their own directory separate from that.

shortened and moved.

tgamblin · 2017-03-22T01:09:18Z

lib/spack/docs/testing_guide.rst

+with versions. Using the enabled field will allow you to focus on specific packages.
+To narrow down the scope of a package or compiler, you can use the exclusion field.
+
+--Example of a yaml file---


Call this a "test suite file" or a "test suite YAML file" or something more specific.

The file format should start with a top-level key like other YAML files in Spack. Can you make it start with test-suite: or something, to identify the type of file to Spack?

For the yaml file, use:

.. code-block:: yaml

and indent it.

tgamblin · 2017-03-22T01:11:49Z

lib/spack/docs/testing_guide.rst

+
+dashboard: ["https://spack.io/cdash/submit.php?project=spack"]
+
+path: "~/home/username"


Output path should be a parameter of the command -- not a hard-coded value in the file. I think it should be possible to either specify an existing output directory on the command line, or to run without output and just spit the results out to a new directory within the current working dir, named, e.g., spack-tests-YYYY-MM-DD-timestamp. See spack mirror create for similar semantics.

timestamp = datetime.now().strftime("%Y-%m-%d")
directory = 'spack-mirror-' + timestamp

current output from dashboard is similar to this now.

adamjstewart · 2017-03-22T02:46:29Z

lib/spack/spack/cmd/test_suite.py

@@ -0,0 +1,322 @@
+#!/usr/bin/env python


Needs a license

adamjstewart · 2017-03-22T02:51:38Z

lib/spack/spack/cmd/test_suite.py

+        help='using this option switches from simple cdash output to compelet: simple is only build, complete is configure, build and test xml output.')
+    subparser.add_argument(
+        'yamlFile', nargs=argparse.REMAINDER,
+        help="yaml file that contains a list of tests, example yaml file can be found in /lib/spack/docs/tutorial/examples/test.yaml")


Since you're adding a new command, can you add tab completion as well? It would look like:

function _spack_test-suite { if $list_options then compgen -W "-c --complete" -- "$cur" fi }

adamjstewart · 2017-03-22T02:53:49Z

lib/spack/docs/testing_guide.rst

+with versions. Using the enabled field will allow you to focus on specific packages.
+To narrow down the scope of a package or compiler, you can use the exclusion field.
+
+--Example of a yaml file---


For the yaml file, use:

.. code-block:: yaml

and indent it.

adamjstewart · 2017-03-24T12:52:39Z

lib/spack/spack/cmd/test_suite.py

+                                            data=mydata,
+                                            headers={'content-type':'text/plain'},
+                                            params={'file': cdash_path + file}
+                                            )


4 space indentation, not tabs.

kielfriedt

changes have been completed

kielfriedt · 2017-03-23T20:41:22Z

lib/spack/docs/testing_guide.rst

+
+dashboard: ["https://spack.io/cdash/submit.php?project=spack"]
+
+path: "~/home/username"


timestamp = datetime.now().strftime("%Y-%m-%d")
directory = 'spack-mirror-' + timestamp

current output from dashboard is similar to this now.

kielfriedt · 2017-03-24T00:39:16Z

lib/spack/docs/testing_guide.rst

+with versions. Using the enabled field will allow you to focus on specific packages.
+To narrow down the scope of a package or compiler, you can use the exclusion field.
+
+--Example of a yaml file---


kielfriedt · 2017-03-24T00:39:57Z

lib/spack/docs/tutorial/examples/test.yaml

@@ -0,0 +1,2231 @@
+---


shortened and moved.

kielfriedt · 2017-03-24T00:42:37Z

lib/spack/spack/cmd/test_suite.py

@@ -0,0 +1,322 @@
+#!/usr/bin/env python


kielfriedt · 2017-03-24T00:47:04Z

lib/spack/spack/cmd/test_suite.py

+        help='using this option switches from simple cdash output to compelet: simple is only build, complete is configure, build and test xml output.')
+    subparser.add_argument(
+        'yamlFile', nargs=argparse.REMAINDER,
+        help="yaml file that contains a list of tests, example yaml file can be found in /lib/spack/docs/tutorial/examples/test.yaml")


kielfriedt · 2017-03-24T00:56:16Z

lib/spack/spack/cmd/test_suite.py

+        return spec, "PackageStillNeededError"
+    return spec, ""
+
+def installSpec(spec,cdash,test):


kielfriedt · 2017-03-24T00:56:22Z

lib/spack/spack/cmd/test_suite.py

+        cdash = '--log-format=cdash-simple'
+
+    cdash_root = "/var/spack/cdash/"
+    enabledTests = []


kielfriedt · 2017-03-24T00:56:30Z

lib/spack/spack/cmd/test_suite.py

+                compilers = data['compilers']
+                exclusions = data['exclusions']
+            except:
+                tty.msg("Testing yaml files must contain atleast enable, packages, exclusions and compilers to produce results. Exclusions can be empty, IE []")


kielfriedt · 2017-03-24T00:56:43Z

lib/spack/spack/schema/test.py

+                },
+            },
+        },
+        'exclusions': {


kielfriedt · 2017-03-24T00:56:59Z

lib/spack/spack/schema/test.py

+            'default': [],
+            'items': {'type': 'string'}
+        },
+        'dashboard': {


tgamblin · 2017-03-28T15:59:01Z

@adamjstewart: are you ok with these changes?

adamjstewart · 2017-03-28T17:33:59Z

lib/spack/docs/example_files/test.yaml

@@ -0,0 +1,22 @@
+---


What is the --- for?

I think that is the standard start of a YAML file (see http://yaml.org/), but all spack YAML files start with <name>:. We should change this.

--- means: is this document part of a single stream (or not): https://en.wikipedia.org/wiki/YAML

adamjstewart · 2017-03-28T17:34:19Z

lib/spack/docs/example_files/test.yaml

@@ -0,0 +1,22 @@
+---
+test-suite:
+    include: [ ape, atompaw, transset]


Can you remove the space between [ and ape?

adamjstewart · 2017-03-28T17:34:30Z

lib/spack/docs/example_files/test.yaml

+---
+test-suite:
+    include: [ ape, atompaw, transset]
+    exclude: [binutils,tk]


Can you add a space after the comma?

adamjstewart · 2017-03-28T17:36:16Z

lib/spack/docs/testing_guide.rst

+
+   $ spack install --log-format=junit <spec>
+
+Per default the log results will be placed into `var/<format>/test-<short_spec>.xml`.


Single backticks make things italic, you probably want double backticks to make it monospace.

adamjstewart · 2017-03-28T17:36:42Z

lib/spack/docs/testing_guide.rst

+
+
+----------------
+using test-suite


Using should probably start with a capital U.

adamjstewart · 2017-03-28T17:37:31Z

lib/spack/docs/testing_guide.rst

+
+   ---
+   test-suite:
+   enable: [bzip2, libelf, .. ,libdwarf]


Shouldn't everything after test-suite be indented further?

adamjstewart · 2017-03-28T17:41:17Z

lib/spack/docs/testing_guide.rst

+
+  $ spack test-suite -c /location/of/yamlFile
+
+Currently output files are stored in you current working directory. A folder name spack-test-YYYY-MM-DD this may change in the future.


adamjstewart · 2017-03-28T17:41:30Z

lib/spack/docs/testing_guide.rst

+Junit
+^^^^^
+
+With 


Add a colon to the end.

adamjstewart · 2017-03-28T17:41:52Z

lib/spack/docs/testing_guide.rst

+^^^^^
+
+Spack supports the XML format used by `CDash <http://www.cdash.org/>`_.
+To create reports in this format, do


Add a colon to the end.

adamjstewart · 2017-03-28T17:42:19Z

lib/spack/docs/testing_guide.rst

+
+   $ spack install --log-file report --log-format=cdash <spec>
+
+This will produce each one file for the configure, build, and test step.


adamjstewart · 2017-03-28T17:42:44Z

lib/spack/docs/testing_guide.rst

+
+   $ curl --upload-file report.build.xml <cdash url>/submit.php?project=<projectname>
+   $ curl --upload-file report.configure.xml <cdash url>/submit.php?project=<projectname>
+   $ curl --upload-file report.test.xml <cdash url>/submit.php?project=<projectname>


I wonder if we can add a Spack command to automate this?

tgamblin · 2017-04-15T06:56:46Z

@kielfriedt: The libstdc++ issue you were having is fixed by #3854 (or by removing [email protected] from your compilers.yaml on your Mac) , and the concretization issues were fixed by merging the latest develop. I'm going to clean this up over the weekend and get it merged into the mainline.

tgamblin · 2017-04-18T10:22:43Z

@kielfriedt: I reworked this. Can you look at the code and comments in 7b50e15 and let's go over it tomorrow. I think this is pretty close to ready.

TODO:

docs need updating as I changed the test-suite format to be more generic.
test-suite shouldn't be using requests. Use curl instead to be consistent (and to not require a new dependency)

alalazo · 2017-04-18T15:55:05Z

lib/spack/docs/testing_guide.rst

+             gcc:
+               versions: [4.9.0, 4.7.1, 4.6.3, 4.6.1]
+             clang:
+               versions: [7.3.0, 3.4, 3.1]


I was taking a brief look at this PR, and it looks great. Thanks @kielfriedt !

A random thought: if in the future we'll be able to extend this yaml file to treat virtual dependencies and similar things correctly, we may use it beyond build tests. What I am thinking is reproducing the build of an entire environment on a different machine. Roughly:

spack export ... # Exports a YAML file similar to this one # Move to another machine... spack install -f suite.yaml # Reinstall the "same" software, for some notion of "same"

I know a lot of Spack users are currently creating massive dummy-packages to do something similar to this. A suite.yaml file would be much cleaner.

That's the intent with spack.util.spec_set. The way I've reworked it, you can basically cartesian-product arbitrary lists of specs. The compilers and packages elements are shorthand for particular types of spec lists.

The schema is currently written with spack test-suite in mind, but it can be factored out to a suite.yaml or maybe stack.yaml. Which do you like better? I kind of like stack.yaml and thought about calling the format spack-stack instead of test-suite.

No preference.

alalazo · 2017-05-12T16:35:48Z

@kielfriedt I think you need to update the command test-suite for the Travis test to pass.

- Created new command called testsuite. - Takes a yaml file describing tests, produces XML and sends to CDash. - two XML formats: simple and complete -. simple is only build - complete is configure, build, test. - added an example yaml file - CDash builds display with platform icons. - Added documentation to the testing_guide for the test-suite command - document include, exclude, packages, compilers elements - test-suite takes options to display multiple dashboards. - added tests for test-suite command

- Split out CombinatorialSpecSet into its own spack.util.spec_set module. - class is iterable and encaspulated YAML parsing and validation. - Get rid of copy/pasted code in test-suite command. - consolidate line-numbered YAML error code into spack.schema. - Rework test-suite schema format - previous format wasn't actually validating -- only had definitions. - Adjust YAML format to be more generic - YAML test-suite format now has a `matrix` section, which can contain multiple lists of specs, generated different ways. Including: - specs: a raw list of specs. - packages: a list of package names and versions - compilers: a list of compiler names and versions - All of the elements of `matrix` are dimensions for the build matrix; we take the cartesian product of these lists of specs to generate a build matrix. This means we can add things like [^mpich, ^openmpi] to get builds with different MPI versions. It also means we can multiply the build matrix out with lots of different parameters. - clean up logic in test-suite command - Bug fixes: - fix some bugs with compilers_for_arch() - fix bug with constraining an anonymous spec by name, add a test.

pramodskumbhar · 2017-09-14T06:07:39Z

@kielfriedt : this is awesome and would like to try! it would be great if pending issues solved and get merged.

tgamblin · 2017-09-14T06:08:58Z

@pramodk: we're revising this and it'll be in by the end of the month.

pramodskumbhar · 2017-09-14T06:28:24Z

thanks @tgamblin. With the current format below, seems like not possible to specify spec saying hdf5%intel ^zlib%gcc (?).

  test-suite:
    matrix:
     - packages:
       libelf:
         versions: [0.8.12]
       libdwarf:
         versions: [0.8.12]
 
     - compilers:
       clang:
         versions: [7.0.2-apple]
       gcc:
         versions: [6.2.0]

I see that test-suite is more for testing individual packages. But for deployment purpose similar functionality will be useful? I am using space-packagelist from @alalazo which is very useful. (just pointing out in case relevant here as well).

tgamblin · 2017-09-14T08:13:43Z

@pramodk: I pushed my WIP branch, which still needs rebasing on develop, but it supports another syntax where you can take cartesian products of arbitrary spec lists. It's not currently documented, but I think it'll let you do what you want, e.g.:

matrix:
    - specs: ['hdf5%intel', 'hdf5%gcc']
    - specs: ['^zlib%intel', '^zlib%gcc']

That would concretize each of these (the products), and try to build the result:

hdf5%intel ^zlib%intel
hdf5%intel ^zlib%gcc
hdf5%gcc ^zlib%intel
hdf5%gcc ^zlib%gcc

You could add another specs element if you, say, wanted to try all of them with and without MPI, and you could use include and exclude to poke holes or slice the matrix if you wanted something less dense.

pramodskumbhar · 2017-10-14T09:48:24Z

I was testing this today and come across following issues :

--log-format=cdash-complete : only cdash instead (docs).
if I do spack install -v --log-format=cdash then I expected reports to be in var/spack/cdash-report similar to junit. But they get written in the current directory.
when I used default spack cdash server, I saw error/warning like :

<cdash version="2.5.0">
 <status>ERROR</status>
 <message>Not a valid project.</message>
</cdash>

when I removed cdash key from yaml file then I see:

Cannot open file (/opt/cdash/CDash/backup/spack_bluebrain355-epfl-ch_mod2c-devopt-gcc-4-4-arch-darwin-sierra-x86_64-rhiljpp_20171410-11:00:36-Experimental_150797163783.62_Build.xml)

I should able to run test-suite without considering cdash server details?

I see messages like :

==> Creating mock database for building.
....
==> Restoring user database.

what happens if I cancel test-suite in the middle?

It will be of great help if I can see "preview" of test-suite before building it. I mean something like spack spec -I for every combination that Spack is going to build. Currently I don't get an idea about combinations that are going to be built (as a new user).
Not sure where I can specify package variant to build:

    matrix:
    - packages:
        coreneuron:
            versions: [develop]

(docs update)

tgamblin · 2018-02-21T21:57:35Z

Superseded by #7114.

tgamblin changed the title ~~Wip~~ Add spack testsuite command for posting combinatorial tests to CDash Jan 24, 2017

tgamblin requested changes Jan 24, 2017

View reviewed changes

tgamblin added tests General test capability(ies) commands labels Jan 24, 2017

tgamblin changed the title ~~Add spack testsuite command for posting combinatorial tests to CDash~~ Add spack test-suite command for posting combinatorial tests to CDash Jan 24, 2017

tgamblin mentioned this pull request Jan 30, 2017

Add first version of real cdash logging #2445

Closed

tgamblin requested changes Mar 22, 2017

View reviewed changes

adamjstewart requested changes Mar 22, 2017

View reviewed changes

kielfriedt force-pushed the wip branch from 5f56f7b to d2244e6 Compare March 24, 2017 01:37

adamjstewart reviewed Mar 24, 2017

View reviewed changes

kielfriedt commented Mar 27, 2017

View reviewed changes

adamjstewart reviewed Mar 28, 2017

View reviewed changes

adamjstewart requested changes Mar 28, 2017

View reviewed changes

tgamblin force-pushed the wip branch from 7b50e15 to e2aae80 Compare April 18, 2017 10:25

alalazo reviewed Apr 18, 2017

View reviewed changes

tgamblin force-pushed the wip branch from e2aae80 to c108d97 Compare April 18, 2017 16:01

alalazo mentioned this pull request Apr 18, 2017

Getting Updates to Spack Packages Quickly #1774

Closed

alalazo mentioned this pull request Apr 27, 2017

Separate integration tests; simplify test scripts #4006

Merged

4 tasks

tgamblin force-pushed the wip branch from ab303b1 to 7a25d7f Compare May 23, 2017 09:57

add first version of cdash reporting

21bc35d

Benedikt Hegner and others added 8 commits June 28, 2017 20:57

add blank line as required by flake8 checker

4c6a79c

spack test-suite uses curl instead of requests.

65f0fe7

Make test-suite interruptible and fail if matrix yields anonymous specs.

4457594

Bug fixes.

f9aecd2

Refactor dashboard and CDash output.

9e7df93

Revert broken formatting in spec.py

4ffe000

pramodskumbhar mentioned this pull request Sep 14, 2017

No support for cdash log format? #5350

Closed

tgamblin added the WIP label Sep 14, 2017

WIP

6dbd429

tgamblin force-pushed the wip branch from f56eb28 to 6dbd429 Compare September 14, 2017 08:07

tgamblin modified the milestones: v1.0, v0.12.0 Nov 12, 2017

This was referenced Nov 22, 2017

Spack needs integration tests #1507

Closed

User-Based Integration Tests #1951

Closed

alalazo mentioned this pull request Jan 17, 2018

Cleaned up JUnit report generation on install #6977

Merged

This was referenced Jan 30, 2018

Finish CDash report generation for spack install #7114

Closed

Spack release testing #7116

Closed

tgamblin modified the milestones: v0.12.0, v0.13.0 Feb 21, 2018

tgamblin closed this Feb 21, 2018

tgamblin deleted the wip branch January 12, 2019 09:26



		def prepare_build_report(self):
		report = self.create_template()


		dashboard: ["https://spack.io/cdash/submit.php?project=spack"]

		path: "~/home/username"


		$ spack install --log-format=junit <spec>

		Per default the log results will be placed into `var/<format>/test-<short_spec>.xml`.


		$ spack test-suite -c /location/of/yamlFile

		Currently output files are stored in you current working directory. A folder name spack-test-YYYY-MM-DD this may change in the future.

+              Junit
+              ^^^^^
+              With


		$ spack install --log-file report --log-format=cdash <spec>

		This will produce each one file for the configure, build, and test step.



		'''
		fucntion Name: removeTests

Conversation

kielfriedt commented Jan 24, 2017

Uh oh!

kielfriedt commented Jan 24, 2017

Uh oh!

tgamblin commented Jan 24, 2017

Uh oh!

tgamblin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kielfriedt commented Feb 16, 2017

Uh oh!

tgamblin commented Mar 20, 2017

Uh oh!

kielfriedt commented Mar 21, 2017

Uh oh!

tgamblin commented Mar 21, 2017

Uh oh!

tgamblin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment