Testing: Optionally run tests on externally installed packages by tldahlgren · Pull Request #28701 · spack/spack

tldahlgren · 2022-02-01T02:11:16Z

Since Spack does not install external packages, this PR skips them by default when running stand-alone tests. The assumption is that such packages have likely undergone an acceptance test process. However, the tests can be run against installed externals using spack test run --externals.

$ spack test run -h
usage: spack test run [-h] [--alias ALIAS] [--fail-fast] [--fail-first]
                      [--externals] [--keep-stage]
                      [--log-format {None,junit,cdash}] [--log-file LOG_FILE]
                      [--help-cdash] [--clean | --dirty]
                      ...

Run tests for the specified installed packages.

    If no specs are listed, run tests for all packages in the current
    environment or all installed packages if there is no active environment.
    

positional arguments:
  installed_specs       one or more installed package specs

optional arguments:
  --alias ALIAS         Provide an alias for this test-suite for subsequent access.
  --clean               unset harmful variables in the build environment (default)
  --dirty               preserve user environment in spack's build environment (danger!)
  --externals      Test packages that are externally installed.
  --fail-fast           Stop tests for each package after the first failure.
  --fail-first          Stop after the first failed package.
  --help-cdash          Show usage instructions for CDash reporting
  --keep-stage          Keep testing directory for debugging
  --log-file LOG_FILE   filename for the log file. if not passed a default will be used
  --log-format {None,junit,cdash}
                        format to be used for log files
  -h, --help            show this help message and exit

Results from running tests against externally installed packages without the option are, for example:

$ spack test run openmpi xz
==> Spack test h53e5oth7pwfqtm4d6z6nt3doc32iy34

$ spack test results
==> Results for test suite 'h53e5oth7pwfqtm4d6z6nt3doc32iy34':
==>   openmpi-4.1.2-37vuqvc SKIPPED
==>   xz-5.2.2-iwgisde SKIPPED
==>   xz-5.2.2-i5m6iu6 SKIPPED

$ spack test results -l
==> Results for test suite 'h53e5oth7pwfqtm4d6z6nt3doc32iy34':
==>   openmpi-4.1.2-37vuqvc SKIPPED
Testing package openmpi-4.1.2-37vuqvc
Skipped external package

==>   xz-5.2.2-iwgisde SKIPPED
Testing package xz-5.2.2-iwgisde
Skipped external package

==>   xz-5.2.2-i5m6iu6 SKIPPED
Testing package xz-5.2.2-i5m6iu6
Skipped external package

The junit output from spack test run --log-format junit openmpi xz is:

<?xml version="1.0" encoding="UTF-8"?>
<!--
    This file has been modeled after the basic
    specifications at this url:

    http://help.catchsoftware.com/display/ET/JUnit+Format
-->
<testsuites>
    <testsuite name="openmpi_37vuqvc"
               errors="0"
               tests="1"
               failures="0"
               time="0.002861499786376953"
               timestamp="Tue, 01 Feb 2022 18:20:42" >
        <properties>
            <property name="architecture" value="linux-rhel7-skylake_avx512" />
            <property name="compiler" value="[email protected]" />
        </properties>
        <testcase classname="openmpi"
                  name="37vuqvcq7pp2yeattawjomxfrw6a5mo3"
                  time="0.002861499786376953">
            <skipped />
            <system-out>
Skipped external package
            </system-out>
        </testcase>
    </testsuite>
    <testsuite name="xz_iwgisde"
               errors="0"
               tests="1"
               failures="0"
               time="0.002824068069458008"
               timestamp="Tue, 01 Feb 2022 18:20:42" >
        <properties>
            <property name="architecture" value="linux-rhel7-skylake_avx512" />
            <property name="compiler" value="[email protected]" />
        </properties>
        <testcase classname="xz"
                  name="iwgisdepiy7nislfoooycg4arjqjlc5s"
                  time="0.002824068069458008">
            <skipped />
            <system-out>
Skipped external package
            </system-out>
        </testcase>
    </testsuite>
    <testsuite name="xz_i5m6iu6"
               errors="0"
               tests="1"
               failures="0"
               time="0.002808094024658203"
               timestamp="Tue, 01 Feb 2022 18:20:42" >
        <properties>
            <property name="architecture" value="linux-rhel7-cascadelake" />
            <property name="compiler" value="[email protected]" />
        </properties>
        <testcase classname="xz"
                  name="i5m6iu6vcvp5mbmyh5ah6vfdbzvfjzgo"
                  time="0.002808094024658203">
            <skipped />
            <system-out>
Skipped external package
            </system-out>
        </testcase>
    </testsuite>

Results when using --externals:

$ spack test run --externals openmpi xz
==> Spack test h53e5oth7pwfqtm4d6z6nt3doc32iy34
==> Testing package openmpi-4.1.2-37vuqvc
==> Error: TestFailure: 9 tests failed.
<..snip..>

See test log for details:
  /g/g21/dahlgren/.spack/test/h53e5oth7pwfqtm4d6z6nt3doc32iy34/openmpi-4.1.2-37vuqvc-test-out.txt

==> Testing package xz-5.2.2-iwgisde
==> Testing package xz-5.2.2-i5m6iu6
==> Error: 1 test(s) in the suite failed.

$ spack test results
==> Results for test suite 'h53e5oth7pwfqtm4d6z6nt3doc32iy34':
==>   openmpi-4.1.2-37vuqvc FAILED
==>   xz-5.2.2-iwgisde NO-TESTS
==>   xz-5.2.2-i5m6iu6 NO-TESTS

$ spack test results -l
==> Results for test suite 'h53e5oth7pwfqtm4d6z6nt3doc32iy34':
==>   openmpi-4.1.2-37vuqvc FAILED
==> Testing package openmpi-4.1.2-37vuqvc
<..snip..>

==>   xz-5.2.2-iwgisde NO-TESTS
==> Testing package xz-5.2.2-iwgisde
No tests to run

==>   xz-5.2.2-i5m6iu6 NO-TESTS
==> Testing package xz-5.2.2-i5m6iu6
No tests to run

tldahlgren · 2022-02-01T02:57:38Z

~~WARNING: This approach will NOT work for CDash data collection and reporting of skipped tests.~~

The test results are now captured and reported when using the junit reporter.

adamjstewart · 2022-02-01T19:45:40Z

Probably won't have time to properly review and test this, but I trust that it resolves the original issue in one way or another.

tldahlgren · 2022-02-01T19:51:02Z

Probably won't have time to properly review and test this, but I trust that it resolves the original issue in one way or another.

@adamjstewart Are you fine with simply skipping tests if the package is external?

My rationale was that if Spack doesn't install the package, why should it be testing it since, in theory, the sysadmins who installed it likely had an acceptance testing process they had to undergo. BUT does that make sense to you?

Alternatively, we could skip tests that cache files as you'd suggested. My concern with that approach is the behavior would appear inconsistent and we'd have to explain the special case(s).

adamjstewart · 2022-02-01T20:10:21Z

I'm personally fine with either solution.

tldahlgren · 2022-02-03T17:48:22Z

I'm personally fine with either solution.

Since Todd mentioned (in Slack) that having the ability to test externals can be useful, I went ahead and added an option to spack test run so we don't completely lose the capability.

tldahlgren · 2022-02-03T20:16:55Z

Oops. Accidentally pushed change for a different PR. Looks like the reset didn't help.

alalazo

Tried to test-drive the PR and it LGTM. I left a minor comment and a question, mainly for my understanding of this part of the code.

lib/spack/spack/cmd/test.py

lib/spack/spack/package.py

alalazo

Works fine for me on a test drive:

This support requires adding the '--tests' option to 'spack ci rebuild'. Packages whose stand-alone tests are broken (in the CI environment) can be configured in gitlab-ci to be skipped by adding them to broken-tests-packages. Highlights include: - Restructured 'spack ci' help to provide better subcommand summaries; - Ensured only one InstallError (i.e., installer's) rather than allowing build_environment to have its own; and - Refactored CI and CDash reporting to keep CDash-related properties and behavior in a separate class. This allows stand-alone tests from `spack ci` to run when the `--tests` option is used. With `--tests`, stand-alone tests are run **after** a **successful** (re)build of the package. Test results are collected and report(able) using CDash. This PR adds the following features: - Adds `-t` and `--tests` to `spack ci rebuild` to run stand-alone tests; - Adds `--fail-fast` to stop stand-alone tests after the first failure; - Ensures a *single* `InstallError` across packages (i.e., removes second class from build environment); - Captures skipping tests for externals and uninstalled packages (for CDash reporting); - Copies test logs and outputs to the CI artifacts directory to facilitate debugging; - Parses stand-alone test results to report outputs from each `run_test` as separate test parts (CDash reporting); - Logs a test completion message to allow capture of timing of the last `run_test` part; - Adds the runner description to the CDash site to better distinguish entries in CDash tables; - Adds `gitlab-ci` `broken-tests-packages` to CI configuration to skip stand-alone testing for packages with known issues; - Changes `spack ci --help` so description of each subcommand is a single line; - Changes `spack ci <subcommand> --help` to provide the full description of each command (versus no description); and - Ensures `junit` test log file ends in an `.xml` extension (versus default where it does not). Tasks: - [x] Include the equivalent of the architecture information, or at least the host target, in the CDash output - [x] Upload stand-alone test results files as `test` artifacts - [x] Confirm tests are run in GitLab - [x] Ensure CDash results are uploaded as artifacts - [x] Resolve issues with CDash build-and test results appearing on same row of the table - [x] Add unit tests as needed - [x] Investigate why some (dependency) packages don't have test results (e.g., related from other pipelines) - [x] Ensure proper parsing and reporting of skipped tests (as `not run`) .. post- #28701 merge - [x] Restore the proper CDash URLand or mirror ONCE out-of-band testing completed

This support requires adding the '--tests' option to 'spack ci rebuild'. Packages whose stand-alone tests are broken (in the CI environment) can be configured in gitlab-ci to be skipped by adding them to broken-tests-packages. Highlights include: - Restructured 'spack ci' help to provide better subcommand summaries; - Ensured only one InstallError (i.e., installer's) rather than allowing build_environment to have its own; and - Refactored CI and CDash reporting to keep CDash-related properties and behavior in a separate class. This allows stand-alone tests from `spack ci` to run when the `--tests` option is used. With `--tests`, stand-alone tests are run **after** a **successful** (re)build of the package. Test results are collected and report(able) using CDash. This PR adds the following features: - Adds `-t` and `--tests` to `spack ci rebuild` to run stand-alone tests; - Adds `--fail-fast` to stop stand-alone tests after the first failure; - Ensures a *single* `InstallError` across packages (i.e., removes second class from build environment); - Captures skipping tests for externals and uninstalled packages (for CDash reporting); - Copies test logs and outputs to the CI artifacts directory to facilitate debugging; - Parses stand-alone test results to report outputs from each `run_test` as separate test parts (CDash reporting); - Logs a test completion message to allow capture of timing of the last `run_test` part; - Adds the runner description to the CDash site to better distinguish entries in CDash tables; - Adds `gitlab-ci` `broken-tests-packages` to CI configuration to skip stand-alone testing for packages with known issues; - Changes `spack ci --help` so description of each subcommand is a single line; - Changes `spack ci <subcommand> --help` to provide the full description of each command (versus no description); and - Ensures `junit` test log file ends in an `.xml` extension (versus default where it does not). Tasks: - [x] Include the equivalent of the architecture information, or at least the host target, in the CDash output - [x] Upload stand-alone test results files as `test` artifacts - [x] Confirm tests are run in GitLab - [x] Ensure CDash results are uploaded as artifacts - [x] Resolve issues with CDash build-and test results appearing on same row of the table - [x] Add unit tests as needed - [x] Investigate why some (dependency) packages don't have test results (e.g., related from other pipelines) - [x] Ensure proper parsing and reporting of skipped tests (as `not run`) .. post- spack#28701 merge - [x] Restore the proper CDash URLand or mirror ONCE out-of-band testing completed

tldahlgren mentioned this pull request Feb 1, 2022

Testing: Summarize test results and add verbose output #28700

Merged

tldahlgren changed the title ~~tests/skip external packages~~ Testing: Skip externally installed packages Feb 1, 2022

tldahlgren requested a review from adamjstewart February 1, 2022 02:18

tldahlgren added external-packages stand-alone-tests Stand-alone (or smoke) tests for installed packages labels Feb 1, 2022

tldahlgren removed the request for review from adamjstewart February 1, 2022 02:54

tldahlgren force-pushed the tests/skip-external-packages branch from b877f93 to d48f77a Compare February 1, 2022 04:50

tldahlgren requested a review from adamjstewart February 1, 2022 04:56

tests/skip external packages

4e10a67

tldahlgren force-pushed the tests/skip-external-packages branch from d48f77a to 4e10a67 Compare February 1, 2022 18:18

Add a unit test

722a4a4

tldahlgren requested review from RikkiButler20, alalazo, becker33 and tgamblin February 2, 2022 01:25

spackbot-app bot added commands tests General test capability(ies) labels Feb 3, 2022

tldahlgren force-pushed the tests/skip-external-packages branch from 3349c1e to 3e57654 Compare February 3, 2022 01:52

tldahlgren changed the title ~~Testing: Skip externally installed packages~~ Testing: Optionally skip externally installed packages Feb 3, 2022

Add --test-externals option to 'spack test run'

a8901e3

tldahlgren force-pushed the tests/skip-external-packages branch from 3e57654 to a8901e3 Compare February 3, 2022 02:02

tldahlgren force-pushed the tests/skip-external-packages branch from 3b3b20e to a8901e3 Compare February 3, 2022 18:15

tldahlgren closed this Feb 3, 2022

tldahlgren reopened this Feb 3, 2022

alalazo reviewed Feb 4, 2022

View reviewed changes

lib/spack/spack/cmd/test.py Outdated Show resolved Hide resolved

lib/spack/spack/package.py Outdated Show resolved Hide resolved

tldahlgren changed the title ~~Testing: Optionally skip externally installed packages~~ Testing: Optionally run externally installed packages Feb 11, 2022

tldahlgren changed the title ~~Testing: Optionally run externally installed packages~~ Testing: Optionally run tests on externally installed packages Feb 11, 2022

Replace --test-externals with --externals

975fafd

alalazo approved these changes Feb 17, 2022

View reviewed changes

alalazo merged commit fefe65a into spack:develop Feb 17, 2022

tldahlgren mentioned this pull request Feb 26, 2022

spack ci: add support for running stand-alone tests #27877

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Testing: Optionally run tests on externally installed packages#28701

Testing: Optionally run tests on externally installed packages#28701
alalazo merged 4 commits intospack:developfrom
tldahlgren:tests/skip-external-packages

tldahlgren commented Feb 1, 2022 •

edited

Loading

Uh oh!

tldahlgren commented Feb 1, 2022 •

edited

Loading

Uh oh!

adamjstewart commented Feb 1, 2022

Uh oh!

tldahlgren commented Feb 1, 2022 •

edited

Loading

Uh oh!

adamjstewart commented Feb 1, 2022

Uh oh!

tldahlgren commented Feb 3, 2022

Uh oh!

tldahlgren commented Feb 3, 2022

Uh oh!

alalazo left a comment

Uh oh!

Uh oh!

Uh oh!

alalazo left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

tldahlgren commented Feb 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tldahlgren commented Feb 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamjstewart commented Feb 1, 2022

Uh oh!

tldahlgren commented Feb 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamjstewart commented Feb 1, 2022

Uh oh!

tldahlgren commented Feb 3, 2022

Uh oh!

tldahlgren commented Feb 3, 2022

Uh oh!

alalazo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alalazo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tldahlgren commented Feb 1, 2022 •

edited

Loading

tldahlgren commented Feb 1, 2022 •

edited

Loading

tldahlgren commented Feb 1, 2022 •

edited

Loading