feat(libtest): Add JUnit formatter #84568

andoriyu · 2021-04-25T22:57:56Z

tracking issue: #85563

Add an alternative formatter to libtest. Formatter produces valid xml that later can be interpreted as JUnit report.

Caveats:

timestamp is required by schema, but every viewer/parser ignores it. Attribute is not set to avoid depending on chrono;
Running all "suits" (unit tests, doc-tests and integration tests) will produce a mess;
I couldn't find a way to get integration test binary name, so it's just goes by "integration";

Sample output for unit tests (pretty printed by 3rd party tool):

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="test" package="test" id="0" errors="0" failures="0" tests="13" skipped="1">
    <testcase classname="results::tests" name="test_completed_bad" time="0"/>
    <testcase classname="results::tests" name="suite_started" time="0"/>
    <testcase classname="results::tests" name="suite_ended_ok" time="0"/>
    <testcase classname="results::tests" name="suite_ended_bad" time="0"/>
    <testcase classname="junit::tests" name="test_failed_output" time="0"/>
    <testcase classname="junit::tests" name="test_simple_output" time="0"/>
    <testcase classname="junit::tests" name="test_multiple_outputs" time="0"/>
    <testcase classname="results::tests" name="test_completed_ok" time="0"/>
    <testcase classname="results::tests" name="test_stared" time="0"/>
    <testcase classname="junit::tests" name="test_generate_xml_no_error_single_testsuite" time="0"/>
    <testcase classname="results::tests" name="test_simple_output" time="0"/>
    <testcase classname="test" name="should_panic" time="0"/>
    <system-out/>
    <system-err/>
  </testsuite>
</testsuites>

Sample output for integration tests (pretty printed by 3rd party tool):

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="test" package="test" id="0" errors="0" failures="0" tests="1" skipped="0">
    <testcase classname="integration" name="test_add" time="0"/>
    <system-out/>
    <system-err/>
  </testsuite>
</testsuites>

Sample output for Doc-tests (pretty printed by 3rd party tool):

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="test" package="test" id="0" errors="0" failures="0" tests="1" skipped="0">
    <testcase classname="src/lib.rs" name="(line 2)" time="0"/>
    <system-out/>
    <system-err/>
  </testsuite>
</testsuites>

rust-highfive · 2021-04-25T22:57:58Z

r? @yaahc

(rust-highfive has picked a reviewer for you, use r? to override)

library/test/src/formatters/junit.rs

yaahc · 2021-04-29T18:09:16Z

Hey, just wanted to drop in to let you know that I've done an initial review and this looks good so far. Based on the comments it looks like you have some additional changes still planned so I'll hold off on approving until you've made the changes you mentioned.

Mark-Simulacrum · 2021-05-01T00:53:12Z

Hm, I have some mild reservations about supporting junit (and other more specialized frameworks). We already have a json backend, and it seems likely that could be mapped to junit with an external crate if desired. That does hurt usability somewhat, but I'm not sure it makes sense to support all possible test output formats in-tree either.

Can you perhaps share what is expected to consume the junit output? Or what the motivation is?

andoriyu · 2021-05-01T01:21:10Z

Hm, I have some mild reservations about supporting junit (and other more specialized frameworks). We already have a json backend, and it seems likely that could be mapped to junit with an external crate if desired. That does hurt usability somewhat, but I'm not sure it makes sense to support all possible test output formats in-tree either.

Can you perhaps share what is expected to consume the junit output? Or what the motivation is?

Well, JUnit is universally supported format: a lot of test harnesses/runners support JUnit as an output format, many Continuous Integration platforms support consuming it. I think CI would be a common use case to use JUnit as output. There is no point on supporting all other formats, like you said, because JUnit format is the least common denominator across all of them.

As for JSON backend goes, well, it's not stabilized and not documented. You can map it to JUnit, like you said, I did that in cargo-suity. In my opinion, this is a task for test harness, rather than 3rd party library.

Personally, I would prefer if I could keep console output, and write JUnit report into a file in parallel.

tschuett · 2021-05-01T04:16:21Z

Hm, I have some mild reservations about supporting junit (and other more specialized frameworks). We already have a json backend, and it seems likely that could be mapped to junit with an external crate if desired. That does hurt usability somewhat, but I'm not sure it makes sense to support all possible test output formats in-tree either.

Can you perhaps share what is expected to consume the junit output? Or what the motivation is?

As @andoriyu said most CI systems can consume JUnit reports. Of course you can transform json output into anything, but it seems to indirect to me.

jyn514 · 2021-05-14T00:11:59Z

I am not an appropriate reviewer for this PR, please don't request reviews from me.

andoriyu · 2021-05-14T00:24:55Z

I am not an appropriate reviewer for this PR, please don't request reviews from me.

Oh, you just left some comment in the past. My bad.

library/test/src/formatters/junit.rs

yaahc · 2021-05-21T01:28:18Z

Hm, I have some mild reservations about supporting junit (and other more specialized frameworks). We already have a json backend, and it seems likely that could be mapped to junit with an external crate if desired. That does hurt usability somewhat, but I'm not sure it makes sense to support all possible test output formats in-tree either.

Can you perhaps share what is expected to consume the junit output? Or what the motivation is?

@Mark-Simulacrum do you still feel reservations given @andoriyu's response?

Mark-Simulacrum · 2021-05-21T02:24:33Z

Mostly resolved, yeah. I think we should get a tracking issue filed, particularly given the presence of several outstanding defects (as noted in the PR description). It's not clear to me what the path to stabilization for these custom test outputs would be, but it needn't block unstable addition.

tschuett · 2021-05-21T06:16:15Z

Mostly resolved, yeah. I think we should get a tracking issue filed, particularly given the presence of several outstanding defects (as noted in the PR description). It's not clear to me what the path to stabilization for these custom test outputs would be, but it needn't block unstable addition.

I don't believe that junit is custom at all. Allmost all CIs have support for junit, e.g.:
https://plugins.jenkins.io/junit/:

yaahc · 2021-05-21T18:41:12Z

Mostly resolved, yeah. I think we should get a tracking issue filed, particularly given the presence of several outstanding defects (as noted in the PR description). It's not clear to me what the path to stabilization for these custom test outputs would be, but it needn't block unstable addition.

Sounds good. @andoriyu, can you go ahead and create a tracking issue? Once that's setup I think this PR is good to go.

andoriyu · 2021-05-21T19:02:42Z

Mostly resolved, yeah. I think we should get a tracking issue filed, particularly given the presence of several outstanding defects (as noted in the PR description). It's not clear to me what the path to stabilization for these custom test outputs would be, but it needn't block unstable addition.

Sounds good. @andoriyu, can you go ahead and create a tracking issue? Once that's setup I think this PR is good to go.

@yaahc Done - #85563. I've never filled tracking issues before, if I made any mistakes please let me know.

yaahc · 2021-05-25T00:04:04Z

Looks perfect @andoriyu. I went ahead and edited your original post in this PR to add a link to the tracking issue. Thanks again for all the work!

@bors r+

bors · 2021-05-25T00:04:05Z

📌 Commit 9f83e22 has been approved by yaahc

bors · 2021-05-25T16:57:41Z

⌛ Testing commit 9f83e22 with merge 90f23ae3440a957bc9de51e4e2e2a99330f875ac...

rust-log-analyzer · 2021-05-25T19:14:16Z

The job dist-aarch64-apple failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

      Memory: 14 GB
      Boot ROM Version: VMW71.00V.13989454.B64.1906190538
      Apple ROM Info: [MS_VM_CERT/SHA1/27d66596a61c48dd3dc7216fd715126e33f59ae7]Welcome to the Virtual Machine
      SMC Version (system): 2.8f0
      Serial Number (system): VMLFRk6HhKKs

hw.ncpu: 3
hw.byteorder: 1234
hw.memsize: 15032385536

bors · 2021-05-25T19:15:25Z

💔 Test failed - checks-actions

andoriyu · 2021-05-26T19:03:37Z

@yaahc looks like crates.io was down briefly?

yaahc · 2021-05-27T19:35:50Z

@yaahc looks like crates.io was down briefly?

oo, sorry about that

@bors retry

jonasbb · 2021-05-27T20:01:51Z

library/test/src/formatters/junit.rs

+                         name=\"{}\" time=\"{}\">",
+                        class_name,
+                        test_name,
+                        duration.as_secs()


It would be nice if the output supports subsecond precision for the duration. This is what cargo2junit does. That way you don't loose the runtime information for fast tests.

bors · 2021-05-27T21:15:00Z

⌛ Testing commit 9f83e22 with merge 1c6868a...

bors · 2021-05-27T23:55:46Z

☀️ Test successful - checks-actions
Approved by: yaahc
Pushing 1c6868a to master...

klensy · 2021-05-28T13:03:57Z

Why this so green and somehow part of benchmarks don't run: https://perf.rust-lang.org/compare.html?start=e51830b90afd339332892a8f20db1957d43bf086&end=1c6868aa21981b37cbd3fc95828ee3b0ac22d494&stat=instructions%3Au

edit: source of this #85757

feat(libtest): Add JUnit formatter

38485a9

rust-highfive assigned yaahc Apr 25, 2021

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 25, 2021

jyn514 reviewed Apr 26, 2021

View reviewed changes

library/test/src/formatters/junit.rs Outdated Show resolved Hide resolved

jyn514 added A-libtest Area: `#[test]` / the `test` library T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. labels Apr 26, 2021

tschuett reviewed Apr 26, 2021

View reviewed changes

library/test/src/formatters/junit.rs Outdated Show resolved Hide resolved

Update junit.rs

4b4d06a

yaahc added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 29, 2021

Better output for junit formatter

9f83e22

andoriyu requested a review from jyn514 May 13, 2021 23:26

yaahc reviewed May 14, 2021

View reviewed changes

library/test/src/formatters/junit.rs Show resolved Hide resolved

andoriyu mentioned this pull request May 21, 2021

Tracking Issue for libtest's JUnit reporter #85563

Open

3 tasks

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 25, 2021

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels May 25, 2021

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 27, 2021

jonasbb reviewed May 27, 2021

View reviewed changes

bors added the merged-by-bors This PR was explicitly merged by bors. label May 27, 2021

bors merged commit 1c6868a into rust-lang:master May 27, 2021

rustbot added this to the 1.54.0 milestone May 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(libtest): Add JUnit formatter #84568

feat(libtest): Add JUnit formatter #84568

andoriyu commented Apr 25, 2021 •

edited by yaahc

Loading

rust-highfive commented Apr 25, 2021

yaahc commented Apr 29, 2021

Mark-Simulacrum commented May 1, 2021

andoriyu commented May 1, 2021

tschuett commented May 1, 2021 •

edited

Loading

jyn514 commented May 14, 2021

andoriyu commented May 14, 2021

yaahc commented May 21, 2021

Mark-Simulacrum commented May 21, 2021 •

edited

Loading

tschuett commented May 21, 2021

yaahc commented May 21, 2021

andoriyu commented May 21, 2021

yaahc commented May 25, 2021

bors commented May 25, 2021

bors commented May 25, 2021

rust-log-analyzer commented May 25, 2021

bors commented May 25, 2021

andoriyu commented May 26, 2021

yaahc commented May 27, 2021

jonasbb May 27, 2021

bors commented May 27, 2021

bors commented May 27, 2021

klensy commented May 28, 2021 •

edited

Loading

feat(libtest): Add JUnit formatter #84568

feat(libtest): Add JUnit formatter #84568

Conversation

andoriyu commented Apr 25, 2021 • edited by yaahc Loading

rust-highfive commented Apr 25, 2021

yaahc commented Apr 29, 2021

Mark-Simulacrum commented May 1, 2021

andoriyu commented May 1, 2021

tschuett commented May 1, 2021 • edited Loading

jyn514 commented May 14, 2021

andoriyu commented May 14, 2021

yaahc commented May 21, 2021

Mark-Simulacrum commented May 21, 2021 • edited Loading

tschuett commented May 21, 2021

yaahc commented May 21, 2021

andoriyu commented May 21, 2021

yaahc commented May 25, 2021

bors commented May 25, 2021

bors commented May 25, 2021

rust-log-analyzer commented May 25, 2021

bors commented May 25, 2021

andoriyu commented May 26, 2021

yaahc commented May 27, 2021

jonasbb May 27, 2021

Choose a reason for hiding this comment

bors commented May 27, 2021

bors commented May 27, 2021

klensy commented May 28, 2021 • edited Loading

andoriyu commented Apr 25, 2021 •

edited by yaahc

Loading

tschuett commented May 1, 2021 •

edited

Loading

Mark-Simulacrum commented May 21, 2021 •

edited

Loading

klensy commented May 28, 2021 •

edited

Loading