Cleaned up JUnit report generation on install#6977
Conversation
alalazo
force-pushed
the
refactoring/cleanup_junit
branch
3 times, most recently
from
a1c790a to
3c030d6
Compare
|
Tried to import today the |
lib/spack/spack/report.py
Outdated
|
|
||
| def fetch_text(path): | ||
| if not os.path.exists(path): | ||
| return 'No such file or directory: {0}'.format(path) |
There was a problem hiding this comment.
Looks like the No such file or directory would end up in the report in place of the actual test's stdout when the latter is missing. When this happens, is there a way to tell the difference between this error message (a kind of internal error) and something emitted by the test itself?
There was a problem hiding this comment.
Looks like the No such file or directory would end up in the report in place of the actual test's stdout when the latter is missing.
You're right on that. I am a bit undecided what to do, in the sense that I am trying to distinguish errors from failures based on the exception type. As we have the real stdout and stderr from the build process merged and dumped into build_log_path I was thinking to:
- map
build_log_pathto<stdout/> - map the exception stackstrace to
<stderr/>- still to be implemented - map a fixed message (like I do right now) or maybe
str(exc_val)to themessageproperty.
The end result looks like this:
on the Jenkins JUnit plugin. Suggestions?
There was a problem hiding this comment.
The report looks quite clear, imo as long as you have the stack trace any other data becomes (nice to have tho) decor.
alalazo
force-pushed
the
refactoring/cleanup_junit
branch
from
17306dc to
e06e4cf
Compare
The generation of a JUnit report was previously part of the install command. This commit factors the logic into its own module, and uses a template for the generation of the report. It also improves report generation, that now can deal with multiple specs installed at once. Finally, extending the list of supported formats is much easier than before, as it entails just writing a new template.
The generation of a JUnit report has been polished, so that the stacktrace is correctly displayed with Jenkins JUnit plugin. Standard error is still not used. Added unit tests to cover for installation failures and installation errors.
alalazo
force-pushed
the
refactoring/cleanup_junit
branch
from
e06e4cf to
93dee1a
Compare
tgamblin
left a comment
tgamblin
left a comment
There was a problem hiding this comment.
I have minor comments on this, but in general this is great. I think it can be merged as-is.
The main question I've got is how we should factor this with respect to CDash support. CDash is more extensive; there are phases (update, configure, build, test, deploy) and it supports zooming in on specific errors in a log (as parsed by CTest or external/ctest_log_parser). I don't think JUnit will ever support all that; it's mostly for unit tests.
We're working with folks at Kitware so maybe they can base CDash support on this.
Another higher-level question, maybe for a new PR:
-
From the CDash perspective (maybe not junit) it would be nice to show that certain tests failed to concretize. The JUnit support currently assumes that things concretized successfully.
-
I think that eventually,
do_installneeds to be refactored to clean this up properly. Monkey-patching package is kind of nasty, and I thinkPackageshould really be providing hooks that this uses. Also, when we do parallel builds, it's going to require a refactor of how dependency installs are done, too.
| start_time = time.time() | ||
| value = None | ||
| try: | ||
|
|
There was a problem hiding this comment.
Ok, got it, you don't like blank lines after function definition if there's no docstring 😄
| ) | ||
|
|
||
| def __exit__(self, exc_type, exc_val, exc_tb): | ||
|
|
👍 to all this. In fact, I have a refactoring in #6833 that extracts dependency installs. If you think it's a good start, we may continue working there. |
* Cleaned up JUnit report generation on install The generation of a JUnit report was previously part of the install command. This commit factors the logic into its own module, and uses a template for the generation of the report. It also improves report generation, that now can deal with multiple specs installed at once. Finally, extending the list of supported formats is much easier than before, as it entails just writing a new template. * Polished report generation + added tests for failures and errors The generation of a JUnit report has been polished, so that the stacktrace is correctly displayed with Jenkins JUnit plugin. Standard error is still not used. Added unit tests to cover for installation failures and installation errors.
3 participants
The generation of a JUnit report was previously part of the install command. This commit factors the logic into its own module, and uses a template for the generation of the report.
It also improves report generation, that now can deal with multiple specs installed at once. Finally, extending the list of supported formats is much easier than before, as it entails just writing a new template.
refers #2917
Example
Running this command:
spack install --log-format=junit --log-file=hdf5.xml szip hdf5+szip~mpi tarproduces the following file: