Report jest-junit testsuite errors as failures#155
Merged
jozefizso merged 2 commits intodorny:mainfrom Mar 15, 2026
Merged
Conversation
Contributor
|
@johnbartholomew , I ran into a similar issue as yourself where there were features that I needed from this project and looked like it wasn't being updated any time soon, so I took the initiative to fork the project myself and build the enhancements into it with a colleague. https://github.com/phoenix-actions/test-reporting/ |
Author
|
Thanks for collecting these improvements together! |
|
Hi! Can we get this PR updated and merged please? |
jozefizso
requested review from
DominikPalo,
dharmendrasha,
j-catania and
jozefizso
Collaborator
|
@johnbartholomew, could you please resolve the conflicts ? |
Test errors are different (represented differently in the JUnit XML output from jest-junit) to test _failures_. Failures are tests which ran and failed, errors are for tests/test suites which did not even run because the test code itself didn't build or didn't execute correctly. jest-junit has an option to enable reporting of test suite errors, but test-reporter then interprets these as successful tests.
jozefizso
force-pushed
the
feat-report-test-suite-errors
branch
from
fe8fe90 to
5e23310
Compare
jozefizso
approved these changes
Mar 15, 2026
Collaborator
|
Hi @johnbartholomew , thanks a lot for this fix! |
mergify bot
added a commit
to ArcadeData/arcadedb
that referenced
this pull request
Mar 22, 2026
Bumps [dorny/test-reporter](https://github.com/dorny/test-reporter) from 2.6.0 to 3.0.0. Release notes *Sourced from [dorny/test-reporter's releases](https://github.com/dorny/test-reporter/releases).* > v3.0.0 > ------ > > **Note:** The v3 release requires NodeJS 24 runtime on GitHub Actions runners. > > What's Changed > -------------- > > * Upgrade action runtime to Node.js 24 by [`@dav-tb`](https://github.com/dav-tb) in [dorny/test-reporter#738](https://redirect.github.com/dorny/test-reporter/pull/738) > * Explicitly use lowest permissions required to run workflow by [`@jozefizso`](https://github.com/jozefizso) in [dorny/test-reporter#745](https://redirect.github.com/dorny/test-reporter/pull/745) > > ### Other Changes > > * Bump `@typescript-eslint/parser` from 8.57.0 to 8.57.1 by [`@dependabot`](https://github.com/dependabot)[bot] in [dorny/test-reporter#742](https://redirect.github.com/dorny/test-reporter/pull/742) > * Bump `@types/adm-zip` from 0.5.7 to 0.5.8 by [`@dependabot`](https://github.com/dependabot)[bot] in [dorny/test-reporter#743](https://redirect.github.com/dorny/test-reporter/pull/743) > * Bump flatted from 3.4.1 to 3.4.2 by [`@dependabot`](https://github.com/dependabot)[bot] in [dorny/test-reporter#744](https://redirect.github.com/dorny/test-reporter/pull/744) > > New Contributors > ---------------- > > * [`@dav-tb`](https://github.com/dav-tb) made their first contribution in [dorny/test-reporter#738](https://redirect.github.com/dorny/test-reporter/pull/738) > > **Full Changelog**: <dorny/test-reporter@v2.7.0...v3.0.0> > > v2.7.0 > ------ > > What's Changed > -------------- > > * Feature: Add `slug-prefix` output for link anchors [dorny/test-reporter#731](https://redirect.github.com/dorny/test-reporter/pull/731) > * Feature: Report `jest-junit` testsuite errors as failures [dorny/test-reporter#155](https://redirect.github.com/dorny/test-reporter/pull/155) > * Security: Update dependencies to fix reported security vulnerabilities > > ### Other Changes > > * Bump eslint from 9.39.3 to 9.39.4 by [`@dependabot`](https://github.com/dependabot)[bot] in [dorny/test-reporter#732](https://redirect.github.com/dorny/test-reporter/pull/732) > * Bump `@typescript-eslint/parser` from 8.56.1 to 8.57.0 by [`@dependabot`](https://github.com/dependabot)[bot] in [dorny/test-reporter#733](https://redirect.github.com/dorny/test-reporter/pull/733) > * Bump jest from 30.2.0 to 30.3.0 by [`@dependabot`](https://github.com/dependabot)[bot] in [dorny/test-reporter#734](https://redirect.github.com/dorny/test-reporter/pull/734) > * Bump undici from 6.23.0 to 6.24.1 by [`@dependabot`](https://github.com/dependabot)[bot] in [dorny/test-reporter#736](https://redirect.github.com/dorny/test-reporter/pull/736) > * Update `flatted` package to v3.4.1 to fix a vulnerability by [`@jozefizso`](https://github.com/jozefizso) in [dorny/test-reporter#739](https://redirect.github.com/dorny/test-reporter/pull/739) > > New Contributors > ---------------- > > * [`@dt-thomas-durand`](https://github.com/dt-thomas-durand) made their first contribution in [dorny/test-reporter#731](https://redirect.github.com/dorny/test-reporter/pull/731) > * [`@johnbartholomew`](https://github.com/johnbartholomew) made their first contribution in [dorny/test-reporter#155](https://redirect.github.com/dorny/test-reporter/pull/155) > > **Full Changelog**: <dorny/test-reporter@v2.6.0...v2.7.0> Changelog *Sourced from [dorny/test-reporter's changelog](https://github.com/dorny/test-reporter/blob/main/CHANGELOG.md).* > Changelog > ========= > > 3.0.0 > ----- > > * Feature: Use NodeJS 24 LTS as default runtime [dorny/test-reporter#738](https://redirect.github.com/dorny/test-reporter/pull/738) > > 2.7.0 > ----- > > * Feature: Add `slug-prefix` output for link anchors [dorny/test-reporter#731](https://redirect.github.com/dorny/test-reporter/pull/731) > * Feature: Report `jest-junit` testsuite errors as failures [dorny/test-reporter#155](https://redirect.github.com/dorny/test-reporter/pull/155) > * Security: Update dependencies to fix reported security vulnerabilities > > 2.6.0 > ----- > > * Fix: For `workflow_run` events, resolve the commit of the check run from related pull request head commits first (matching `workflow_run.head_branch`, then first PR), and fall back to `workflow_run.head_sha` for non-PR runs [dorny/test-reporter#673](https://redirect.github.com/dorny/test-reporter/pull/673) > * Change: The `test-reporter` action will listed all artifacts associated with the build run [dorny/test-reporter#693](https://redirect.github.com/dorny/test-reporter/pull/693) > * Maintenance: Upgrade to ESLint v9 [dorny/test-reporter#629](https://redirect.github.com/dorny/test-reporter/pull/629) > > 2.5.0 > ----- > > * Feature: Add Nette Tester support with `tester-junit` reporter [dorny/test-reporter#707](https://redirect.github.com/dorny/test-reporter/pull/707) > * Maintenance: Bump actions/upload-artifact from 5 to 6 [dorny/test-reporter#695](https://redirect.github.com/dorny/test-reporter/pull/695) > > 2.4.0 > ----- > > * Feature: Add PHPUnit support with JUnit XML dialect parser [dorny/test-reporter#422](https://redirect.github.com/dorny/test-reporter/pull/422) > * Feature: Add JUnit XML sample files and tests for validation [dorny/test-reporter#701](https://redirect.github.com/dorny/test-reporter/pull/701) > * Fix: Refactor deprecated `String.substr()` function to use `String.substring()` [dorny/test-reporter#704](https://redirect.github.com/dorny/test-reporter/pull/704) > > 2.3.0 > ----- > > * Feature: Add Python support with `python-xunit` reporter (pytest) [dorny/test-reporter#643](https://redirect.github.com/dorny/test-reporter/pull/643) > * Feature: Add pytest traceback parsing and `directory-mapping` option [dorny/test-reporter#238](https://redirect.github.com/dorny/test-reporter/pull/238) > * Performance: Update sax.js to fix large XML file parsing [dorny/test-reporter#681](https://redirect.github.com/dorny/test-reporter/pull/681) > * Documentation: Complete documentation for all supported reporters [dorny/test-reporter#691](https://redirect.github.com/dorny/test-reporter/pull/691) > * Security: Bump js-yaml and mocha in /reports/mocha (fixes prototype pollution) [dorny/test-reporter#682](https://redirect.github.com/dorny/test-reporter/pull/682) > > 2.2.0 > ----- > > * Feature: Add collapsed option to control report summary visibility [dorny/test-reporter#664](https://redirect.github.com/dorny/test-reporter/pull/664) > * Fix badge encoding for values including underscore and hyphens [dorny/test-reporter#672](https://redirect.github.com/dorny/test-reporter/pull/672) > * Fix missing `report-title` attribute in action definition [dorny/test-reporter#637](https://redirect.github.com/dorny/test-reporter/pull/637) > * Refactor variable names to fix shadowing issues [dorny/test-reporter#630](https://redirect.github.com/dorny/test-reporter/pull/630) > > 2.1.1 > ----- > > * Fix error when a TestMethod element does not have a className attribute in a trx file [dorny/test-reporter#623](https://redirect.github.com/dorny/test-reporter/pull/623) > * Add stack trace from trx to summary [dorny/test-reporter#615](https://redirect.github.com/dorny/test-reporter/pull/615) > * List only failed tests [dorny/test-reporter#606](https://redirect.github.com/dorny/test-reporter/pull/606) > * Add type definitions to `github-utils.ts` [dorny/test-reporter#604](https://redirect.github.com/dorny/test-reporter/pull/604) > * Avoid split on undefined [dorny/test-reporter#258](https://redirect.github.com/dorny/test-reporter/pull/258) > * Return links to summary report [dorny/test-reporter#588](https://redirect.github.com/dorny/test-reporter/pull/588) > * Add step summary short summary [dorny/test-reporter#589](https://redirect.github.com/dorny/test-reporter/pull/589) > * Fix for empty TRX TestDefinitions [dorny/test-reporter#582](https://redirect.github.com/dorny/test-reporter/pull/582) > * Increase step summary limit to 1MiB [dorny/test-reporter#581](https://redirect.github.com/dorny/test-reporter/pull/581) > * Fix input description for list options [dorny/test-reporter#572](https://redirect.github.com/dorny/test-reporter/pull/572) > > 2.1.0 > ----- ... (truncated) Commits * [`a43b3a5`](dorny/test-reporter@a43b3a5) test-reporter release v3.0.0 * [`79216dc`](dorny/test-reporter@79216dc) Merge pull request [#744](https://redirect.github.com/dorny/test-reporter/issues/744) from dorny/dependabot/npm\_and\_yarn/flatted-3.4.2 * [`62c3873`](dorny/test-reporter@62c3873) Merge pull request [#745](https://redirect.github.com/dorny/test-reporter/issues/745) from dorny/feature/workflow\_permissions * [`bd74ca6`](dorny/test-reporter@bd74ca6) Explicitly use lowest permissions required to run workflow * [`0c2b2aa`](dorny/test-reporter@0c2b2aa) Merge pull request [#738](https://redirect.github.com/dorny/test-reporter/issues/738) from dav-tb/feature/node24-runtime * [`f7eee30`](dorny/test-reporter@f7eee30) Use the `@types/node` definitions for NodeJS 24 runtime * [`ad0a96f`](dorny/test-reporter@ad0a96f) Upgrade action runtime to Node.js 24 * [`543415c`](dorny/test-reporter@543415c) Merge pull request [#741](https://redirect.github.com/dorny/test-reporter/issues/741) from dorny/release/v2.7.0 * [`f1284d0`](dorny/test-reporter@f1284d0) Bump flatted from 3.4.1 to 3.4.2 * [`3c1b53f`](dorny/test-reporter@3c1b53f) Merge pull request [#743](https://redirect.github.com/dorny/test-reporter/issues/743) from dorny/dependabot/npm\_and\_yarn/types/adm-zip-0.5.8 * Additional commits viewable in [compare view](dorny/test-reporter@3d76b34...a43b3a5) [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test errors are different (represented differently in the JUnit XML output from jest-junit) to test failures. Failures are tests which ran and failed, errors are for tests/test suites which did not even run because the test code itself didn't build or didn't execute correctly.
jest-junit has an option to enable reporting of test suite errors (
JEST_JUNIT_REPORT_TEST_SUITE_ERRORS/reportTestSuiteErrors). This PR modifies the test-reporter to treat test suite errors as failures, instead of treating them as successful.