Currently, we run a bunch of tests on the moby/moby codebase for each PR and against the master branch of the repo after a PR is merged, but the results of the tests are in approximately 400KB of plain-text format. To find out which test failed, a user would have to view the console logs of the jenkins job and do a browser text search for FAIL:.

Would be better to have the test results also reported in a structured format for consumption by report generators to make pretty graphs and tables and such. Having the test result in a junit xml format would allow jenkins UI to consume and generate reports with the junit plugin.

The type of information we could easily find with a junit xml format output:

• name of all the test case failures (assuming the jenkins job does not stop running tests upon first failure)
• time it took to run each test case
• summary of how many tests were run, how many passed, how many failed, and how many were skipped

Would need to generate junit xml format for each of the 4 test frameworks in moby/moby:

1. unit tests - uses golang testing and can output junit xml by using gotestsum to execute tests
2. integration tests - uses gotest.tools and can output junit xml by using gotestsum to execute tests
3. integration-cli tests - uses go-check and can output junit xml by using go2xunit
4. docker-py tests - uses pytest and can output junit xml by setting --junitxml flag

Progress:

• unit tests: WIP added entry for running unit tests with junit report #39637 added entry for running unit tests with junit report #39638
  • unit test reports for non-amd64 architectures Jenkinsfile: collect junit.xml for all architectures #39719
  • unit test reports for windows WIP: Windows unit tests - move to junit #39971
• integration Jenkinsfile: create junit.xml for integration tests #39724
• integration (windows) Windows: Build and use gotestsum for running all tests #39998
• integration-cli hack: have integration-cli use gotestsum codepath #39911
• integration-cli (windows) (no PR yet)
• docker-py docker-py: fix linting, generate junit.xml, save bundles, and various improvements #39716