Skip to content

fix: Run all sub-project tests in test_pack_doc.yml workflow#1382

Merged
bdferris-v2 merged 10 commits intomasterfrom
issue/1381/all_tests
Apr 11, 2023
Merged

fix: Run all sub-project tests in test_pack_doc.yml workflow#1382
bdferris-v2 merged 10 commits intomasterfrom
issue/1381/all_tests

Conversation

@bdferris-v2
Copy link
Copy Markdown
Collaborator

@bdferris-v2 bdferris-v2 commented Apr 7, 2023 •
edited
Loading

As noted in #1381, the test_pack_doc.yml is not currently running all of the project's unit-tests, which is problematic if we want to be sure new PRs aren't introducing test breakages.

To address the issue, this PR does the following:

  • Upgrade to Gradle 7.4 (currently at 7.2), which introduces a new Test Report Aggregation plugin. This plugin offers better support for test aggregation across sub-projects.
  • Refactor the project to use the testAggregateTestReport support. We do this in a way where all sub-projects are automatically added for report aggregation. Previously, each project had to be hand-configured, which caused us to miss a number of new projects as they were added (including output-comparator, per the bug).
  • Fix all the tests that were actually failing and we didn't know about it (mostly on Windows).
  • I also took this opportunity to move a bunch of repeated unit-test configuration from individual build.gradle files and specify it once in the root project.

@bdferris-v2
Upgrade to Gradle 7.4 so that we can use the new `testReportAggregati…
e53bef7 
…on`, which is a more standard way of aggregating test reports across all sub-projects. Use this opportunity to consolidate a bunch of test options into the root project for consistent configuration.
@bdferris-v2
Comment tweak.
139d140
@bdferris-v2
Restore code to make sure copyRulesMarkdown runs before test execut…
0b5b8f3 
…ion.
@bdferris-v2
Make sure affected sources are in consistent sorted order for repeata…
a8952b6 
…ble test execution.
@bdferris-v2
Disable UI tests in headless environments.
807d8a8
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 7, 2023

✅ Rule acceptance tests passed.
New Errors: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1425 sources (~0 %) are corrupted.
Commit: 989023d
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

@bdferris-v2 bdferris-v2 requested a review from davidgamez April 7, 2023 06:28
@bdferris-v2 bdferris-v2 linked an issue Apr 7, 2023 that may be closed by this pull request
output-comparator sub-module is not being included in test_pack_doc.yml workflows #1381
Closed
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 7, 2023

✅ Rule acceptance tests passed.
New Errors: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1425 sources (~0 %) are corrupted.
Commit: dd3b81e
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 7, 2023

✅ Rule acceptance tests passed.
New Errors: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1425 sources (~0 %) are corrupted.
Commit: 433756e
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 11, 2023

✅ Rule acceptance tests passed.
New Errors: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1425 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1425 sources (~0 %) are corrupted.
Commit: 14a2ba0
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

@bdferris-v2 bdferris-v2 merged commit f2f43e3 into master Apr 11, 2023
@bdferris-v2 bdferris-v2 deleted the issue/1381/all_tests branch April 11, 2023 16:36
@davidgamez davidgamez mentioned this pull request Apr 13, 2023
Merged
4 tasks
bradyhunsaker pushed a commit to bradyhunsaker/gtfs-validator that referenced this pull request Apr 25, 2023
@bdferris-v2 @bradyhunsaker
fix: Run all sub-project tests in test_pack_doc.yml workflow (Mobil…
95f2f58 
…ityData#1382)

* Upgrade to Gradle 7.4 so that we can use the new `testReportAggregation`, which is a more standard way of aggregating test reports across all sub-projects.  Use this opportunity to consolidate a bunch of test options into the root project for consistent configuration.

* Comment tweak.

* Restore code to make sure `copyRulesMarkdown` runs before test execution.

* Make sure affected sources are in consistent sorted order for repeatable test execution.

* Disable UI tests in headless environments.

* Fix failing test on Windows due to line separators.

* Attempt MobilityData#2 to test correctly under Windows.

* Consolidate Java compile options for consistency across projects.  This will also help some tests pass on Windows environments.

* Get another test passing on Windows.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@davidgamez davidgamez davidgamez approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

output-comparator sub-module is not being included in test_pack_doc.yml workflows

2 participants

@bdferris-v2 @davidgamez