Skip to content

feat: Update acceptance test report to include changes in WARNINGS.#1354

Merged
bdferris-v2 merged 9 commits intomasterfrom
issue/acceptance_test_warnings
Mar 17, 2023
Merged

feat: Update acceptance test report to include changes in WARNINGS.#1354
bdferris-v2 merged 9 commits intomasterfrom
issue/acceptance_test_warnings

Conversation

@bdferris-v2
Copy link
Copy Markdown
Collaborator

@bdferris-v2 bdferris-v2 commented Mar 13, 2023

Refactor the acceptance test validation report comparison logic to consider changes to WARNING-level notices, not just ERRORs. We could more easily expand this to include INFO-level notices as well, if we so desire. Currently, this change does not cause automatic blocking of a change if the # of warnings changes significantly, but we could change that as well.

As part of this change, I refactored a bunch of logic out of ValidationReport and moved it into a new NoticeSet to be separate some concerns. I also refactored a bunch of references to "errors" to more generic "notices".

Closes #1349.

Please make sure these boxes are checked before submitting your pull request - thanks!

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 13, 2023

✅ Rule acceptance tests passed.
New Errors: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1420 sources (~0 %) are corrupted.
Commit: ad4009a
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

davidgamez
davidgamez approved these changes Mar 17, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@davidgamez davidgamez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 17, 2023

✅ Rule acceptance tests passed.
New Errors: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1420 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1420 sources (~0 %) are corrupted.
Commit: bd3a8b4
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

@bdferris-v2 bdferris-v2 merged commit aa4c5a9 into master Mar 17, 2023
@bdferris-v2 bdferris-v2 deleted the issue/acceptance_test_warnings branch March 17, 2023 18:15
@isabelle-dr
Copy link
Copy Markdown
Contributor

isabelle-dr commented Mar 17, 2023

Do we need to update https://github.com/MobilityData/gtfs-validator/blob/master/docs/ACCEPTANCE_TESTS.md as a result of this PR?

@bdferris-v2 bdferris-v2 mentioned this pull request Mar 22, 2023
Closed
ryon pushed a commit to JarvusInnovations/gtfs-validator that referenced this pull request Apr 1, 2023
@bdferris-v2 @ryon
feat: Update acceptance test report to include changes in WARNINGS. (M…
5464a12 
…obilityData#1354)

* Track changes to warnings in addition to errors in acceptance tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@davidgamez davidgamez davidgamez approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Acceptance Tests only report changes in ERROR notices, not WARNINGS or INFOs

3 participants

@bdferris-v2 @isabelle-dr @davidgamez