Automated tests to see if a PR will trigger new errors in datasets #1045
Description
Description
In order for the Canonical GTFS Schedule Validatorto to be safely integrated with different pipelines, we need to ensure the stability of this tool. Currently, the Pull Request (PR) process contains some tests that are making sure the code is working properly by running it on selected datasets.
With this project, we want to run a new test (acceptance tests) that can measure the impact of a new rule (or any Pull Request) on all available datasets from the Mobility Database, with the objective of ensuring we know when a PR is introducing errors in previously valid datasets.
Outcome
Automatically flag Pull Requests that are introducing new errors in the datasets that are present in the Mobility Database.
How will it work?
For each commit in a Pull Request, we'll be using Github actions to run the validator on all datasets from the mobility database and the validation results will be compared.
If more than 1% of the datasets have 1 additional error, the test won't pass.
A commit message will allow not to run this test, in order to avoid the frustrations of having to wait for the test to be over at every commit.
✅ If the acceptance test pass, the PR is accepted and can safely be merged.
❌ If the test doesn't pass, the user can see the acceptance test validation report to get more information (how many datasets are invalidated with this PR? Which ones?).
If there is no error with the code or the logic of a rule and this PR is "truly" resulting in datasets getting additional errors, a discussion should happen to decide when to do, depending on the situation:
- If the new rule is invalidating a majority of datasets, we will discuss the possibility of updating the spec to reflect practice, and the decision to merge or not the PR will be treated on a case by case basis.
- If the new rule is invalidating a handful of datasets, the PR can still be merged but the acceptance test results have to be communicated to producers and the consumers because their pipeline might be impacted by it.