ci: Add more feeds to end-to-end GitHub workflow#736
ci: Add more feeds to end-to-end GitHub workflow#736barbeau merged 15 commits intoMobilityData:masterfrom
Conversation
touch for workflow extension
As suggested by @timMillet here: MobilityData#712 (comment)
- road to a 100, 25 new feeds + original 11
nackko
changed the title
|
Great! The more feeds we test, the better. Thank you, Fabrice, LGTM! |
|
Cool! Note that it seems the feeds don't pass validation for what looks like an out of memory error with Norway. |
-set Java heap space to maximum size -https://docs.github.com/en/actions/reference/specifications-for-github-hosted-runners#supported-runners-and-hardware-resources -https://alvinalexander.com/blog/post/java/java-xmx-xms-memory-heap-size-control/ -add 25 feeds
|
@aababilov Increasing heap size in the run command fixed the IDFM feed but it seems there is an issue with Bueno Aires even with the increased heap size. I'll comment it until I get to a 100 feed at which point I'll switch the PR status to ready and let you guys decide. EDIT: turns out the Buenos Aires feed just took a significantly longer time to validate (90 minutes). |
Now roughly 100 feeds
A feed is big when it requires more memory than the default amount to be processed.
|
Thanks for working on this @f8full! Is this ready for review/merge? Also, just to clarify, this PR will ensure that the validator doesn't crash on all of these feeds. However, it won't flag if we output errors on datasets that should be clean - that type of testing is being discussed in #734. |
|
Yes @barbeau , it is ready. It doesn't analyse any reports generated so it can show crashes and feeds that take significantly longer to validate. I also proposed to split feeds that need special memory configuration in their own workflow. |
|
And you're welcome! :) |
barbeau
changed the title
barbeau
left a comment
barbeau
left a comment
There was a problem hiding this comment.
Thanks @f8full, this is great! 💯
|
Always a pleasure! :) |
|
cc @e-lo @mcplanner |
Following the work done in #736
Following the work done in #736
3 participants
Summary:
Following the conversation in #712 , this PR extends end to end worflow as follow
end_to_end_100.ymlwhich contains 100 feeds, including the remaining ones from Tim's suggestionend_to_end_big.ymldedicate to feeds that need more memory than the default amount to be validated successfully.They were taken from http://transitfeeds.com/feeds somewhat randomly as long as they didn't seem staled. Some exceptions were made for big or original data.
Expected behavior:
Please make sure these boxes are checked before submitting your pull request - thanks!
- [ ] Run the unit tests withNO CODE CHANGEgradle testto make sure you didn't break anything- [ ] Linked all relevant issues