I've noticed an increased incidence of GitHub workflow action failures (link) for our end_to_end and acceptance_test workflows. Digging deeper into these failures, the validator itself isn't failing. Instead, our workflow runners are being terminated with the following message:

The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

The process itself includes the following in its logs:

Error: Process completed with exit code 143.

This is consistent with the process being terminated externally via SIGTERM.

So what's actually going on? Other folks are reporting the same issue to GitHub and it's pointed out that most of these runners are using ubuntu-latest, which has been upgraded to 22.04 over the past week (link). That potentially changes the resource envelope of the running worker, which might lead to resource exhaustion at different points than the previous runner.

Indeed, looking at our action failures, it seems to occur most often for large feeds such as no-unknown-agder-kollektivtrafikk-as-gtfs-1078 (11M stop-time entries) and de-unknown-ulmer-eisenbahnfreunde-gtfs-1081 (huge shapes.txt file).

So what do we do about this? Still digging into that.

But just to check, @aababilov and @asvechnikov2, your recent changes over the past month should have reduced memory usage for large feeds, not increased it, correct?