mypy_primer results

freqtrade (https://github.com/freqtrade/freqtrade)
-     memo fields = ~304MB
+     memo fields = ~276MB

schemathesis (https://github.com/schemathesis/schemathesis)
- TOTAL MEMORY USAGE: ~156MB
+ TOTAL MEMORY USAGE: ~171MB

alerta (https://github.com/alerta/alerta)
- TOTAL MEMORY USAGE: ~117MB
+ TOTAL MEMORY USAGE: ~106MB

paasta (https://github.com/yelp/paasta)
-     memo fields = ~156MB
+     memo fields = ~171MB

scikit-learn (https://github.com/scikit-learn/scikit-learn)
- TOTAL MEMORY USAGE: ~652MB
+ TOTAL MEMORY USAGE: ~717MB

It looks like this is about a minute slower (4m as compared to 3m), which seems reasonable? A separate job would likely take more than a minute anyways.

Running ty in single-threaded mode in the mypy-primer mode should fix this.

I'm assuming this can only be verified after merging this PR?

It looks like this is about a minute slower (4m as compared to 3m), which seems reasonable? A separate job would likely take more than a minute anyways.

It looks more like ~1m30s slower to me -- it looks like primer generally takes around 3m10s on main, and it takes 4m50s with this PR.

I really love how quickly we get feedback from mypy_primer on PRs currently -- it makes it very easy to quickly spot bugs and iterate on PRs. From that perspective, I would selfishly sort-of prefer the memory report to be a separate primer job that runs concurrently with the existing primer job, so that we can still get feedback on the impact on diagnostics as quickly as possible. I'd expect ty PRs to change the diagnostics being emitted much more often than they have a significant impact on memory usage, anyway.

Neither CI job would be marked as "required", so we would still be able to merge PRs even if one or neither job hadn't yet completed and we wanted to merge a trivial PR quickly. The only cost would be additional CI resources -- but I actually think that having primer reports be as quick as possible might be worth that cost.

Curious if @sharkdp has opinions

Curious if @sharkdp has opinions

I was hesitant to bring it up, but I fully agree with you. mypy_primer feedback is also a crucial development tool for me. I am often waiting for its output. So in that sense, every saved minute is a developer-experience improvement. But it's probably also a bit annoying to create a completely separate job for this.

I guess the problem with this approach is that we're bottlenecked on the largest project instead of evenly spreading the load. I'm fine to make this a separate job if the speed is important. There's also probably some things we could do to speed it up as is.