Skip to content

tests: Speed up test_batch_run#2039

Merged
rht merged 2 commits intomesa:mainfrom
rht:test-faster
Feb 16, 2024
Merged

tests: Speed up test_batch_run#2039
rht merged 2 commits intomesa:mainfrom
rht:test-faster

Conversation

@rht
Copy link
Copy Markdown
Contributor

@rht rht commented Feb 16, 2024

This is an attempt to speed up overall CI testing time, as much as possible without reducing comprehensiveness. Currently, it takes 12s on Python 3.12 on Ubuntu.

@github-actions
Copy link
Copy Markdown

Performance benchmarks:

Model Size Init time [95% CI] Run time [95% CI]
Schelling small 🔵 +0.7% [+0.5%, +0.9%] 🔵 -0.1% [-0.2%, +0.1%]
Schelling large 🔵 +0.0% [-1.7%, +1.9%] 🔵 +0.5% [-0.5%, +1.6%]
WolfSheep small 🔵 +0.7% [+0.4%, +1.0%] 🔵 +0.1% [-0.0%, +0.2%]
WolfSheep large 🔵 +0.4% [-1.2%, +2.0%] 🔵 +2.4% [+1.0%, +4.0%]
BoidFlockers small 🔵 +0.5% [-0.1%, +1.1%] 🔵 +0.3% [-0.4%, +0.9%]
BoidFlockers large 🔵 +0.1% [-0.3%, +0.5%] 🔵 -0.1% [-0.6%, +0.4%]

@quaquel
Copy link
Copy Markdown
Member

quaquel commented Feb 16, 2024

Can you explain why you are concerned about CI testing time?

@rht
Copy link
Copy Markdown
Contributor Author

rht commented Feb 16, 2024

  1. It speeds up the review feedback loop between the reviewers and PR (that has been marked as ready) authors. The sub 1 min vs 2 min+ speedup benefit compounds for a single PR that requires several testing such as Add cell-centric discrete spaces (experimental) #1994. The Ubuntu CI runs used to be ~1 min, now about 2x faster at 30s.
  2. More immediate feedback to contributors.

@quaquel
Copy link
Copy Markdown
Member

quaquel commented Feb 16, 2024

Fair enough, although this does not make a massive difference for me. I always think of this in a more categorical manner: can I wait behind my desk, can I go for a cup of coffee, can it run over lunch, or does it take even longer? The difference between 30 seconds and 2 minutes is still very much in the behind-my-desk category.

Also, I see CI stuff as stuff that runs after having tested everything locally anyway, so I don't really see the feedback loop argument.

To be clear: speedups are still nice.

Copy link
Copy Markdown
Member

@EwoutH EwoutH left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see significant degradation in test functionality, so looks good. Faster CI is always nice, thanks!

@rht rht merged commit ac9fdd1 into mesa:main Feb 16, 2024
@rht rht deleted the test-faster branch February 16, 2024 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants