Repository navigation
How we Visualize Benchmarks #117
Description
Throwing out a suggestion. Rather than ASV I recommend https://docs.codspeed.io/.
codspeed is really good, free for open-source, and integrates deeply with pytest (see pytest-benchmark). With codspeed + pytest ecosystem it's easy for use to have a specific benchmark test suite and also have benchmarked tests in the normal test suite.
Activity
Cadair commentedon Apr 20, 2024
you lost me at "free for open source" I will generally push back against anything which isn't actually open source, especially if it's a hosted service at the whims of VC (I am assuming).
nstarman commentedon Apr 20, 2024
Like many of GH's tools? Or RTD? Or Pre-commit.ci? 😁 I think most of our CI is in the category "free for open source".
matteobachetti commentedon Apr 20, 2024
Yes, but also like Travis CI 😜
I see what @Cadair says, maybe I'm not so negative about it, but "free for open source" is not really something that screams trustworthiness these days
nstarman commentedon Apr 20, 2024
I'm just not sure where we would go from there. We should definitely vet our tools. But if we fundamentally don't trust anything in industry then we shouldn't use GH Actions under the free-for-open-source tier (we do), RTD under the free-for-open-source tier, Pre-commit under the same, encourage new users to use GH virtual development environments, CircleCI, etc. Our actual inert code is one of the few things that isn't in a free-for-open-source tier. And Codspeed is kind of like codecov, in that it is a user-friendly layer on top of open-source tooling, like pytest-benchmark.
Now, maybe we vet it and don't like it, but that hasn't happened yet.
hamogu commentedon Apr 20, 2024
We actually pat for ReadtheDocs because we think they need the money, plus we wanted something fomr them, but I now forget what. Details are here astropy/astropy-project#105
hamogu commentedon Apr 20, 2024
Personally, I'm less worried about something like codespeed.io going away. It might be helpful, but it's not critical. If it goes away, we're just where we are now without regular benchmarks. That's different from e.g. github itself; while we could move to bitbucket or gitlab, that would be a lot more disruptive. So the question is how much effort it would be to set up. If someone can set it on in 4 h, and they start charging, we only loose 4 h of work. If it's a lot more effort to set up an maintain, then we should be more careful in vetting.
nstarman commentedon Apr 20, 2024
https://docs.codspeed.io/#how-long-does-it-take-to-install
pllim commentedon Apr 22, 2024
I don't think this replaces the part where we run the benchmark for every commit on
mainetc? Looks like it only does PR continuous integration. I would like a more technical blog on the pros and cons, and what projects have used this service to-date and what lessons are learned.nstarman commentedon Apr 23, 2024
https://news.ycombinator.com/item?id=36682012
https://blog.pydantic.dev/blog/2022/07/10/pydantic-v2-plan/#performance
codspeedwas one of the tools used by Pydantic v2 to achieve their 17x speedup.[-]How we Benchmark[/-][+]How we Visualize Benchmarks[/+]