You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
msgspec is a performance oriented library - would be useful to set up continuous benchmarking as part of our CI to catch any performance regressions. These might make use of airspeed velocity, or something else.
From reading their Python client code, it looks like they're using callgrind to handle instrumentation. Depending on metrics used this may be a nice way to eliminate variation between runs in a public CI environment. Worth considering at least.
Running benchmarks using a traditional statistical approaches is usually unreliable on CI environments. To measure the performance of your code in an accurate and reproducible way, CodSpeed uses various instruments when running your code. Those instruments isolate noisy neighbors and all the side effect they can have on when measuring the performance of our workloads. 1
I'm not super convinced this is viable for the sort of measurements we are taking; the required accuracy is quite high, and to get comparable results in the microsecond realm between entirely different runners sounds implausible.
fyi, codespeed also have bare metal runners, might be similar to bencher
Ah! I didn't know that, thanks for sharing! Will check that out as well. bencher.dev setup is quite tedious and I'm not suuper happy about the reporting, so maybe that codespeed bare metal thing is the best compromise.
I'm not super impressed by the results to be honest. With no changes, it's still showing a variance of up to 5% between runs, so it doesn't seem super reliable.
codspeed also seems to have some random fluctuations (~50% in one case: provinzkraut#5 (comment)). They do report that this was because the benchmarks ran on different GitHub runners, but I'm still no super convinced. Seems that the "cpu simulation" approach has some limitations as to the stability it can provide.
Maybe still worth a try, and we can see how good it works for us?
Re variance: On average codspeed seems to be the best in terms of variance, because of its instrumentation. The variance I've observed is when benchmarks with very tight timings are run on a GH Actions Runner with a significantly different spec. Codespeed flags this though, so we could easily re-run in those cases
I've also managed to get it to run on amd64 and arm, so we got the most important targets covered IMO (targeting macOS + Windows would be a nice to have, but amd+arm is important to test, because we do have some switches and ops that behave differently there).
So overall, it sounds like we should give codspeed a try. I'll see that I can get my experiments cleaned up, and prepare a PR shortly.
Activity
jcrist commentedon Feb 10, 2023
A few links that might be useful:
jcrist commentedon Feb 24, 2023
I've seen some libraries are using CodSpeed (https://codspeed.io/) for this now. I'm slightly hesitant to rely on a new closed-source service.
From reading their Python client code, it looks like they're using callgrind to handle instrumentation. Depending on metrics used this may be a nice way to eliminate variation between runs in a public CI environment. Worth considering at least.
jcrist commentedon Feb 25, 2023
Another potentially useful reference: https://pythonspeed.com/articles/consistent-benchmarking-in-ci/
provinzkraut commentedon Jun 13, 2026
One thing to note about https://codspeed.io/ is that it runs the benchmark itself within the GitHub actions runner. While they claim that
I'm not super convinced this is viable for the sort of measurements we are taking; the required accuracy is quite high, and to get comparable results in the microsecond realm between entirely different runners sounds implausible.
https://bencher.dev would be an alternative, where the benchmark is actually run in the cloud, on a bare metal machine, and they guarantee that the same benchmarks run on the same machine every time, which sounds good. It's also open source.
Footnotes
https://codspeed.io/docs/what-is-codspeed ↩
ofek commentedon Jun 13, 2026
Bencher looks way better, nice find!
UnknownPlatypus commentedon Jun 13, 2026
fyi, codespeed also have bare metal runners, might be similar to bencher
provinzkraut commentedon Jun 14, 2026
Ah! I didn't know that, thanks for sharing! Will check that out as well. bencher.dev setup is quite tedious and I'm not suuper happy about the reporting, so maybe that codespeed bare metal thing is the best compromise.
provinzkraut commentedon Jun 14, 2026
I've set up bencher.dev in a fork, you can see the results here: provinzkraut#2 (comment)
I'm not super impressed by the results to be honest. With no changes, it's still showing a variance of up to 5% between runs, so it doesn't seem super reliable.
I'll evaluate codspeed next.
provinzkraut commentedon Jun 14, 2026
Mhm. Codspeed only supports linux, including the bare metal runners. We can do amd and arm, but can't test perf on macOS or windows.
macOS seems to be supported in some limited capacity: CodSpeedHQ/codspeed#123 (comment)
So, we can have:
Semi-homegrown benchmarks in CI
bencher.dev
codspeed
@msgspec/committers would like some input from you how to best proceed with this.
provinzkraut commentedon Jun 14, 2026
codspeed also seems to have some random fluctuations (~50% in one case: provinzkraut#5 (comment)). They do report that this was because the benchmarks ran on different GitHub runners, but I'm still no super convinced. Seems that the "cpu simulation" approach has some limitations as to the stability it can provide.
Maybe still worth a try, and we can see how good it works for us?
ofek commentedon Jun 14, 2026
I'm curious what we all think but I personally would rank the priorities as:
sobolevn commentedon Jun 14, 2026
I am using codspeed in
django-modern-rest: wemake-services/django-modern-rest#1074 (comment) It kinda works for me, I didn't see any problems in run variations for the last 4 months.provinzkraut commentedon Jun 14, 2026
Re variance: On average codspeed seems to be the best in terms of variance, because of its instrumentation. The variance I've observed is when benchmarks with very tight timings are run on a GH Actions Runner with a significantly different spec. Codespeed flags this though, so we could easily re-run in those cases
I've also managed to get it to run on amd64 and arm, so we got the most important targets covered IMO (targeting macOS + Windows would be a nice to have, but amd+arm is important to test, because we do have some switches and ops that behave differently there).
So overall, it sounds like we should give codspeed a try. I'll see that I can get my experiments cleaned up, and prepare a PR shortly.
sobolevn commentedon Jun 14, 2026
@provinzkraut feel free to take my setup: