try reducing codspeed variance - #1082

• Apply suggestions from https://codspeed.io/blog/why-glibc-faster-github-actions#how-to-fix-it-
• Use a fixed seed for pytest-randomly: This was actually a big source of variance in 2 of our tests
• Use warmup where necessary (had been disabled before per codspeed recommendation, but seems to make things more stable in a few cases)

Re-ran the benchmarks a couple of times now, and they seem more stable within this branch. Let's see if this also works across branches.