So we're going to run the benches on cloud VMs right now? Hm, I understand that's what most people are running so in a way it makes sense, although it might make dealing with the benchmarks more painful due to the inherent variability of cloud machines.

Two random possibilities I see which could make this better.

Just use bare metal anyway

Run the benchmarks on bare metal, and manually calculate a static multiplier(s) which would allow us to translate the results from bare hardware into the cloud machine, and have those as official weights.

Use multiple cloud machines in parallel

The benchmark bot is triggered to run benches.
It runs the benches not on one machine, but on X machines in parallel (where X is picked high enough to make them "stable"); every run generates a .json with the results
After they all finish the bot takes all of the results and merges them into one big json, and then generates the weights for that. (I've added the ability to generate the weights from the json file without running the benches so this is mostly done; you'd just need to write some simple program to merge the jsons)

Basically you'd essentially have the benchmarks which average the results from one machine (so it reduces the variability within that single VM), and a "meta" benchmark like this which sits a lever higher and averages results from multiple machines (which reduces the variability across the VMs).

We'd also have to make sure that our VMs are going to be put on separate machines. Not sure if/which cloud providers provide such service, but an easy way to guarantee this would be to just pick the same type of VMs in multiple regions.

Update hardware requirements for benchmark machine #13308

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Update hardware requirements for `benchmark machine` #13308