LGTM modulo that I think doing the average (or maybe the worst time? but probably the average) would be more representative and that we should remove the cache before each run of the analyzis subprocess rather than before the run of four analyses.

<bikeshed>
There's an argument that taking the best time is most representative of what's actually measured by the benchmark, since includes everything that happens on every benchmark run, but excludes things that slow down the benchmark, but only happen some of the time (for example, caches not being warmed up or the machine it's running on having a bit more load than usual). To accurately measure worst-case, you want to make sure it happens on every run (by clearing caches, etc) rather than trusting to chance.
</bikeshed>

Taking the smallest time would entirely hide cases where the code takes substantially longer on first or second run (e.g. because the code has a bi-modal bug where on odd-numbered runs it runs a different code path than even-numbered runs).

We are actually intentionally clearing the .dartServer cache between runs, but also we're not interested in the absolute numbers, but rather regressions.

Yes, clearing the cache every time is good.

It's true that taking the minimum hides bi-modal bugs but I claim that's a feature. Any average will hide variation between test runs to some extent (that's kind of the point). For benchmarking, taking the minimum seems to do a better job of it, revealing regressions that affect every test run, which tend to be easier to work on.

Showing variation between runs is also useful for detecting some bugs and understanding noise added by the environment, but I think it's less confusing to plot it separately. If there's enough data, plotting the median, 90%, 99%, and so on would be useful. Or in this case I think there are only three points, so maybe just plot all of them?

(But I don't think it's important to make a change now; just theorizing.)

@skybrian good points. Next time I get cycles to work on benchmarks, I'll see if I can come up with a more scalable system. Right now it's very simple, a chart card for every metric, which is great for maintenance, but doesn't scale well if we want to collect more numbers. That the only reason we limit the metrics we publish.

For some of the other benchmarks we actually plot the average and the worst time. The worst time is (obviously) very noisy, but it is still a useful metric to track, and has shown regressions. It's also useful to see the actual worst case variance (e.g. complex_layout_scroll_perf_ios__timeline_summary worst_frame_build_time_millis shows we regularly have 2x worst-case times than normal).