-
-
Notifications
You must be signed in to change notification settings - Fork 457
Description
Currently, the wall time is reported with detailed statistics both in the TUI (mean + stddev + median + min + max), and in the JSON output (list of all the wall times via the times field).
However, the user and system times only report one number. The UI doesn't specify which number it is, but looking at the code reveals that it's the mean (https://github.com/sharkdp/hyperfine/blob/master/src/benchmark/mod.rs#L376-L377).
In general, I think that the other statistics are relevant, as the mean can suffer from outlier bias and doesn't show the whole picture. And of course a detailed distribution plot can be worthwhile in some cases (as the various scripts allow), which requires the list of all the recorded times.
As such, would you be open to export more details of the user/system times?
- At the minimum, I think that adding
user_timesandsystem_timesarrays to the JSON output would allow further tools to reconstruct the median and stddev (or any other interesting statistic). - Additionally, the JSON output could also be extended with
user_stddev,user_median,user_min,user_maxfields (and likewise for system). - Lastly (but it's maybe more disruptive), the TUI could also report these details for user/system, when some CLI flag is enabled.
What do you think? I'll be happy to contribute PR(s) if you think some of these are worth adding.