Currently redis-benchmark outputs some strange latency distribution results that make it very hard to graph consistently when comparing multiple runs that give different output. Part of this is because it's not measuring microseconds (μs). redis-benchmark omits latency results smaller than 1 millisecond which means sometimes the latency distribution output only starts at higher percentages.

HdrHistogram is perfect for this solution and there's even a website for plotting latency distribution graphs. Here's an exmaple of the file format.

HdrHistogram output is more consistently and more accurately than redis-benchmark outputs currently. Wrk2 is an example of Wrk written in C that implements HdrHistogram.

This combined with #3534 (running benchmark for a duration) and the -r argument would basically work the same as wrk2 -R argument.

I can run redis-benchmark multiple times and one time I might get percentiles like this.

Running benchmark against redis on localhost:6379
====== SET ======
  50000 requests completed in 0.15 seconds
  32 parallel clients
  3 bytes payload
  keep alive: 1

0.13% <= 1 milliseconds
3.33% <= 2 milliseconds
68.64% <= 3 milliseconds
98.79% <= 4 milliseconds
100.00% <= 4 milliseconds
328947.38 requests per second

And another time like this.

Running benchmark against redis on localhost:6379
====== SET ======
  50000 requests completed in 0.15 seconds
  32 parallel clients
  3 bytes payload
  keep alive: 1

0.07% <= 1 milliseconds
2.82% <= 2 milliseconds
80.61% <= 3 milliseconds
99.68% <= 4 milliseconds
100.00% <= 4 milliseconds
342465.75 requests per second

None of those percentiles compare which means the points on the graph will be misaligned and also a different amount of them can occur.