Added INFO LATENCYSTATS section: latency by percentile distribution/latency by cumulative distribution of latencies by filipecosta90 · Pull Request #9462 · redis/redis

filipecosta90 · 2021-09-05T23:43:30Z

Short description

The Redis extended latency stats track per command latencies and enables:

exporting the per-command percentile distribution via the INFO LATENCYSTATS command. ( percentile distribution is not mergeable between cluster nodes ).
exporting the per-command cumulative latency distributions via the LATENCY HISTOGRAM command. Using the cumulative distribution of latencies we can merge several stats from different cluster nodes to calculate aggregate metrics .

By default, the extended latency monitoring is enabled since the overhead of keeping track of the command latency is very small.

If you don't want to track extended latency metrics, you can easily disable it at runtime using the command:

CONFIG SET latency-tracking no

By default, the exported latency percentiles are the p50, p99, and p999. You can alter them at runtime using the command:

CONFIG SET latency-tracking-info-percentiles "0.0 50.0 100.0"

Some details:

The total size per histogram should sit around 40 KiB. We only allocate those 40KiB when a command was called for the first time.
With regards to the WRITE overhead As seen below, there is no measurable overhead on the achievable ops/sec or full latency spectrum on the client. Including also the measured redis-benchmark for unstable vs this branch.
We track from 1 nanosecond to 1 second ( everything above 1 second is considered +Inf )

overhead test using redis-benchmark on unstable vs commands.latency.histogram

I've tested using ping and set commands ( fast ones ) to assess the largest possible overhead of histogram latency tracking.
As seen below there is no measurable overhead on the achievable ops/sec or full latency spectrum on the client. This matches the expected behaviour, given on previous microbenchmarks of the hdr_record_value (the function used) >>see reference<< took on average 5-6 ns/op.

results on unstable ( commit hash = `af0b50f` )

Used redis-server command

taskset -c 0 ./src/redis-server --save "" --daemonize yes

taskset -c 1,2 redis-benchmark -c 50 --threads 2 -d 1024 -r 1000000 -n 10000000 --csv -t ping,set,get
"test","rps","avg_latency_ms","min_latency_ms","p50_latency_ms","p95_latency_ms","p99_latency_ms","max_latency_ms"
"PING_INLINE","212512.75","0.220","0.040","0.215","0.279","0.295","24.383"
"PING_MBULK","220716.45","0.213","0.040","0.215","0.263","0.287","28.127"
"SET","175198.86","0.269","0.040","0.263","0.359","0.391","20.383"
"GET","179121.59","0.263","0.040","0.263","0.351","0.367","32.127"

results on commands.latency.histogram branch with latency track disabled

Used redis-server command

taskset -c 0 ./src/redis-server --save "" --latency-tracking no --daemonize yes

taskset -c 1,2 redis-benchmark -c 50 --threads 2 -d 1024 -r 1000000 -n 10000000 --csv -t ping,set,get
"test","rps","avg_latency_ms","min_latency_ms","p50_latency_ms","p95_latency_ms","p99_latency_ms","max_latency_ms"
"PING_INLINE","212408.94","0.221","0.040","0.223","0.279","0.295","24.383"
"PING_MBULK","219375.22","0.214","0.032","0.215","0.263","0.287","28.367"
"SET","173644.27","0.270","0.040","0.263","0.367","0.391","28.207"
"GET","177619.89","0.265","0.040","0.263","0.351","0.375","28.127"

results on commands.latency.histogram branch with latency track enabled

taskset -c 1,2 redis-benchmark -c 50 --threads 2 -d 1024 -r 1000000 -n 10000000 --csv -t ping,set,get
"test","rps","avg_latency_ms","min_latency_ms","p50_latency_ms","p95_latency_ms","p99_latency_ms","max_latency_ms"
"PING_INLINE","211528.28","0.223","0.040","0.223","0.279","0.295","24.191"
"PING_MBULK","219664.36","0.214","0.040","0.215","0.263","0.287","24.127"
"SET","175204.98","0.271","0.040","0.263","0.359","0.383","24.399"
"GET","179172.94","0.263","0.040","0.263","0.351","0.375","24.239"

`INFO LATENCYSTATS` exposition format

Format: latency_percentiles_usec_<CMDNAME>:p0=XX,p50....

latencystats output sample:

root@hpe10:~# redis-cli INFO LATENCYSTATS
# Latencystats - latency by percentile distribution
latency_percentiles_usec_zadd:p50.000000=0.001,p99.000000=1.003,p99.900000=4.015
latency_percentiles_usec_hset:p50.000000=0.001,p99.000000=1.003,p99.900000=4.015
latency_percentiles_usec_rpop:p50.000000=0.001,p99.000000=1.003,p99.900000=6.015
latency_percentiles_usec_set:p50.000000=0.001,p99.000000=1.003,p99.900000=4.015
latency_percentiles_usec_zpopmin:p50.000000=0.001,p99.000000=1.003,p99.900000=3.007
latency_percentiles_usec_spop:p50.000000=0.001,p99.000000=1.003,p99.900000=3.007
latency_percentiles_usec_lpush:p50.000000=0.001,p99.000000=1.003,p99.900000=4.015
latency_percentiles_usec_ping:p50.000000=0.001,p99.000000=1.003,p99.900000=1.003
latency_percentiles_usec_lpop:p50.000000=0.001,p99.000000=1.003,p99.900000=5.023
latency_percentiles_usec_lrange:p50.000000=14.015,p99.000000=22.015,p99.900000=56.063
latency_percentiles_usec_incr:p50.000000=0.001,p99.000000=1.003,p99.900000=4.015
latency_percentiles_usec_get:p50.000000=0.001,p99.000000=1.003,p99.900000=3.007
latency_percentiles_usec_mset:p50.000000=1.003,p99.000000=1.003,p99.900000=2.007
latency_percentiles_usec_rpush:p50.000000=0.001,p99.000000=1.003,p99.900000=4.015
latency_percentiles_usec_sadd:p50.000000=0.001,p99.000000=1.003,p99.900000=3.007

`LATENCY HISTOGRAM [command ...]` exposition format

Return a cumulative distribution of latencies in the format of a histogram for the specified command names.

The histogram is composed of a map of time buckets:

Each representing a latency range, between 1 nanosecond and roughly 1 second.
Each bucket covers twice the previous bucket's range.
Empty buckets are not printed.
Everything above 1 sec is considered +Inf.
At max there will be log2(1000000000)=30 buckets

We reply a map for each command in the format <command name> : { calls: <total command calls> , histogram : { <bucket 1> : latency , < bucket 2> : latency, ... } }

RESP2:

 redis-cli latency histogram set spop
1) "set"
2) 1) "calls"
   2) (integer) 100000
   3) "histogram_usec"
   4) 1) (integer) 1
      2) (integer) 99986
      3) (integer) 4
      4) (integer) 99991
      5) (integer) 16
      6) (integer) 100000
3) "spop"
4) 1) "calls"
   2) (integer) 100000
   3) "histogram_usec"
   4) 1) (integer) 1
      2) (integer) 99991
      3) (integer) 4
      4) (integer) 99995
      5) (integer) 8
      6) (integer) 99997
      7) (integer) 16
      8) (integer) 100000

RESP3:

# redis-cli -3 latency histogram
 1# "set" => 1# "calls" => (integer) 100000
    2# "histogram_usec" => 1# (integer) 1 => (integer) 99986
       2# (integer) 4 => (integer) 99991
       3# (integer) 16 => (integer) 100000
 2# "spop" => 1# "calls" => (integer) 100000
    2# "histogram_usec" => 1# (integer) 1 => (integer) 99991
       2# (integer) 4 => (integer) 99995
       3# (integer) 8 => (integer) 99997
       4# (integer) 16 => (integer) 100000

…tion/latency by cumulative distribution of latencies

oranagra

Code LGTM (with minor code style issues).
I've briefly read the discussion in the issue, and the compromises you made seem OK.

The one part that's not solved is the fact the request and some of the discussion was around tracking the full command processing + waiting time, and that's too complicated to handle.
Considering the low impact of this PR on both complexity and performance, I suppose we can take it even if the value is not as high as we wished.

Regarding the format, the only concern I have is what if we'll want to add more metrics in the future, like one that's including the blocked time for blocking commands.
I.E. For normal command stats (total time) we can simply add more fields, but here if we wanna do that we need to add more sections.

madolson · 2021-09-06T23:51:05Z

@filipecosta90 For the sake of completeness, can you run your perf test again with an r values greater than 100k. I'd like to make sure the benchmark holds up when basically everything isn't in some level of the CPU cache you probably tested on.

I think this is really exciting!

madolson

Also looks like you're missing module init?

src/server.c

Co-authored-by: Madelyn Olson <[email protected]>

YacineTALEB · 2021-10-07T23:55:17Z

The size of hdr_histogram struct from

redis/deps/hdr_histogram/hdr_histogram.h

Lines 17 to 35 in d96f47c

    
           struct hdr_histogram 
        
           { 
        
               int64_t lowest_trackable_value; 
        
               int64_t highest_trackable_value; 
        
               int32_t unit_magnitude; 
        
               int32_t significant_figures; 
        
               int32_t sub_bucket_half_count_magnitude; 
        
               int32_t sub_bucket_half_count; 
        
               int64_t sub_bucket_mask; 
        
               int32_t sub_bucket_count; 
        
               int32_t bucket_count; 
        
               int64_t min_value; 
        
               int64_t max_value; 
        
               int32_t normalizing_index_offset; 
        
               double conversion_ratio; 
        
               int32_t counts_len; 
        
               int64_t total_count; 
        
               int64_t* counts; 
        
           };

is 104 bytes itself. The hdr_histogram count array is stored in int64_t* counts; and is of size counts_len when initialized.

Initializing a histogram with the values from aaac165, i.e.,

LATENCY_HISTOGRAM_MIN_VALUE 1L 
LATENCY_HISTOGRAM_MAX_VALUE 1000000L 
LATENCY_HISTOGRAM_PRECISION 3

Would lead to the creation of a histogram with a counts_len of 11264. Overall the overhead size of a single histogram, according to

redis/deps/hdr_histogram/hdr_histogram.c

Lines 319 to 358 in d96f47c

    
           int hdr_calculate_bucket_config( 
        
                   int64_t lowest_trackable_value, 
        
                   int64_t highest_trackable_value, 
        
                   int significant_figures, 
        
                   struct hdr_histogram_bucket_config* cfg) 
        
           { 
        
               int32_t sub_bucket_count_magnitude; 
        
               int64_t largest_value_with_single_unit_resolution; 
        
               if (lowest_trackable_value < 1 || 
        
                       significant_figures < 1 || 5 < significant_figures || 
        
                       lowest_trackable_value * 2 > highest_trackable_value) 
        
               { 
        
                   return EINVAL; 
        
               } 
        
               cfg->lowest_trackable_value = lowest_trackable_value; 
        
               cfg->significant_figures = significant_figures; 
        
               cfg->highest_trackable_value = highest_trackable_value; 
        
               largest_value_with_single_unit_resolution = 2 * power(10, significant_figures); 
        
               sub_bucket_count_magnitude = (int32_t) ceil(log((double)largest_value_with_single_unit_resolution) / log(2)); 
        
               cfg->sub_bucket_half_count_magnitude = ((sub_bucket_count_magnitude > 1) ? sub_bucket_count_magnitude : 1) - 1; 
        
               cfg->unit_magnitude = (int32_t) floor(log((double)lowest_trackable_value) / log(2)); 
        
               cfg->sub_bucket_count      = (int32_t) pow(2, (cfg->sub_bucket_half_count_magnitude + 1)); 
        
               cfg->sub_bucket_half_count = cfg->sub_bucket_count / 2; 
        
               cfg->sub_bucket_mask       = ((int64_t) cfg->sub_bucket_count - 1) << cfg->unit_magnitude; 
        
               if (cfg->unit_magnitude + cfg->sub_bucket_half_count_magnitude > 61) 
        
               { 
        
                   return EINVAL; 
        
               } 
        
               cfg->bucket_count = buckets_needed_to_cover_value(highest_trackable_value, cfg->sub_bucket_count, (int32_t)cfg->unit_magnitude); 
        
               cfg->counts_len = (cfg->bucket_count + 1) * (cfg->sub_bucket_count / 2); 
        
               return 0; 
        
           }

would be (11264 * 8) + 104 = 90216 bytes

oranagra · 2021-11-02T16:01:22Z

@filipecosta90 do you wanna pick this up?

filipecosta90 · 2021-12-02T11:23:10Z

@filipecosta90 do you wanna pick this up?

@oranagra this weekend will focus on it. 👍

…/redis into commands.latency.histogram

…histogram

…s configuration parameters

filipecosta90 · 2021-12-21T20:58:43Z

@oranagra and @madolson I've picked this one again, and now all tests pass, and added the ability to enable/disable this at runtime, and to configure the exported percentiles.
IMO missing extra testing, and extra benchmarks, that I will address in the following days.

filipecosta90 · 2021-12-27T19:46:54Z

@filipecosta90 For the sake of completeness, can you run your perf test again with an r values greater than 100k. I'd like to make sure the benchmark holds up when basically everything isn't in some level of the CPU cache you probably tested on.

@madolson I've updated the benchmark results with the latest unstable vs this feature branch ( with the feature enabled disabled ) and a larger keyspace len as requested

overhead test using redis-benchmark on unstable vs commands.latency.histogram

I've tested using ping and set commands ( fast ones ) to assess the largest possible overhead of histogram latency tracking.
As seen below there is no measurable overhead on the achievable ops/sec or full latency spectrum on the client. This matches the expected behaviour, given on previous microbenchmarks of the hdr_record_value (the function used) >>see reference<< took on average 5-6 ns/op.

results on unstable ( commit hash = `af0b50f` )

Used redis-server command

taskset -c 0 ./src/redis-server --save "" --daemonize yes

taskset -c 1,2 redis-benchmark -c 50 --threads 2 -d 1024 -r 1000000 -n 10000000 --csv -t ping,set,get
"test","rps","avg_latency_ms","min_latency_ms","p50_latency_ms","p95_latency_ms","p99_latency_ms","max_latency_ms"
"PING_INLINE","212512.75","0.220","0.040","0.215","0.279","0.295","24.383"
"PING_MBULK","220716.45","0.213","0.040","0.215","0.263","0.287","28.127"
"SET","175198.86","0.269","0.040","0.263","0.359","0.391","20.383"
"GET","179121.59","0.263","0.040","0.263","0.351","0.367","32.127"

results on commands.latency.histogram branch with latency track disabled

Used redis-server command

taskset -c 0 ./src/redis-server --save "" --latency-track no --daemonize yes

taskset -c 1,2 redis-benchmark -c 50 --threads 2 -d 1024 -r 1000000 -n 10000000 --csv -t ping,set,get
"test","rps","avg_latency_ms","min_latency_ms","p50_latency_ms","p95_latency_ms","p99_latency_ms","max_latency_ms"
"PING_INLINE","212408.94","0.221","0.040","0.223","0.279","0.295","24.383"
"PING_MBULK","219375.22","0.214","0.032","0.215","0.263","0.287","28.367"
"SET","173644.27","0.270","0.040","0.263","0.367","0.391","28.207"
"GET","177619.89","0.265","0.040","0.263","0.351","0.375","28.127"

results on commands.latency.histogram branch with latency track enabled

Used redis-server command

taskset -c 0 ./src/redis-server --save "" --latency-track yes --daemonize yes

taskset -c 1,2 redis-benchmark -c 50 --threads 2 -d 1024 -r 1000000 -n 10000000 --csv -t ping,set,get
"test","rps","avg_latency_ms","min_latency_ms","p50_latency_ms","p95_latency_ms","p99_latency_ms","max_latency_ms"
"PING_INLINE","211528.28","0.223","0.040","0.223","0.279","0.295","24.191"
"PING_MBULK","219664.36","0.214","0.040","0.215","0.263","0.287","24.127"
"SET","175204.98","0.271","0.040","0.263","0.359","0.383","24.399"
"GET","179172.94","0.263","0.040","0.263","0.351","0.375","24.239"

…histogram

filipecosta90 · 2021-12-27T20:54:41Z

@madolson and @oranagra I believe all requested changes have been addressed. When you have the time can you check it?

redis.conf

src/config.c

src/server.c

src/server.h

tests/unit/info.tcl

ranshid

@filipecosta90 LGTM. I think @oranagra touched most of the remaining issues.
placed some small comments. but looks good. thank you for this!

src/blocked.c

tests/unit/info.tcl

…am allocators: iteration 2

deps/hdr_histogram/hdr_alloc.h

src/zmalloc.c

src/server.h

deps/hdr_histogram/hdr_alloc.h

deps/hdr_histogram/hdr_histogram.c

deps/hdr_histogram/hdr_alloc.h

…tiplied, do not wrap. zmalloc_oom_handler logs the num_items trying to allocate alongside the item size.

src/zmalloc.c

…t changing logging).

deps/hdr_histogram/hdr_alloc.h

deps/hdr_histogram/hdr_histogram.h

tests/unit/info.tcl

Co-authored-by: Oran Agra <[email protected]>

…ds:debug. Removed unecessary stddef header in hdrhistogram

…locking commands test

oranagra · 2022-01-05T11:56:37Z

For the record, the core-team approved this PR on a voice meeting.

oranagra · 2022-01-05T12:01:41Z

@filipecosta90 thank you.
can you please make a redis-doc PR for LATENCY HISTOGRAM and INFO

filipecosta90 · 2022-01-05T12:16:11Z

@filipecosta90 thank you. can you please make a redis-doc PR for LATENCY HISTOGRAM and INFO

thank you all the review cycles!

can you please make a redis-doc PR for LATENCY HISTOGRAM and INFO

will do so

filipecosta90 · 2022-01-05T14:13:30Z

@oranagra @yoav-steinberg @itamarhaber redis-doc PR: redis/redis-doc#1733

filipecosta90 added 2 commits September 6, 2021 00:13

[add] Added INFO LATENCYSTATS section: latency by percentile distribu…

aaac165

…tion/latency by cumulative distribution of latencies

Fixed CI build issues

bf183fc

filipecosta90 mentioned this pull request Sep 5, 2021

Question : Latency Histograms per Command #4707

Closed

Fixed CI build issues

d992764

oranagra reviewed Sep 6, 2021

View reviewed changes

madolson reviewed Sep 7, 2021

View reviewed changes

Apply suggestions from code review

a4ed897

Co-authored-by: Madelyn Olson <[email protected]>

filipecosta90 added 4 commits December 20, 2021 19:24

Moved latency tracking to nanos to extend precision of reported metrics

381b537

Merge branch 'commands.latency.histogram' of github.com:filipecosta90…

9d84431

…/redis into commands.latency.histogram

Merge remote-tracking branch 'origin/unstable' into commands.latency.…

5ca379b

…histogram

Fixes per PR review. Added latency-track and latency-track-percentile…

6fe8072

…s configuration parameters

filipecosta90 requested review from madolson and oranagra December 21, 2021 09:51

filipecosta90 added 4 commits December 21, 2021 09:54

Fixes per PR review on INFO LATENCYSTATS work

52c4321

Fixed CONFIG REWRITE issue/leak for latency-track-percentiles option

e2a9223

Fixed call() latency tracking

6123f84

freeing hdrhistogram memory on moduleUnregisterCommands()

2fba864

Added latencystats tests: disable/enable and configure percentiles

82b4788

filipecosta90 added the action:run-benchmark Triggers the benchmark suite for this Pull Request label Dec 27, 2021

filipecosta90 added 2 commits December 27, 2021 20:05

fixed double tracking of blocking commands. Added unit test

7301b12

Merge remote-tracking branch 'origin/unstable' into commands.latency.…

2aca879

…histogram

oranagra reviewed Dec 27, 2021

View reviewed changes

ranshid reviewed Dec 28, 2021

View reviewed changes

src/blocked.c Show resolved Hide resolved

tests/unit/info.tcl Show resolved Hide resolved

Fixes maxmemory tests after the change that overrided the hdr-histogr…

71bec05

…am allocators: iteration 2

oranagra reviewed Jan 5, 2022

View reviewed changes

yoav-steinberg reviewed Jan 5, 2022

View reviewed changes

deps/hdr_histogram/hdr_alloc.h Outdated Show resolved Hide resolved

oranagra mentioned this pull request Jan 5, 2022

redis.cli: Add --print-duration-threshold option #10055

Open

Fixes per PR review: Ensure that the arguments to zcalloc(), when mul…

ec6451c

…tiplied, do not wrap. zmalloc_oom_handler logs the num_items trying to allocate alongside the item size.

filipecosta90 requested review from oranagra and yoav-steinberg January 5, 2022 09:58

oranagra reviewed Jan 5, 2022

View reviewed changes

src/zmalloc.c Outdated Show resolved Hide resolved

yoav-steinberg reviewed Jan 5, 2022

View reviewed changes

src/zmalloc.c Outdated Show resolved Hide resolved

filipecosta90 added 2 commits January 5, 2022 10:52

Fixes per PR review: reduce the changes of zcalloc to the minimum (no…

2753011

…t changing logging).

Reverted spurious change on zmalloc_default_oom log

fb1786e

oranagra approved these changes Jan 5, 2022

View reviewed changes

deps/hdr_histogram/hdr_alloc.h Outdated Show resolved Hide resolved

deps/hdr_histogram/hdr_histogram.h Outdated Show resolved Hide resolved

tests/unit/info.tcl Show resolved Hide resolved

tests/unit/info.tcl Outdated Show resolved Hide resolved

filipecosta90 and others added 2 commits January 5, 2022 11:26

Update deps/hdr_histogram/hdr_alloc.h

55bf0a7

Co-authored-by: Oran Agra <[email protected]>

Fixes per PR review: tagged latencystats: measure latency test as nee…

104270a

…ds:debug. Removed unecessary stddef header in hdrhistogram

filipecosta90 requested a review from yoav-steinberg January 5, 2022 11:28

filipecosta90 and others added 2 commits January 5, 2022 11:30

Fixes per PR review: using wait_for_blocked_client on latencystats: b…

f925f71

…locking commands test

test cleanups

78efe8f

oranagra merged commit 5dd1544 into redis:unstable Jan 5, 2022

filipecosta90 mentioned this pull request Feb 24, 2022

[BUG] ZREVRANGE 50% slower after upgrading from 5.0.7 to 6.2.6 #10310

Closed

filipecosta90 mentioned this pull request Mar 21, 2022

5-7% Performance regression from v5 to v6.2 to unstable due to added features ( more visible on pipeline ) #10460

Open

filipecosta90 mentioned this pull request May 25, 2022

Added latency_percentiles_usec summary (A summary of latency percentile distribution per command) oliver006/redis_exporter#652

Merged

filipecosta90 mentioned this pull request Jul 15, 2022

Performance degrade 7.0.3 vs 6.2.7 #10981

Closed

tezc mentioned this pull request Aug 28, 2022

Investigate latency tracking RedisLabs/redisraft#481

Closed

oranagra mentioned this pull request Apr 4, 2023

Add basic eventloop latency measurement. #11963

Merged

filipecosta90 mentioned this pull request Dec 13, 2023

[QUESTION]Performance degrade of GET command on version 7.x & 6.x compare with 5.x #12748

Open

Conversation

filipecosta90 commented Sep 5, 2021 • edited by oranagra Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Short description

Some details:

overhead test using redis-benchmark on unstable vs commands.latency.histogram

results on unstable ( commit hash = af0b50f )

results on commands.latency.histogram branch with latency track disabled

results on commands.latency.histogram branch with latency track enabled

INFO LATENCYSTATS exposition format

latencystats output sample:

LATENCY HISTOGRAM [command ...] exposition format

Uh oh!

oranagra left a comment

Choose a reason for hiding this comment

Uh oh!

madolson commented Sep 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

madolson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

YacineTALEB commented Oct 7, 2021

Uh oh!

oranagra commented Nov 2, 2021

Uh oh!

filipecosta90 commented Dec 2, 2021

Uh oh!

filipecosta90 commented Dec 21, 2021

Uh oh!

filipecosta90 commented Dec 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

overhead test using redis-benchmark on unstable vs commands.latency.histogram

results on unstable ( commit hash = af0b50f )

results on commands.latency.histogram branch with latency track disabled

results on commands.latency.histogram branch with latency track enabled

Uh oh!

filipecosta90 commented Dec 27, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ranshid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

oranagra commented Jan 5, 2022

Uh oh!

oranagra commented Jan 5, 2022

filipecosta90 commented Sep 5, 2021 •

edited by oranagra

Loading

results on unstable ( commit hash = `af0b50f` )

`INFO LATENCYSTATS` exposition format

`LATENCY HISTOGRAM [command ...]` exposition format

madolson commented Sep 6, 2021 •

edited

Loading

filipecosta90 commented Dec 27, 2021 •

edited

Loading

results on unstable ( commit hash = `af0b50f` )