These charts show how the local ratekeeper protects us from the heavily skewed read write workload.
I used a highly skewed workload which generates heavy range reads and writes targeting a particular single team. Without the local ratekeeper, when a storage server gets too many read requests, it cannot process pending write tasks, so the NDV won't decrease. With the local ratekeeper, when NDV goes up to certain threshold, it starts throttling the reads (Shown in the most left 2 charts). While the reads are throttled, the storage server can process the write tasks, so the NDV can go down.

In comparison, here are the charts with the current master with the same workload.

Note that NDV does not come down because of the heavy incoming reads.

These charts show how the local ratekeeper protects us from the heavily skewed read write workload.
I used a highly skewed workload which generates heavy range reads and writes targeting a particular single team. Without the local ratekeeper, when a storage server gets too many read requests, it cannot process pending write tasks, so the NDV won't decrease. With the local ratekeeper, when NDV goes up to certain threshold, it starts throttling the reads (Shown in the most left 2 charts). While the reads are throttled, the storage server can process the write tasks, so the NDV can go down.

In comparison, here are the charts with the current master with the same workload.

Note that NDV does not come down because of the heavy incoming reads.

This is an insightful evaluation.

While I'm trying to understand the figures in the evaluations, I'm confused at what each line means in the figures ReadOPS, WriteOPS and NDV. Maybe you posted the legend for those figures before but I didn't know where it is?

ReadOPS = Read operations / second
WriteOPS = Write operations / second
NDV = Non-Durable Version (= Current Version - Durable Version)
These are time-series chart. X-axis is time in seconds.

ReadOPS = Read operations / second
WriteOPS = Write operations / second
NDV = Non-Durable Version (= Current Version - Durable Version)
These are time-series chart. X-axis is time in seconds.

I think I didn't explain myself clearly. I understood those abbreviations.

What I asked is what do the lines in different colors mean in each of those figures.
I guess each line in the figures is the data for one client?

Ah, got it. Each line represents each storage process. In the NDV chart, there are 3 lines moving because one team (triple redundancy) is being hot.

In addition to those per-server-process charts, the "Local Ratekeeper" chart shows the aggregate of the "probability of serving reads" of all storage processes. The "Ratekeeper" chart and "Worst Storage Queue" are taken from the ratekeeper metrics.

My impression for local ratekeeper was that ratekeeper would only allow two (with triple replication) storage servers that they were allowed to start limiting their reads.

This implementation lets all storage servers limit themselves, and additionally will cause the main ratekeeper to throttle if they fall too far behind. With this implementation I am concerned that a saturating workload could cause all storage servers to each decide to throttle reads. This would lead to higher read latencies, which could cause transactions to start taking longer than 5 seconds, and lead to a death spiral.