I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Better read task scheduling for JBOD architecture and MergeTree storage. New setting read_backoff_min_concurrency which serves as the lower limit to the number of reading threads.

Detailed description / Documentation draft:

Disk-aware read scheduling is useful to avoid tail latency issues when dealing with huge data on JBOD array. I've observed a lot of read clustering issues, that is, we concurrently read from one disk for 20 seconds, and then switch to another one for the next 20 seconds.

I've tested it in some production environment with 12 disks JBOD array setup, and the results are very promising.

The baseline takes 573.039 sec, with JBOD task split, it reaches 429.389 sec, with random read task stealing, it gets 185.612 sec.

It works well with current read backoff mechanism.

update

Random stealing incurs reader reinit cost. Now we use a different scheme. First we try if any backoff threads can be resurrected. If no, we steal the next one. Thanks to the pre-balanced workloads, it should have a pretty good uniform distribution in general.

With this steal strategy, the runtime varies from 105 ~ 135 secs.