Motivation

We need to monitor CPU usage of the current qdrant process,
to determine if we want to decrease / increase search thread pool at the current moment.

Thread pool used to have size equal to number of CPUs, but it doesn't work well if we have high IO.
In this case we want more threads (up to 4x of CPU count). But this affects the full in-RAM usa case (because of CPU contention).
So we want to dynamically adjust thread pool size based on the current CPU usage of the process.

Implementation

Step 1: Get CPU usage of the current process

We need a function, which would return CPU usage of qdrant process in last N (configured constant) seconds.
This function should work on linux and other platforms are optional.

Request to this function should be cheap, so we need to create a "TTL cache" functionality,
so it would actually read CPU usage from OS only once per N seconds.

Function semantic:

If 2 CPU cores were used 100% in last N seconds, we should return value 2.0

It is prefered to use procfs to read process CPU usage.

Function should be available globally, and implemented in common crate.

Step 2: telemetry

Additionally, we want to have CPU usage value in telemetry, so it would be easy to monitor and debug it.
If function is not supported on platform, it should return None.

Step 3: Auto-adjust thread pool

(old version, for historical reasons)

Step 3: thread pool selection

After initial experiments, it turns out that having semaphore in spawn-blocking creates overhead:

• it requires async
• with it RPS drops from 4k into 3k under cpu-heavy workflow

We approach gets read of it by introducing 2 runtimes instead of 1 - high_cpu and high_io.

Choise between runtimes is same - based on cpu usage thresholds. But check is much cheaper - just single atomic.
With this approach we have practically no overhead for thread selection.
The downside is that we loose granularity in threads count, but that seems fine.

Note, that having more runtimes doesn't actually introduce more threads, as tokio blocking threads are started on-demand. There might be only over-use of threads during switch from one runtime into another, but that doesn't seem like a big problem

Benchmarks

BQ + rescoring from disk

3-node cluster with 20M 1024D vectors, which doesn't fit RAM.
Binary quantization enabled with rescroing.

# Async scorer
# Avg rps: 9.783414976222488
# IOPS: 7.5K / 10K
# CPU: 0.53 / 2

# Async scorer + dynamic thread pool
# Avg rps: 13.985548929909106
# IOPS: 10K / 10K
# CPU: 0.87 / 2

The result: dynamic threadpool can utlize disk resources better

Full in-ram

1 million 128d, 2 cpu limit

Avg rps: 

1.17.1: 4000
dev: 3000
this: 4000