/prombench main

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-17959 and main

After the successful deployment (check status here), the benchmarking results can be viewed at:

• Prometheus Meta
• Prombench Dashboard
• Grafana Explorer, Loki logs
• Parca profiles (e.g. in-use memory)

Available Commands:

• To restart benchmark: /prombench restart main
• To stop benchmark: /prombench cancel
• To print help: /prombench help

Summary of the dual prombench run on the current state of things.

• S) scrape switch (this PR)
• O) scrape switch + 3 micro optimizations ([BENCHMARK] scrape: optimizations + switch scrape manager to AppenderV2 #17960)

Health

Same head chunks, series, similar head appenders grow, etc

Similar traffic (although load on PR S is kind of smaller for /query)

I see weird lags on query engine for (S):

Same for (O):

Latency is clearly impacted for (S)

On (O) not so much, but something on query_range

Resources

Generally this V2 still allocates and uses more CPU, especially weirdly heap over time.

Last 24h:

S) +5-6% alloc/s more, +5-10% of total memory

O) +1-3% alloc/s more, +6-10% of total memory

CPU also higher:

S) +5%

O) +7%

Profiles are clear, there's tons of more memory in use

(S) Mem in use diff: https://pprof.me/cdb32dbd9df06c4bef820b357ab115d8/ (10)
(O) Mem in use diff: https://pprof.me/d2b0a5ff472da2f8a1b8a17a67815138/ (13)

(S) mem https://pprof.me/bb8b6f165aa5d057ab28b973050fa4b4/ (11)
(O) mem https://pprof.me/3e64450b19e17c11d960d0578df53005/ (12)

(O) cumulative allocs diff https://pprof.me/f66f50fa52ffde05693190a727fb2695/ (15)
(O) cumulative allocs https://pprof.me/f58d258ff5c1ce857d4f908c1a0bbe46/ (16)

CPU diff is harder to stop but there's something:

(O) CPU diff https://pprof.me/2cc5907d056b242783d5a7fdeb9c076f/ (14)

Maybe I look on it wrong, but I think the culprint is here:

Somehow AppenderV2 allocates more on those:

How would that be possible?

Diff values:

Closed (O) prombench and PR (#17960). Leaving this prombench for a day more just in case anyone wants to investigate.

Cumulative diff shows appending float clearly

Are we not cleaning sync.Pool buffers somewhow for V2?

Diff profiles confuse me.

(O reference) mem cumulative (17) https://pprof.me/fe800d53fcf62825c9b6cbef6706a062/
(O) Mem cumulative (18) https://pprof.me/c05e517ad47ac2f2a9c0abe942f1cb20/

(O reference) mem inuse (19) https://pprof.me/7c8afeca6f175d13c48a36773e5eea55/
(O) Mem inuse (20) https://pprof.me/79e8f9ee26b13bec60864d5b97c7345b/

Cumulative, those two are fun:

• ref append used 117 GB over time
• O used 237 GB

Is this... fanout?

vs ref

Yup that's it, got it:

/prombench cancel

Benchmark cancel is in progress.