/prombench main

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-17101 and main

After the successful deployment (check status here), the benchmarking results can be viewed at:

• Prometheus Meta
• Prombench Dashboard
• Grafana Explorer, Loki logs
• Parca profiles (e.g. in-use memory)

Available Commands:

• To restart benchmark: /prombench restart main
• To stop benchmark: /prombench cancel
• To print help: /prombench help

Looks as this uses more memory (on avg) over time:

..and more CPU 🙈

BTW @bboreham I keep changing our prombench dashboard manually for longer averages to really tell the difference, because mem & CPU spikes are hard to compare (they happen on different times for each version). Should we add avg set of panels? (the spikes are still useful.. although max_over_time could tell us the same 🤔 ) WDYT?

If you plot go_sync_mutex_wait_total_seconds_total there's more wait after this change:

Looks as this uses more memory (on avg) over time:

RSS is pretty much meaningless; the Go heap is where I would expect to see any difference. But the expected difference is 3MB which is not visible on a 24GB scale.

Thanks @prymitive, the increased mutex time is interesting, seems to hit particularly during Head GC - the 5-second scrape interval lets me pin down the timing. Explore

2025-08-29T09:01:22.052Z level=INFO source=compact.go:598 msg="write block" component=tsdb mint=1756447200014 maxt=1756454400000 ulid=01K3TGW7MPSZ9QC4FF5NYRGRXS duration=1m22.029964316s ooo=false
2025-08-29T09:01:42.321Z level=INFO source=head.go:1422 msg="Head GC completed" component=tsdb caller=truncateMemory duration=20.267960764s
2025-08-29T09:01:42.394Z level=INFO source=checkpoint.go:100 msg="Creating checkpoint" component=tsdb from_segment=100 to_segment=114 mint=1756454400000
2025-08-29T09:02:05.639Z level=INFO source=head.go:1384 msg="WAL checkpoint complete" component=tsdb first=100 last=114 duration=23.244824427s

Logs Explore

I guess this proves that 256 is too low, and/or we should look at how Head GC takes locks.

@prymitive I also notice your picture shows PR #17089. There is a bug in the way URLs are generated so it defaults to the lowest one active, sorry.

/prombench cancel

Benchmark cancel is in progress.

/prombench main

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-17101 and main

After the successful deployment (check status here), the benchmarking results can be viewed at:

• Prometheus Meta
• Prombench Dashboard
• Grafana Explorer, Loki logs
• Parca profiles (e.g. in-use memory)

Available Commands:

• To restart benchmark: /prombench restart main
• To stop benchmark: /prombench cancel
• To print help: /prombench help

/prombench cancel

Benchmark cancel is in progress.

I guess this isn't a slam-dunk.

I'm running a bench for main against main #17110 (comment) prombench to see what go_sync_mutex_wait_total_seconds_total variance is acceptable.

(the next step after #15339 is to have it under a Go metrics panel in the dashboard, I'll try to add that. the metric was at least relevant once in the past #15242 (comment))

go_sync_mutex_wait_total_seconds_total main vs main:

Note that a variance of ~90% could be "acceptable", they tend to catch up eventually but we shouldn’t expect them to be and stay close all the time.