[EXPERIMENT] Prombench metadata-wal-records feature#15907
[EXPERIMENT] Prombench metadata-wal-records feature#15907
metadata-wal-records feature#15907Conversation
Signed-off-by: Bartlomiej Plotka <[email protected]>
|
/prombench main --bench.version=bench/on-and-off-same-prom-version/metadata-wal-records |
|
⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️ Compared versions: Custom benchmark version: After the successful deployment (check status here), the benchmarking results can be viewed at: Available Commands:
|
|
/prombench restart main --bench.version=bench/on-and-off-same-prom-version/metadata-wal-records |
|
⏱️ Welcome (again) to Prometheus Benchmarking Tool. ⏱️ Compared versions: Custom benchmark version: After successful deployment (check status here), the benchmarking results can be viewed at: Available Commands:
|
|
/prombench restart main --bench.version=bench/cross-feature/metadata-wal-records |
|
⏱️ Welcome (again) to Prometheus Benchmarking Tool. ⏱️ Compared versions: Custom benchmark version: After successful deployment (check status here), the benchmarking results can be viewed at: Available Commands:
|
bwplotka
changed the title
metadata-wal-records featuremetadata-wal-records feature
|
/prombench main --bench.version=bench/cross-feature/metadata-wal-records |
|
⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️ Compared versions: Custom benchmark version: After the successful deployment (check status here), the benchmarking results can be viewed at: Available Commands:
|
|
/prombench main --bench.version=bench/cross-feature/metadata-wal-records |
|
⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️ Compared versions: Custom benchmark version: After the successful deployment (check status here), the benchmarking results can be viewed at: Available Commands:
|
|
Deleting manually both Prometheus pods to test restarts too. Will add separate proposal for automation for that on prombench. |
|
So there is an increase in memory (below 3 GB more RSS, sometimes 4GB more of heap allocs) when you enable metadata-wal-records. This is somewhat expected as we now track metadata per series, but maybe not to that degree. Is it acceptable? Who knows, the lower the better. One alternative is the @dashpole proposal which could replace this feature (not for help though): prometheus/proposals#39
Heap profile: heap.mem.pprof.gz |
|
I see some metadata work increase for loadWAL (I assume after restart operation). But nothing in |
|
There are some massive mem spikes
Looking on profile again, it's actually no only WAL load but also compaction (checkpoint logic)
|
|
This does not explain 14% of constant RSS increase and 30% heap increase, though. |
|
Create issue for it #15911 closing this experiment for now. |
|
/prombench cancel |
|
Benchmark cancel is in progress. |
2 participants
Just for prombench.