tsdb: Early compaction of stale series by codesome · Pull Request #16929 · prometheus/prometheus

codesome · 2025-07-25T23:31:50Z

Stale series tracking was added in #16925. This PR compacts the stale series into its own block before the normal compaction hits. Here is how the config works:

stale_series_compaction_threshold: As soon as the ratio of stale series in the head block crosses StaleSeriesImmediateCompactionThreshold, TSDB performs a stale series compaction and puts all the stale series into a block and removed it from the head, but it does not remove it from the WAL. (technically this condition is checked every minute and not exactly immediate)

Additional details

WAL replay: after a stale series compaction, tombstones are added with (MinInt64, MaxInt64) for all these stale series. During WAL replay we add a special condition where when we find such tombstone, it immediately removes the series from the memory instead of storing the tombstone. This is required so that we don't spike up memory during WAL replay and also don't keep the compacted stale series in the memory.
Head block truncation ignores this block via the added metadata, similar to out-of-order blocks.

[ENHANCEMENT] tsdb: Experimental support for early compaction of stale series in the memory with configurable threshold.

codesome · 2025-07-25T23:41:18Z

/prombench main

prombot · 2025-07-25T23:41:21Z

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-16929 and main

After the successful deployment (check status here), the benchmarking results can be viewed at:

Available Commands:

To restart benchmark: /prombench restart main
To stop benchmark: /prombench cancel
To print help: /prombench help

codesome · 2025-07-26T02:52:39Z

/prombench cancel

prombot · 2025-07-26T02:52:41Z

Benchmark cancel is in progress.

codesome · 2025-07-26T02:53:24Z

Looks like stale series tracking is not working. Stale samples are not being put for the series I guess.

codesome · 2025-07-29T01:00:06Z

/prombench main

prombot · 2025-07-29T01:00:10Z

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-16929 and main

After the successful deployment (check status here), the benchmarking results can be viewed at:

Available Commands:

To restart benchmark: /prombench restart main
To stop benchmark: /prombench cancel
To print help: /prombench help

codesome · 2025-07-29T01:27:12Z

/prombench stop

prombot · 2025-07-29T01:27:13Z

Incorrect /prombench syntax; command requires one argument that matches (master|main|v[0-9]+\.[0-9]+\.[0-9]+\S*) regex.

Available Commands:

To start benchmark: /prombench <branch or git tag to compare with>
To restart benchmark: /prombench <branch or git tag to compare with>
To stop benchmark: /prombench cancel
To print help: /prombench help

Advanced Flags for start and restart Commands:

--bench.directory=<sub-directory of github.com/prometheus/test-infra/prombench
- See the details here, defaults to manifests/prombench.
--bench.version=<branch | @commit>
- See the details here, defaults to master.

Examples:

/prombench v3.0.0
/prombench v3.0.0 --bench.version=@aca1803ccf5d795eee4b0848707eab26d05965cc --bench.directory=manifests/prombench

codesome · 2025-07-29T01:37:32Z

/prombench main

prombot · 2025-07-29T01:37:35Z

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-16929 and main

After the successful deployment (check status here), the benchmarking results can be viewed at:

Available Commands:

To restart benchmark: /prombench restart main
To stop benchmark: /prombench cancel
To print help: /prombench help

codesome · 2025-07-29T03:08:39Z

/prombench cancel

prombot · 2025-07-29T03:08:42Z

Benchmark cancel is in progress.

codesome · 2025-07-29T03:14:20Z

Here are the results from the PoC and the results are in-line with the expectations.

The good: memory and number of timeseries stayed down. For the prombench data pattern, the memory used was consistently 30-60% down.

The bad: The CPU usage is higher and instant queries took a hit. This is mostly because now it requires merging stale block and head block results for instant queries. 20-30% higher for this benchmark and peaks up to 50% sometimes.

I will fix some of the edge cases in the code and run it in our internal cluster to get some real world numbers.

codesome · 2025-07-29T19:03:17Z

/prombench cancel

prombot · 2025-07-29T19:03:20Z

Benchmark cancel is in progress.

codesome · 2025-07-29T19:07:59Z

Prombench decided to keep running even after stopping. So we have more numbers to look at.

Memory savings is great, but query/cpu is bonkers. I am guessing the queries in prombench are trying to touch a whole lot of series continuously.

codesome · 2025-07-29T19:18:43Z

Took a quick look at the profiles and confirmed that instant queries is taking the extra CPU. In the below pic, the red box is an additional CPU that stale series compaction introduces, since now it has to look at the block on disk for all instant queries. There is no way around it.

Here are the profiles that I downloaded from prombench:
main branch profile.pb.gz
stale series profile.pb.gz

SuperQ · 2025-08-05T10:05:17Z

The memory results look really good. For sure something we will want behind a feature flag for now. If we can improve on the CPU overhead, this may be something to enable by default in the future.

bwplotka · 2025-08-06T09:01:14Z

I'm surprised instant queries (or any queries) against TSDB blocks are so CPU intensive on Prometheus. OR... prombench results are not realistic -- like it spams queries way to often then realistically users would do. One important case is ofc alerting/recording rules - if they hit TSDB block, that block should be partially cached then (see below)

It would be useful to understand a single common query CPU overhead for in-mem and TSDB block... also we could cache index a bit at least for stale near-real time blocks to mitigate some CPU with a bit more memory (hopefully this will diminishes the memory results - this cache should only be short-living for similar queries or heavy instant query load 🤔). Maybe we learn about some need for optimizing the TSDB read path with this work (:

I still think this would be an interesting mode e.g. for us (Google), where we keep local query capability for debugging in some cases but we use cloud as a first order.

Thanks for extensive research!

bboreham · 2025-08-11T13:24:02Z

prombench results are not realistic -- like it spams queries way to often then realistically users would do

It queries frequently, which might be taken to simulate a large user population or a lot of recording rules, but perhaps more importantly it never queries more than 1 hour back. That is what PRs like prometheus/test-infra#782 are seeking to change.

So if you make every query hit every block, that will make quite a difference.

bboreham · 2025-09-30T11:37:41Z

Hello from the bug-scrub!

@jesusvazquez I see you were assigned - do you think you will get a chance to look at it?

config/config.go

tsdb/head_read.go

jesusvazquez · 2025-12-29T17:27:06Z

I'll have a look at this next week, starting my PTO today for a few days 🙏

codesome · 2026-01-09T02:14:28Z

@jesusvazquez I have synced this PR with main branch and fixed the lint and is ready for review

jesusvazquez

Left a few comments, overall in good shape.

tsdb/db.go

tsdb/head_read.go

tsdb/db.go

Signed-off-by: Ganesh Vernekar <[email protected]>

jesusvazquez

LGTM!

SuperQ

Nice. Based on our production testing, I the we found that ~50% was a good threshold.

Should we document any recommendations or wait for more user feedback?

codesome · 2026-01-24T22:45:16Z

Should we document any recommendations or wait for more user feedback?

We should wait for some user feedback. IMO it's more to do with the pattern in which stale series ratio goes up and down and the memory headroom, and less about the actual value of the ratio. As part of my upcoming talk, I plan to do some more testing for config options.

##### [\`v3.10.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.10.0) Prometheus now offers a distroless Docker image variant alongside the default busybox image. The distroless variant provides enhanced security with a minimal base image, uses UID/GID 65532 (nonroot) instead of nobody, and removes the VOLUME declaration. Both variants are available with `-busybox` and `-distroless` tag suffixes (e.g., `prom/prometheus:latest-busybox`, `prom/prometheus:latest-distroless`). The busybox image remains the default with no suffix for backwards compatibility (e.g., `prom/prometheus:latest` points to the busybox variant). For users migrating existing **named** volumes from the busybox image to the distroless variant, the ownership can be adjusted with: ``` docker run --rm -v prometheus-data:/prometheus alpine chown -R 65532:65532 /prometheus ``` Then, the container can be started with the old volume with: ``` docker run -v prometheus-data:/prometheus prom/prometheus:latest-distroless ``` User migrating from bind mounts might need to ajust permissions too, depending on their setup. - \[CHANGE] Alerting: Add `alertmanager` dimension to following metrics: `prometheus_notifications_dropped_total`, `prometheus_notifications_queue_capacity`, `prometheus_notifications_queue_length`. [#16355](prometheus/prometheus#16355) - \[CHANGE] UI: Hide expanded alert annotations by default, enabling more information density on the `/alerts` page. [#17611](prometheus/prometheus#17611) - \[FEATURE] AWS SD: Add MSK Role. [#17600](prometheus/prometheus#17600) - \[FEATURE] PromQL: Add `fill()` / `fill_left()` / `fill_right()` binop modifiers for specifying default values for missing series. [#17644](prometheus/prometheus#17644) - \[FEATURE] Web: Add OpenAPI 3.2 specification for the HTTP API at `/api/v1/openapi.yaml`. [#17825](prometheus/prometheus#17825) - \[FEATURE] Dockerfile: Add distroless image variant using UID/GID 65532 and no VOLUME declaration. Busybox image remains default. [#17876](prometheus/prometheus#17876) - \[FEATURE] Web: Add on-demand wall time profiling under `<URL>/debug/pprof/fgprof`. [#18027](prometheus/prometheus#18027) - \[ENHANCEMENT] PromQL: Add more detail to histogram quantile monotonicity info annotations. [#15578](prometheus/prometheus#15578) - \[ENHANCEMENT] Alerting: Independent alertmanager sendloops. [#16355](prometheus/prometheus#16355) - \[ENHANCEMENT] TSDB: Experimental support for early compaction of stale series in the memory with configurable threshold `stale_series_compaction_threshold` in the config file. [#16929](prometheus/prometheus#16929) - \[ENHANCEMENT] Service Discovery: Service discoveries are now removable from the Prometheus binary through the Go build tag `remove_all_sd` and individual service discoveries can be re-added with the build tags `enable_<sd name>_sd`. Users can build a custom Prometheus with only the necessary SDs for a smaller binary size. [#17736](prometheus/prometheus#17736) - \[ENHANCEMENT] Promtool: Support promql syntax features `promql-duration-expr` and `promql-extended-range-selectors`. [#17926](prometheus/prometheus#17926) - \[PERF] PromQL: Avoid unnecessary label extraction in PromQL functions. [#17676](prometheus/prometheus#17676) - \[PERF] PromQL: Improve performance of regex matchers like `.*-.*-.*`. [#17707](prometheus/prometheus#17707) - \[PERF] OTLP: Add label caching for OTLP-to-Prometheus conversion to reduce allocations and improve latency. [#17860](prometheus/prometheus#17860) - \[PERF] API: Compute `/api/v1/targets/relabel_steps` in a single pass instead of re-running relabeling for each prefix. [#17969](prometheus/prometheus#17969) - \[PERF] tsdb: Optimize LabelValues intersection performance for matchers. [#18069](prometheus/prometheus#18069) - \[BUGFIX] PromQL: Prevent query strings containing only UTF-8 continuation bytes from crashing Prometheus. [#17735](prometheus/prometheus#17735) - \[BUGFIX] Web: Fix missing `X-Prometheus-Stopping` header for `/-/ready` endpoint in `NotReady` state. [#17795](prometheus/prometheus#17795) - \[BUGFIX] PromQL: Fix PromQL `info()` function returning empty results when filtering by a label that exists on both the input metric and `target_info`. [#17817](prometheus/prometheus#17817) - \[BUGFIX] TSDB: Fix a bug during exemplar buffer grow/shrink that could cause exemplars to be incorrectly discarded. [#17863](prometheus/prometheus#17863) - \[BUGFIX] UI: Fix broken graph display after page reload, due to broken Y axis min encoding/decoding. [#17869](prometheus/prometheus#17869) - \[BUGFIX] TSDB: Fix memory leaks in buffer pools by clearing reference fields (Labels, Histogram pointers, metadata strings) before returning buffers to pools. [#17879](prometheus/prometheus#17879) - \[BUGFIX] PromQL: info function: fix series without identifying labels not being returned. [#17898](prometheus/prometheus#17898) - \[BUGFIX] OTLP: Filter `__name__` from OTLP attributes to prevent duplicate labels. [#17917](prometheus/prometheus#17917) - \[BUGFIX] TSDB: Fix division by zero when computing stale series ratio with empty head. [#17952](prometheus/prometheus#17952) - \[BUGFIX] OTLP: Fix potential silent data loss for sum metrics. [#17954](prometheus/prometheus#17954) - \[BUGFIX] PromQL: Fix smoothed interpolation across counter resets. [#17988](prometheus/prometheus#17988) - \[BUGFIX] PromQL: Fix panic with `@` modifier on empty ranges. [#18020](prometheus/prometheus#18020) - \[BUGFIX] PromQL: Fix `avg_over_time` for a single native histogram. [#18058](prometheus/prometheus#18058)

codesome force-pushed the codesome/stale-series-compaction branch 2 times, most recently from 7f92b48 to f6d7ac4 Compare July 25, 2025 23:41

prombot added the prombench label Jul 25, 2025

codesome force-pushed the codesome/stale-series-compaction branch 2 times, most recently from 4486bdb to e693e22 Compare July 29, 2025 00:50

codesome mentioned this pull request Jul 29, 2025

Proposal: Early Compaction of Stale Series from the Head Block prometheus/proposals#55

Merged

codesome force-pushed the codesome/stale-series-compaction branch 3 times, most recently from 2cc719a to 56f761d Compare August 7, 2025 03:04

codesome force-pushed the codesome/stale-series-compaction branch 3 times, most recently from 2bc6610 to 5b1e6fe Compare August 26, 2025 01:40

yeya24 reviewed Oct 15, 2025

View reviewed changes

config/config.go Outdated Show resolved Hide resolved

tsdb/head_read.go Outdated Show resolved Hide resolved

codesome force-pushed the codesome/stale-series-compaction branch from 393bd08 to b209783 Compare November 26, 2025 02:56

codesome requested review from bwplotka and krajorama as code owners January 9, 2026 01:10

codesome force-pushed the codesome/stale-series-compaction branch from 5265f19 to 3f51be0 Compare January 9, 2026 01:20

jesusvazquez reviewed Jan 23, 2026

View reviewed changes

tsdb/db.go Outdated Show resolved Hide resolved

tsdb/db.go Outdated Show resolved Hide resolved

tsdb/head_read.go Show resolved Hide resolved

SuperQ reviewed Jan 23, 2026

View reviewed changes

tsdb/db.go Outdated Show resolved Hide resolved

tsdb/db.go Outdated Show resolved Hide resolved

tsdb/db.go Outdated Show resolved Hide resolved

codesome force-pushed the codesome/stale-series-compaction branch from 519a2d2 to 72590c4 Compare January 24, 2026 01:16

codesome added 5 commits January 23, 2026 17:59

tsdb: Add StaleHead and GC for stale series in the Head block

9b444b5

Signed-off-by: Ganesh Vernekar <[email protected]>

tsdb: Add stale series compaction support in the DB

43e6938

Signed-off-by: Ganesh Vernekar <[email protected]>

tsdb: Clear stale series from the Head during WAL replay

43dc23a

Signed-off-by: Ganesh Vernekar <[email protected]>

tsdb: Add unit tests for stale series compaction

4f3de8d

Signed-off-by: Ganesh Vernekar <[email protected]>

Add stale_series_compaction_threshold config file option

3e4a094

Signed-off-by: Ganesh Vernekar <[email protected]>

codesome force-pushed the codesome/stale-series-compaction branch from 72590c4 to 3e4a094 Compare January 24, 2026 02:18

jesusvazquez approved these changes Jan 24, 2026

View reviewed changes

SuperQ approved these changes Jan 24, 2026

View reviewed changes

codesome merged commit 9eb7873 into main Jan 24, 2026
87 of 90 checks passed

codesome deleted the codesome/stale-series-compaction branch January 24, 2026 23:18

This was referenced Jan 25, 2026

Add test case for loading stale_series_compaction_threshold config #17927

Merged

docs: Document the stale_series_compaction_threshold config file option #17928

Merged

github-actions bot mentioned this pull request Mar 19, 2026

Update prom/prometheus Docker tag to v3.10.0 claytono/infra#1704

Merged

1 task

Conversation

codesome commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codesome commented Jul 25, 2025

Uh oh!

prombot commented Jul 25, 2025

Uh oh!

codesome commented Jul 26, 2025

Uh oh!

prombot commented Jul 26, 2025

Uh oh!

codesome commented Jul 26, 2025

Uh oh!

codesome commented Jul 29, 2025

Uh oh!

prombot commented Jul 29, 2025

Uh oh!

codesome commented Jul 29, 2025

Uh oh!

prombot commented Jul 29, 2025

Uh oh!

codesome commented Jul 29, 2025

Uh oh!

prombot commented Jul 29, 2025

Uh oh!

codesome commented Jul 29, 2025

Uh oh!

prombot commented Jul 29, 2025

Uh oh!

codesome commented Jul 29, 2025

Uh oh!

codesome commented Jul 29, 2025

Uh oh!

prombot commented Jul 29, 2025

Uh oh!

codesome commented Jul 29, 2025

Uh oh!

codesome commented Jul 29, 2025

Uh oh!

SuperQ commented Aug 5, 2025

Uh oh!

bwplotka commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bboreham commented Aug 11, 2025

Uh oh!

bboreham commented Sep 30, 2025

Uh oh!

Uh oh!

Uh oh!

jesusvazquez commented Dec 29, 2025

Uh oh!

codesome commented Jan 9, 2026

Uh oh!

jesusvazquez left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jesusvazquez left a comment

Choose a reason for hiding this comment

Uh oh!

SuperQ left a comment

Choose a reason for hiding this comment

Uh oh!

codesome commented Jan 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

codesome commented Jul 25, 2025 •

edited

Loading

bwplotka commented Aug 6, 2025 •

edited

Loading