tsdb: skip entire stripes in mmapHeadChunks via per-stripe ready count by aknuds1 · Pull Request #18541 · prometheus/prometheus

aknuds1 · 2026-04-18T12:24:59Z

Which issue(s) does the PR fix:

Summary

Follow-up to #18272, optimizing also the outer loop (skipping entire TSDB stripes without series in need of mmapping).

Commit 1 — tsdb: extract stripeSeries.refStripe helper

Extract the repeated ref-to-stripe-index calculation into a refStripe method on stripeSeries, replacing four inline copies that used two different casting styles (int and uint64).

Commit 2 — tsdb: skip entire stripes in mmapHeadChunks via per-stripe ready count

Add a per-stripe mmapReady counter to stripeSeries that tracks how many series in each stripe have headChunkCount >= 2 (i.e., are ready for mmapping). mmapHeadChunks skips stripes where the counter is zero, avoiding the RLock and map iteration entirely.

The counter is maintained at all sites that modify headChunkCount:

Incremented in commit paths when a chunk cut brings count to 2
Decremented when mmapChunks brings count back to 1
Decremented in gc/truncateChunksBefore and series deletion
Maintained during WAL replay and chunk snapshot loading

Guard the decrement in mmapHeadChunks on mmapChunks returning > 0 to prevent a double-decrement race: between the lockless headChunkCount filter and acquiring the series lock, GC/deletion can concurrently reduce headChunkCount and decrement mmapReady, so an unconditional decrement would drive the counter negative.

Pad mmapReady counters to 64-byte cache lines (paddedAtomicInt32) to avoid false sharing under concurrent appends across stripes, following the existing stripeLock padding convention.

Testing

I've tested the optimization on a Grafana Mimir instance with millions of series, and AFAICT ingester CPU utilization was reduced by about 15%, because time is no longer wasted locking each TSDB stripe unless a stripe actually has mmappable series.

Benchmark results (Intel Xeon Platinum 8280, 16 CPUs)

goos: linux
goarch: amd64
pkg: github.com/prometheus/prometheus/tsdb
cpu: Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz
                                             │   main.txt   │            optimized.txt            │
                                             │    sec/op    │    sec/op     vs base               │
MmapHeadChunks/series=1000/ready=10-16         399.36µ ± 3%   44.62µ ± 46%  -88.83% (p=0.002 n=6)
MmapHeadChunks/series=1000/ready=100-16         477.8µ ± 1%   121.5µ ±  5%  -74.57% (p=0.002 n=6)
MmapHeadChunks/series=1000/ready=1000-16       1103.9µ ± 1%   780.5µ ±  4%  -29.29% (p=0.002 n=6)
MmapHeadChunks/series=10000/ready=100-16        895.9µ ± 4%   122.3µ ±  6%  -86.35% (p=0.002 n=6)
MmapHeadChunks/series=10000/ready=1000-16      1716.9µ ± 4%   930.8µ ±  2%  -45.79% (p=0.002 n=6)
MmapHeadChunks/series=10000/ready=10000-16      10.18m ± 3%   10.25m ±  3%        ~ (p=0.394 n=6)
MmapHeadChunks/series=100000/ready=1000-16      5.819m ± 4%   1.081m ±  4%  -81.43% (p=0.002 n=6)
MmapHeadChunks/series=100000/ready=10000-16     16.90m ± 3%   13.83m ±  6%  -18.14% (p=0.002 n=6)
MmapHeadChunks/series=100000/ready=100000-16    111.0m ± 3%   109.0m ±  3%        ~ (p=0.394 n=6)
geomean                                         3.208m        1.261m        -60.71%

                                             │   main.txt   │            optimized.txt            │
                                             │     B/op     │     B/op      vs base               │
MmapHeadChunks/series=1000/ready=10-16         2.723Ki ± 4%   1.705Ki ± 2%  -37.39% (p=0.002 n=6)
MmapHeadChunks/series=1000/ready=100-16        16.58Ki ± 4%   15.48Ki ± 3%   -6.62% (p=0.002 n=6)
MmapHeadChunks/series=1000/ready=1000-16       145.3Ki ± 1%   137.9Ki ± 5%   -5.11% (p=0.002 n=6)
MmapHeadChunks/series=10000/ready=100-16       16.91Ki ± 4%   15.33Ki ± 2%   -9.31% (p=0.002 n=6)
MmapHeadChunks/series=10000/ready=1000-16      146.2Ki ± 2%   140.0Ki ± 2%   -4.26% (p=0.002 n=6)
MmapHeadChunks/series=10000/ready=10000-16     1.327Mi ± 1%   1.345Mi ± 3%        ~ (p=0.171 n=6)
MmapHeadChunks/series=100000/ready=1000-16     158.4Ki ± 3%   144.2Ki ± 1%   -8.96% (p=0.002 n=6)
MmapHeadChunks/series=100000/ready=10000-16    1.463Mi ± 8%   1.389Mi ± 5%        ~ (p=0.132 n=6)
MmapHeadChunks/series=100000/ready=100000-16   13.24Mi ± 3%   13.44Mi ± 2%        ~ (p=0.803 n=6)
geomean                                        160.6Ki        146.1Ki        -8.99%

                                             │  main.txt   │            optimized.txt             │
                                             │  allocs/op  │  allocs/op    vs base                │
MmapHeadChunks/series=1000/ready=10-16          11.00 ± 9%    10.00 ± 10%       ~ (p=0.242 n=6)
MmapHeadChunks/series=1000/ready=100-16         105.5 ± 0%    106.0 ±  1%       ~ (p=0.545 n=6)
MmapHeadChunks/series=1000/ready=1000-16       1.037k ± 0%   1.033k ±  0%  -0.39% (p=0.002 n=6)
MmapHeadChunks/series=10000/ready=100-16        107.0 ± 0%    107.0 ±  0%       ~ (p=1.000 n=6) ¹
MmapHeadChunks/series=10000/ready=1000-16      1.043k ± 0%   1.035k ±  0%  -0.77% (p=0.002 n=6)
MmapHeadChunks/series=10000/ready=10000-16     10.98k ± 1%   11.05k ±  1%       ~ (p=0.545 n=6)
MmapHeadChunks/series=100000/ready=1000-16     1.077k ± 0%   1.036k ±  0%  -3.81% (p=0.002 n=6)
MmapHeadChunks/series=100000/ready=10000-16    11.35k ± 1%   11.14k ±  2%  -1.85% (p=0.017 n=6)
MmapHeadChunks/series=100000/ready=100000-16   142.9k ± 6%   147.1k ±  3%       ~ (p=0.628 n=6)
geomean                                        1.111k        1.096k        -1.38%
¹ all samples are equal

Release notes for end users (ALL commits must be considered).

Reviewers should verify clarity and quality.

[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales.

bboreham

This code is already very complicated, and this PR is adding more.
I'd like to see if modest structural changes can help.

Also I'm unhappy about the name confusion between "stripe" and "shard".

bboreham

I'll approve this as it's very close.

Extract the repeated ref-to-stripe-index calculation into a method on stripeSeries, replacing five inline copies that used two different casting styles (int and uint64). The helper computes with uint64 internally so it is correct on 32-bit architectures. Signed-off-by: Arve Knudsen <[email protected]>

Add a per-stripe mmapReady counter to stripeSeries that tracks how many series in each stripe have headChunkCount >= 2 (i.e., are ready for mmapping). mmapHeadChunks skips stripes where the counter is zero, avoiding the RLock and map iteration entirely. The counter is maintained at all sites that modify headChunkCount: - Incremented in commit paths when a chunk cut brings count to 2 - Decremented when mmapChunks brings count back to 1 - Decremented in gc/truncateChunksBefore and series deletion - Maintained during WAL replay and chunk snapshot loading Guard the decrement in mmapHeadChunks on mmapChunks returning > 0 to prevent a double-decrement race: between the lockless headChunkCount filter and acquiring the series lock, GC/deletion can concurrently reduce headChunkCount and decrement mmapReady, so an unconditional decrement would drive the counter negative. Pad mmapReady counters to 64-byte cache lines (paddedAtomicInt32) to avoid false sharing under concurrent appends across stripes, following the existing stripeLock padding convention. At production-relevant scales (100k series, 1% ready), this reduces mmapHeadChunks latency by ~75%. Zero overhead when all series are ready. Signed-off-by: Arve Knudsen <[email protected]>

- Add incMmapReady/decMmapReady methods on stripeSeries to simplify all call sites from verbose index-and-add expressions to one-liners. - Rename local variable refShard to stripe for consistency with the codebase's stripe terminology. Signed-off-by: Arve Knudsen <[email protected]>

The chunk-creation block (metrics increment + mmapReady increment if headChunkCount reaches 2) repeated verbatim in commitFloats, commitHistograms, and commitFloatHistograms. Extract it into a Head method. Also update a stale comment in loadMmappedChunks: it described removing a single head chunk, but the code clears the entire head chunk chain (which may contain multiple chunks). Signed-off-by: Arve Knudsen <[email protected]>

The previous onChunkCreated helper conflated two concerns: bumping the head chunk metrics (which applies to both in-order and OOO chunks) and incrementing mmapReady (which should apply only to in-order chunks, since headChunkCount tracks only in-order head chunks). For a series at headChunkCount == 2 that received an OOO sample, the spurious extra incMmapReady drifted the per-stripe counter positive. Capture headChunkCount before the commit switch and pass it to onChunkCreated as prevHeadChunkCount. The readiness increment now fires only on the explicit 1->2 transition (prev < 2 && current == 2), which is false for OOO chunk creations because they do not modify headChunkCount. Add a regression test that forces the exact state required to reproduce the bug (headChunkCount == 2 with mmapReady already at 1) and verifies an OOO sample does not double-increment, for floats, histograms, and float histograms. Signed-off-by: Arve Knudsen <[email protected]>

When EnableMemorySnapshotOnShutdown is enabled, loadChunkSnapshot can restore a series with headChunkCount >= 2, incrementing the per-stripe mmapReady counter. If WAL replay then processes a sample for that series that triggers a chunk cut, processWALSamples called ms.append (count += 1) and ms.mmapChunks (count -> 1) but never decremented mmapReady. The shadow counter ended up permanently positive for the affected stripe, and the skip optimization in mmapHeadChunks (mmapReady[i] == 0) never matched it again, silently reverting the optimization. Extract the WAL replay's append-and-mmap pattern into a Head method, appendChunkAndMmap, that decrements mmapReady whenever the prior headChunkCount was >= 2 (mmapChunks always sets the count to 1 when it does work, so prev >= 2 is the only case that needs balancing). The same bug exists in the histogram path; both call sites now use the helper. Add a regression test under TestHead_mmapHeadChunks covering floats, histograms, and float histograms via a table-driven sub-test that builds a series to headChunkCount >= 2, invokes appendChunkAndMmap, and asserts mmapReady is 0. Signed-off-by: Arve Knudsen <[email protected]>

Add prometheus_tsdb_head_chunks_max_mmapped, a gauge updated on each mmapHeadChunks pass to the maximum number of chunks m-mapped from a single series during that pass. The metric is scoped to mmap-ready series (those with headChunkCount >= 2) so it stays correct after the per-stripe mmapReady optimisation in #18541. Signed-off-by: György Krajcsovits <[email protected]>

…➔ v3.12.0) (#730) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [quay.io/prometheus/prometheus](https://github.com/prometheus/prometheus) | minor | `v3.11.3` → `v3.12.0` | --- ### Release Notes <details> <summary>prometheus/prometheus (quay.io/prometheus/prometheus)</summary> ### [`v3.12.0`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0): 3.12.0 / 2026-05-28 [Compare Source](prometheus/prometheus@v3.11.3...v3.12.0) - \[SECURITY] Remote-write: Reject snappy-compressed requests whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate(), and `increase()`calculations, behind a feature flag`use-start-timestamps`. Doesn't work together with extended range selectors `anchored`and`smoothed\`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/730

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

aknuds1 added kind/performance component/tsdb labels Apr 18, 2026

aknuds1 force-pushed the arve/mmap-skip-stripe branch 8 times, most recently from 2a00f8d to 35f7d6d Compare April 19, 2026 12:26

aknuds1 marked this pull request as ready for review April 19, 2026 12:28

aknuds1 requested review from a team, bwplotka, codesome, jesusvazquez and krajorama as code owners April 19, 2026 12:28

aknuds1 force-pushed the arve/mmap-skip-stripe branch 2 times, most recently from ca643aa to 6c40c11 Compare April 19, 2026 12:46

aknuds1 requested a review from bboreham April 20, 2026 06:17

aknuds1 force-pushed the arve/mmap-skip-stripe branch from 6c40c11 to 11143e3 Compare April 21, 2026 07:19

bboreham reviewed Apr 23, 2026

View reviewed changes

Comment thread tsdb/head_append.go Outdated

Comment thread tsdb/head.go Outdated

Comment thread tsdb/head_append.go

Comment thread tsdb/head_append.go Outdated

Comment thread tsdb/head.go Outdated

krajorama reviewed Apr 23, 2026

View reviewed changes

Comment thread tsdb/head.go

aknuds1 force-pushed the arve/mmap-skip-stripe branch from 11143e3 to d1c1bca Compare April 23, 2026 08:35

aknuds1 requested review from bboreham and krajorama April 23, 2026 08:35

krajorama mentioned this pull request Apr 24, 2026

tsdb: add gauge metric for max head chunks m-mapped per series #18572

Draft

bboreham approved these changes May 12, 2026

View reviewed changes

Comment thread tsdb/head.go

Comment thread tsdb/head_append.go

aknuds1 force-pushed the arve/mmap-skip-stripe branch from 7f1b228 to f122b42 Compare May 12, 2026 15:21

aknuds1 added 2 commits May 14, 2026 13:30

aknuds1 force-pushed the arve/mmap-skip-stripe branch from f122b42 to b3baded Compare May 14, 2026 11:31

aknuds1 force-pushed the arve/mmap-skip-stripe branch from 4acc4ec to 746445e Compare May 14, 2026 13:59

aknuds1 merged commit cf4505c into prometheus:main May 14, 2026
34 checks passed

krajorama mentioned this pull request May 15, 2026

update main from upstream 49cf97be7e7d6c63fd14dd60f2f52f776385ca49 grafana/mimir-prometheus#1161

Merged

aknuds1 mentioned this pull request Jun 12, 2026

tsdb: preserve WAL replay accounting and ordering after full deletion #18907

Open

github-actions Bot mentioned this pull request Jun 15, 2026

Update prom/prometheus Docker tag to v3.12.0 claytono/infra#2379

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

tsdb: skip entire stripes in mmapHeadChunks via per-stripe ready count#18541

tsdb: skip entire stripes in mmapHeadChunks via per-stripe ready count#18541
aknuds1 merged 6 commits into
prometheus:mainfrom
aknuds1:arve/mmap-skip-stripe

aknuds1 commented Apr 18, 2026 •

edited

Loading

Uh oh!

bboreham left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bboreham left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

aknuds1 commented Apr 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue(s) does the PR fix:

Summary

Testing

Release notes for end users (ALL commits must be considered).

Uh oh!

bboreham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bboreham left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aknuds1 commented Apr 18, 2026 •

edited

Loading