PromQL: `resets()` function considers start timestamp resets by vpranckaitis · Pull Request #18627 · prometheus/prometheus

vpranckaitis · 2026-05-06T13:26:53Z

Updating resets() function to also consider start timestamp resets. It uses the same isStartTimestampReset() function as rate(), increase() and irate() functions, and it is also hidden behind use-start-timestamps feature flag.

Which issue(s) does the PR fix:

–

Release notes for end users (ALL commits must be considered).

Reviewers should verify clarity and quality.

[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag.

Signed-off-by: vpranckaitis <[email protected]>

vpranckaitis · 2026-05-06T13:28:54Z

+# Resets function also considers start timestamp resets.
+eval range from 7m to 15m step 1m resets(cumulative[5m])
+    {} 0x8
+


There's a comment above that mentions that ST storage is not yet implemented for histograms, which explains why this test doesn't yet return expected results.

vpranckaitis · 2026-05-06T13:36:52Z

 		// Skip the comparison for the first sample, just initialize prevSample.
 		if iFloat+iHistogram == 1+firstSampleIndex {
 			prevSample = curSample
+			prevST = curST


Technically, with STs we could also count resets which have happened before the first sample in the range selector window, if the ST of the first sample in the window also points inside the window. The condition for this would probably look similar to the one used in rate extrapolation change PR (#18619). Once we figure out what condition we want to use there, it might make sense to also add that to resets() function.

I have never used "resets" myself, but I've let Claude search for practical examples, see at the bottom.

Based on that , it seems to be reasonable to count ST of first sample falling inside the range as a reset. Since it answers questions about when something (re)started. Also it feels like this would be in line with extrapolation taking it into account if ST falls into the range.

On the other hand, on the right side of the range, we cannot take ST into account , because it might not have arrived yet. And even with extended range selectors, the next sample might not be in range. So that's a little bit inconsistent. But that's already the case without ST, we might be missing a reset because we don't see the next sample.

Question is whether we want to touch changes ? So far we did not take into account histogram counter resets, it just looks at the value to be very simple and applicable to gauges. So that's a no.

Claude research:

PromQL resets() — Practical Use Cases

resets(v range-vector) counts how many times a counter decreased within the
time window. Unlike rate() and increase(), which silently absorb resets,
resets() makes them explicit.

1. Counting service/process restarts (canonical use)

Every restart resets all counters to 0, so any counter shows a reset:

resets(process_cpu_seconds_total[1h])

2. Counting probe/health-check failures (creative trick)

The Blackbox exporter's probe_success is 0 or 1. Every 1→0 transition looks
like a counter reset:

resets(probe_success{job="blackbox"}[1h])

This counts how many times a target started failing in the last hour — a
historical failure-event count, not just the current state.

3. Counting session establishments

Per-session counters (VPN, DB connections, batch jobs) reset to 0 on teardown.
Each new session is a reset:

resets(vpn_user_bytes_total{user="alice"}[24h])

4. Counting downward gauge movements

Combined with changes(), lets you isolate upward vs downward movements of a
gauge:

changes(queue_depth[1h]) - resets(queue_depth[1h]) # number of times queue grew

5. Diagnostics for IoT / edge devices

Used to verify that increase() is correctly accounting for counter resets on
devices like smart plugs that reset their energy counter on power loss.

Key Limitations

Scrape interval blind spot: resets faster than the scrape interval are invisible.

Staleness gaps: if a series disappears and reappears at the same or higher
value, it is not counted as a reset.

Non-binary gauges: applying resets() to multi-value gauges will count
every downward step, not just full resets to zero.

krajorama

Approved with nit, seems like a nice small change.
We can discuss the merits of detecting the reset for first sample separately.

Signed-off-by: vpranckaitis <[email protected]>

bwplotka · 2026-05-12T07:29:03Z

Amazing, thanks for this!

…➔ v3.12.0) (#730) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [quay.io/prometheus/prometheus](https://github.com/prometheus/prometheus) | minor | `v3.11.3` → `v3.12.0` | --- ### Release Notes <details> <summary>prometheus/prometheus (quay.io/prometheus/prometheus)</summary> ### [`v3.12.0`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0): 3.12.0 / 2026-05-28 [Compare Source](prometheus/prometheus@v3.11.3...v3.12.0) - \[SECURITY] Remote-write: Reject snappy-compressed requests whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate(), and `increase()`calculations, behind a feature flag`use-start-timestamps`. Doesn't work together with extended range selectors `anchored`and`smoothed\`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/730

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

PromQL: resets() function considers start timestamp resets

0baf1ca

Signed-off-by: vpranckaitis <[email protected]>

vpranckaitis requested review from a team and roidelapluie as code owners May 6, 2026 13:26

vpranckaitis commented May 6, 2026

View reviewed changes

krajorama requested review from carrieedwards, krajorama and ywwg May 8, 2026 07:25

krajorama approved these changes May 8, 2026

View reviewed changes

Comment thread promql/functions.go Outdated

vpranckaitis added 3 commits May 8, 2026 12:45

PR comment

f09d073

Signed-off-by: vpranckaitis <[email protected]>

Merge branch 'main' into consider_start_timestamps_in_resets_function

3930124

Merge branch 'main' into consider_start_timestamps_in_resets_function

af4b544

krajorama approved these changes May 11, 2026

View reviewed changes

krajorama merged commit 129ee0d into prometheus:main May 11, 2026
57 of 58 checks passed

This was referenced Jul 1, 2026

PromQL: Add experimental st_of_last_over_time() function #19041

Closed

PromQL: add experimental start_timestamp() function #19089

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

PromQL: `resets()` function considers start timestamp resets#18627

PromQL: `resets()` function considers start timestamp resets#18627
krajorama merged 4 commits into
prometheus:mainfrom
vpranckaitis:consider_start_timestamps_in_resets_function

vpranckaitis commented May 6, 2026

Uh oh!

vpranckaitis May 6, 2026

Uh oh!

vpranckaitis May 6, 2026

Uh oh!

krajorama May 8, 2026

Uh oh!

krajorama left a comment

Uh oh!

Uh oh!

Uh oh!

bwplotka commented May 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

vpranckaitis commented May 6, 2026

Which issue(s) does the PR fix:

Release notes for end users (ALL commits must be considered).

Uh oh!

vpranckaitis May 6, 2026

Choose a reason for hiding this comment

Uh oh!

vpranckaitis May 6, 2026

Choose a reason for hiding this comment

Uh oh!

krajorama May 8, 2026

Choose a reason for hiding this comment

PromQL resets() — Practical Use Cases

1. Counting service/process restarts (canonical use)

2. Counting probe/health-check failures (creative trick)

3. Counting session establishments

4. Counting downward gauge movements

5. Diagnostics for IoT / edge devices

Key Limitations

Uh oh!

krajorama left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bwplotka commented May 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

PromQL `resets()` — Practical Use Cases