discovery: add DigitalOcean Managed Databases service discovery by Skesov · Pull Request #18287 · prometheus/prometheus

Skesov · 2026-03-12T16:04:25Z

This PR adds databases role to digitalocean_sd_config to discover DigitalOcean Managed Database clusters.

It addresses the proposal in #18286.

Changes:

Added role parameter to digitalocean_sd_config (default: droplets).
Added logic to discover DigitalOcean Managed Database clusters when role: databases is used.
Supports pagination for the Databases API.
Provides comprehensive meta labels for database targets.
Updated documentation and tests.

[FEATURE] Add DigitalOcean Managed Databases service discovery

Signed-off-by: Vladimir Skesov [email protected]

roidelapluie · 2026-03-12T16:09:47Z

should we have a role: in the digitalocean SD instead? cc @SuperQ

SuperQ

For vendors with multiple discoveries we're migrating to a multi-role design pattern.

You can see how we did this in #17406

Skesov · 2026-03-12T17:19:44Z

Done! I've switched to the multi-role pattern as suggested. It's now integrated into the existing digitalocean_sd_configs with a new role field. Thanks for the tip!

Skesov · 2026-03-20T08:42:46Z

Updated the branch by rebasing on the latest main to keep the PR up to date.

This adds 'databases' role to digitalocean_sd_config to discover DigitalOcean Managed Database clusters. It follows the multi-role design pattern by introducing a 'role' parameter (default: 'droplets'). Includes: - Support for Managed Databases API. - Pagination handling for Databases API. - Comprehensive meta labels for database targets. - Updated documentation and tests. Signed-off-by: Vladimir Skesov <[email protected]>

Skesov · 2026-03-30T08:31:42Z

Friendly ping @SuperQ @pedro-stanaka -- the requested changes (multi-role pattern) have been applied, and I've rebased onto the latest main. Ready for re-review when you have time. Thanks!

SuperQ

Minor typo nit, otherwise LGTM.

SuperQ · 2026-03-30T08:55:27Z

@matt-gp Mind giving this a once-over since it's a similar change to what we did with AWS?

Co-authored-by: Ben Kochie <[email protected]> Signed-off-by: Skesov <[email protected]>

matt-gp

LGTM

roidelapluie · 2026-03-30T14:48:06Z

Thanks!

…➔ v3.12.0) (#730) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [quay.io/prometheus/prometheus](https://github.com/prometheus/prometheus) | minor | `v3.11.3` → `v3.12.0` | --- ### Release Notes <details> <summary>prometheus/prometheus (quay.io/prometheus/prometheus)</summary> ### [`v3.12.0`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0): 3.12.0 / 2026-05-28 [Compare Source](prometheus/prometheus@v3.11.3...v3.12.0) - \[SECURITY] Remote-write: Reject snappy-compressed requests whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate(), and `increase()`calculations, behind a feature flag`use-start-timestamps`. Doesn't work together with extended range selectors `anchored`and`smoothed\`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/730

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

@hibrian827

##### [\`v3.12.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0) This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes. Thanks to all contributors! #### Key Highlights - **Security**: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery. - **PromQL & Metadata**: Several features and bug fixes related to the experimental "start timestamps" support, including updates to `rate()`, `irate()`, `increase()`, and `resets()`. New experimental functions `start()`, `end()`, `range()`, and `step()` are introduced. - **TSDB Performance**: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage. - **Service Discovery**: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support). - **UI**: Added a web interface for deleting time series and cleaning tombstones. #### Changelog - \[SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to [@hibrian827](https://github.com/hibrian827) for reporting it. [#18642](prometheus/prometheus#18642) - \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@August829](https://github.com/August829) and [@Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#18649](prometheus/prometheus#18649) - \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#18221](prometheus/prometheus#18221) - \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#18411](prometheus/prometheus#18411) - \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#18287](prometheus/prometheus#18287) - \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#18321](prometheus/prometheus#18321) - \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#18139](prometheus/prometheus#18139) - \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#18498](prometheus/prometheus#18498) - \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#17877](prometheus/prometheus#17877) - \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#18627](prometheus/prometheus#18627) - \[FEATURE] Prometheus: Promote auto-reload-config as stable [#18620](prometheus/prometheus#18620) - \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#17948](prometheus/prometheus#17948) - \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#18390](prometheus/prometheus#18390) - \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`. [#18344](prometheus/prometheus#18344) - \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#18279](prometheus/prometheus#18279) - \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#18360](prometheus/prometheus#18360) - \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#18651](prometheus/prometheus#18651) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#18579](prometheus/prometheus#18579) - \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#17171](prometheus/prometheus#17171) - \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#18187](prometheus/prometheus#18187) - \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#18418](prometheus/prometheus#18418) - \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#16088](prometheus/prometheus#16088) - \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#18302](prometheus/prometheus#18302) - \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#18541](prometheus/prometheus#18541) - \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#18272](prometheus/prometheus#18272) - \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#18568](prometheus/prometheus#18568) - \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#17932](prometheus/prometheus#17932) - \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#18624](prometheus/prometheus#18624) - \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#18305](prometheus/prometheus#18305) - \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#18567](prometheus/prometheus#18567) - \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#18209](prometheus/prometheus#18209) - \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#18479](prometheus/prometheus#18479) [#18499](prometheus/prometheus#18499) - \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#18408](prometheus/prometheus#18408) - \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#18523](prometheus/prometheus#18523) - \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#18409](prometheus/prometheus#18409) - \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#18660](prometheus/prometheus#18660) - \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#18623](prometheus/prometheus#18623) - \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#18531](prometheus/prometheus#18531) - \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#18639](prometheus/prometheus#18639) - \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#18414](prometheus/prometheus#18414) - \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#18382](prometheus/prometheus#18382) - \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#18421](prometheus/prometheus#18421) - \[BUGFIX] TSDB: Allow retention percentage with new data path. [#18628](prometheus/prometheus#18628) - \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#18374](prometheus/prometheus#18374) - \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#18401](prometheus/prometheus#18401) - \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#18692](prometheus/prometheus#18692) - \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#18469](prometheus/prometheus#18469) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#18682](prometheus/prometheus#18682) - \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#17614](prometheus/prometheus#17614) - \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#18641](prometheus/prometheus#18641) - \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#18292](prometheus/prometheus#18292) - \[BUGFIX] Config: Update `--enable-feature` flag description and sort feature names. [#18487](prometheus/prometheus#18487)

Skesov requested a review from a team as a code owner March 12, 2026 16:04

Skesov requested a review from pedro-stanaka March 12, 2026 16:04

SuperQ requested changes Mar 12, 2026

View reviewed changes

Skesov force-pushed the feat/digitalocean-db-sd branch 3 times, most recently from bde6983 to 37aae6b Compare March 12, 2026 17:18

Skesov force-pushed the feat/digitalocean-db-sd branch 2 times, most recently from 03566f4 to 7ae2fa5 Compare March 12, 2026 17:22

Skesov requested a review from SuperQ March 12, 2026 17:52

Skesov force-pushed the feat/digitalocean-db-sd branch from 7ae2fa5 to 09624d5 Compare March 20, 2026 08:41

pedro-stanaka removed their request for review March 20, 2026 10:20

Skesov force-pushed the feat/digitalocean-db-sd branch from 09624d5 to 20ff771 Compare March 30, 2026 08:31

SuperQ reviewed Mar 30, 2026

View reviewed changes

Comment thread docs/configuration/configuration.md Outdated

SuperQ approved these changes Mar 30, 2026

View reviewed changes

SuperQ requested review from a team, RichiH and matt-gp and removed request for a team and RichiH March 30, 2026 08:54

Update docs/configuration/configuration.md

cf09d9b

Co-authored-by: Ben Kochie <[email protected]> Signed-off-by: Skesov <[email protected]>

matt-gp approved these changes Mar 30, 2026

View reviewed changes

roidelapluie merged commit ced39f8 into prometheus:main Mar 30, 2026
35 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

discovery: add DigitalOcean Managed Databases service discovery#18287

discovery: add DigitalOcean Managed Databases service discovery#18287
roidelapluie merged 2 commits into
prometheus:mainfrom
Skesov:feat/digitalocean-db-sd

Skesov commented Mar 12, 2026 •

edited

Loading

Uh oh!

roidelapluie commented Mar 12, 2026

Uh oh!

SuperQ left a comment

Uh oh!

Skesov commented Mar 12, 2026

Uh oh!

Skesov commented Mar 20, 2026

Uh oh!

Skesov commented Mar 30, 2026

Uh oh!

Uh oh!

SuperQ left a comment

Uh oh!

SuperQ commented Mar 30, 2026

Uh oh!

matt-gp left a comment

Uh oh!

Uh oh!

roidelapluie commented Mar 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Skesov commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

roidelapluie commented Mar 12, 2026

Uh oh!

SuperQ left a comment

Choose a reason for hiding this comment

Uh oh!

Skesov commented Mar 12, 2026

Uh oh!

Skesov commented Mar 20, 2026

Uh oh!

Skesov commented Mar 30, 2026

Uh oh!

Uh oh!

SuperQ left a comment

Choose a reason for hiding this comment

Uh oh!

SuperQ commented Mar 30, 2026

Uh oh!

matt-gp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

roidelapluie commented Mar 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Skesov commented Mar 12, 2026 •

edited

Loading