Skip to content

scrape: keep staleness tracking in sync when the series ref changes - #19328

Merged
krajorama merged 3 commits into
mainfrom
krajo/scrape-staleness-ref-change
Aug 17, 2026
Merged

krajorama merged 3 commits into
mainfrom
krajo/scrape-staleness-ref-change

Conversation

@krajorama

@krajorama krajorama commented Aug 4, 2026 •

Copy link
Copy Markdown
Member

Follow up on #19325.

Staleness is tracked in scrapeCache.seriesCur/seriesPrev, both keyed by
storage.SeriesRef. When the storage hands out a new reference for a series it
already gave one for (e.g. it was garbage collected and had to be recreated), the
scrape loop updates the cached reference and tracks the current scrape under the
new one, while the previous scrape is still tracked under the old one. The old
reference is then missing from the current scrape, so forEachStale reports the
series as stale even though the target still exposes it.

The stale marker is appended for ce.ref, i.e. for the new, live reference, at
the same timestamp as the sample of this scrape. headAppender.Commit writes the
WAL before applying samples to the head, so while the head itself drops the
marker as a duplicate, it is already persisted:

StaleNaN append at same t returned err=<nil>
WAL sample: ref=1 t=1000 v=1   stale=false
WAL sample: ref=1 t=1000 v=NaN stale=true

Remote write and agent mode read the WAL, so they ship the marker downstream and
the series ends there until the next scrape.

This affects appender v2 since the reference was first written back to the cache,
and appender v1 since #19325 made it consistent with v2. v2 is what the Prometheus
binary runs today (cmd/prometheus/main.go passes nil, fanoutStorage to
scrape.NewManager), so the fix is done for both loops.

scrapeCache.updateRef now re-keys the staleness tracking to the new reference,
and both scrape loops go through it. It deliberately moves the tracking instead
of dropping it: dropping it would suppress the stale marker of a series that is
still exposed but stopped being tracked for another reason in the same scrape,
which would make staleness depend on whether the reference happened to change.
The one case where that is observable is a series that starts carrying an explicit
timestamp, which the last two test cases pin down.

Tests

TestScrapeLoopCacheRefUpdatedOnChange from #19325 is reworked into the table
driven TestScrapeLoopSeriesRefChange, which drives a sequence of scrapes and
asserts on the samples the scrape loop appends, not only on the cached reference.
It reuses the teststorage.Appendable mock with an injected reference rather than
a purpose built appendable, which is what makes the stale marker visible. All
cases run against both scrape loops:

case before after
reference stays the same pass pass
reference changes while the series is still exposed stale marker pass
reference changes on every scrape stale marker per change pass
series stops being exposed after the reference changed extra stale marker pass
explicit timestamp appears, reference stays the same pass pass
explicit timestamp appears and the reference changes pass pass

The last two agree by construction, which is what keeps updateRef from turning
into a plain delete.

Commit 2 adds the test and fails on purpose, commit 3 fixes the bug.

[BUGFIX] Scrape: Do not append a stale marker for a series that is still exposed when the storage returns a new series reference for it.

@krajorama
krajorama marked this pull request as ready for review August 4, 2026 08:49
@krajorama
krajorama requested a review from a team as a code owner August 4, 2026 08:49
kgeckhart
kgeckhart previously approved these changes Aug 4, 2026

@kgeckhart kgeckhart left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch by claude on that initial PR. I missed it when doing the implementation. I only asked about the safety of updated the value on the ref 🙈 .

@krajorama
krajorama marked this pull request as draft August 4, 2026 14:01
WithRefFn lets a test decide the storage.SeriesRef the mock appender hands
back, so it can emulate a storage that returns a new reference for a series
it already handed one out for. This also skips the label hash check on the
incoming reference, which no longer holds once refs are injected.

Signed-off-by: György Krajcsovits <[email protected]>
Reworks TestScrapeLoopCacheRefUpdatedOnChange into a table driven test that
drives a sequence of scrapes and asserts on the samples the scrape loop appends,
not only on the cached reference. It reuses the teststorage.Appendable mock with
an injected reference instead of a purpose built appendable.

This shows that when the storage hands out a new reference for a series that is
still exposed, both the v1 and the v2 scrape loop append a stale marker for it.
Staleness is tracked per storage reference, so updating the cached reference
leaves the previous scrape tracked under the old one, and the series looks like
it stopped being exposed.

The two explicit timestamp cases pass already. They pin down that a series which
starts carrying a timestamp gets a stale marker whether or not its reference
changed in that scrape, so that fixing the above does not silently change it.

This commit fails on purpose, the next one fixes the bug.

Signed-off-by: György Krajcsovits <[email protected]>
Staleness is tracked in seriesCur/seriesPrev, both keyed by storage.SeriesRef.
When the storage hands out a new reference for a series that is still exposed,
the scrape loop updates the cached reference and tracks the current scrape under
the new one, while the previous scrape stays tracked under the old one. The old
reference is then missing from the current scrape, so the series is reported as
stale even though the target still exposes it.

Worse, the stale marker is appended for the new reference, i.e. for the live
series, at the same timestamp as the sample of this scrape. The head drops it as
a duplicate, but only after it was written to the WAL, so remote write and agent
mode ship the marker downstream and the series ends there until the next scrape.

scrapeCache.updateRef, which both the v1 and the v2 scrape loop now use to update
a cached reference, re-keys the staleness tracking to the new reference. It moves
the tracking rather than dropping it, so whether a series is considered stale does
not depend on its reference having changed.

Signed-off-by: György Krajcsovits <[email protected]>
@krajorama

Copy link
Copy Markdown
Member Author

I grew suspicious that this might be too easy and seems like I did the same mistake, but this time it losing the staleness marker afterwards :(

@krajorama
krajorama force-pushed the krajo/scrape-staleness-ref-change branch from a29acc6 to ebb2b33 Compare August 4, 2026 14:42
@krajorama
krajorama dismissed kgeckhart’s stale review August 4, 2026 14:48

I missed checking that the tracking works afterwards, so my earlier fix was just half the story.

@krajorama
krajorama marked this pull request as ready for review August 4, 2026 14:48
Comment thread scrape/scrape.go
if ce.ref == ref {
return
}
if ce.ref != 0 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: The guard here is also present on both callers

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to deal with this in a separate PR, I'd like to simplify the code a little bit as well.

Comment thread scrape/scrape_test.go Resolved
@krajorama
krajorama merged commit b5d8f0a into main Aug 17, 2026
38 checks passed
@krajorama
krajorama deleted the krajo/scrape-staleness-ref-change branch August 17, 2026 07:27
renovate Bot added a commit to sdwilsh/ansible-playbooks that referenced this pull request Sep 26, 2026
##### [\`v3.15.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.15.0)

- \[CHANGE] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. Add tests to prevent regression of the fix made in [#18081](prometheus/prometheus#18081). [#18598](prometheus/prometheus#18598)
- \[CHANGE] PromQL: Do not register a start timestamp reset if the start timestamp hasn't changed between subsequent samples. [#19454](prometheus/prometheus#19454)
- \[CHANGE] Logging: Deprecate `--log.level`; use `runtime.log_level` configuration to supply the default level. [#19511](prometheus/prometheus#19511)
- \[FEATURE] Configuration: Allow changing the process log level through `runtime.log_level` on configuration reload. [#19511](prometheus/prometheus#19511)
- \[FEATURE] Prometheus: Add `--auto-gomemlimit.refresh-interval` flag to periodically re-detect the container or system memory limit and update `GOMEMLIMIT` at runtime. [#18843](prometheus/prometheus#18843)
- \[FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets. [#12024](prometheus/prometheus#12024). [#18091](prometheus/prometheus#18091)
- \[FEATURE] scrape: Implement OM2.0 scrape format. [#18606](prometheus/prometheus#18606)
- \[ENHANCEMENT] Reduce TSDB head CPU utilization when initializing. [#18001](prometheus/prometheus#18001)
- \[ENHANCEMENT] Docker SD: Add labels `__meta_docker_container_image` and `__meta_docker_container_image_id`. [#19386](prometheus/prometheus#19386)
- \[ENHANCEMENT] Mixin: Add a p95/p99 remote-write send-batch latency panel to the remote-write dashboard. [#19500](prometheus/prometheus#19500)
- \[ENHANCEMENT] Mixin: Support native histograms in the remote-write send-batch latency panel. [#19522](prometheus/prometheus#19522)
- \[ENHANCEMENT] PromQL/TSDB: The `--enable-feature=st-storage` flag now automatically enables XOR2 float chunk encoding and ST-capable histogram chunk encoding, so you no longer need to pass `xor2-encoding` and `histograms-st-encoding` alongside it. [#19518](prometheus/prometheus#19518)
- \[ENHANCEMENT] Remote write / Alertmanager: upgrade sigv4 to v0.5.0, adding `session_name` and `tags` fields for STS AssumeRole sessions. The previously undocumented `service_name` field is now also documented. [#19569](prometheus/prometheus#19569)
- \[ENHANCEMENT] Scraping: Support zstd-compressed scrape responses, enabled via feature flag `zstd-scrape`. [#19502](prometheus/prometheus#19502)
- \[ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. `--enable-feature=xor2-encoding` is deprecated; use `storage.tsdb.chunk_encoding.floats: xor2` instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. [#19461](prometheus/prometheus#19461)
- \[ENHANCEMENT] TSDB: add `prometheus_tsdb_head_appenders_created_total` metric. [#19411](prometheus/prometheus#19411)
- \[ENHANCEMENT] Tracing: add more spans to scrapes, API queries and rule evaluations. [#19410](prometheus/prometheus#19410)
- \[ENHANCEMENT] UI: Show the effective configuration for each scrape pool on the Targets and Service Discovery pages. [#19384](prometheus/prometheus#19384)
- \[ENHANCEMENT] scrape: Enable start time synthesis for summary `_count` and `_sum` series in scrape appender v2. [#19323](prometheus/prometheus#19323)
- \[ENHANCEMENT] scrape: stop all pools in parallel for faster shutdowns. [#19295](prometheus/prometheus#19295)
- \[ENHANCEMENT] storage/remote: Add undocumented failed\_request\_logging config field to debug log remote write V2 requests on send errors. [#19249](prometheus/prometheus#19249)
- \[ENHANCEMENT] TSDB: Add fast path for XOR chunk decompression to speed up queries. [#18049](prometheus/prometheus#18049)
- \[ENHANCEMENT] UI: Improve native histogram table formatting and add a background bar indicating the bucket count. [#19332](prometheus/prometheus#19332)
- \[ENHANCEMENT] TSDB: Add `prometheus_tsdb_head_series_pending_commit_underflow_total` to report pending-sample reservation underflows. [#19470](prometheus/prometheus#19470)
- \[PERF] AWS SD: Build RDS cluster labels once per cluster instead of once per instance. [#19504](prometheus/prometheus#19504)
- \[PERF] AWS SD: Describe RDS instances of different clusters concurrently, bounded by `request_concurrency`. [#19506](prometheus/prometheus#19506)
- \[PERF] AWS SD: Describe each ElastiCache resource once per refresh instead of twice. [#19585](prometheus/prometheus#19585)
- \[PERF] Remote read: Avoid cloning labels for sampled reads when no external labels are configured. [#19503](prometheus/prometheus#19503)
- \[PERF] Remote write: Reuse OTLP converter scratch state between requests. [#19388](prometheus/prometheus#19388)
- \[PERF] scrape: conversion from classic to native histograms should only parse start times when enabled. [#19446](prometheus/prometheus#19446)
- \[BUGFIX] PromQL: Fix `info()` enrichment for composite expressions with mixed `@`/offset references or selector-free vector branches, preventing metadata from being evaluated at an unrelated timestamp. [#19387](prometheus/prometheus#19387)
- \[BUGFIX] PromQL: Fix info() enrichment when input series use different subsets of identifying labels. [#19557](prometheus/prometheus#19557)
- \[BUGFIX] PromQL: Preserve metric-name dropping through the info function when delayed name removal is enabled. [#19413](prometheus/prometheus#19413)
- \[BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to. [#19272](prometheus/prometheus#19272)
- \[BUGFIX] TSDB: Keep series with uncommitted samples during selected- and stale-series compaction, including when appenders overlap. [#19470](prometheus/prometheus#19470)
- \[BUGFIX] TSDB: Do not retain head series after a synthetic start-timestamp zero sample is rejected. [#19470](prometheus/prometheus#19470)
- \[BUGFIX] TSDB: Prevent query panics during series eviction after WAL replay. [#19664](prometheus/prometheus#19664)
- \[BUGFIX] TSDB: fix potential deadlock between mmapSeriesChunks and gcSeries. [#19460](prometheus/prometheus#19460)
- \[BUGFIX] TSDB: fix default block reload interval for custom options. [#19368](prometheus/prometheus#19368)
- \[BUGFIX] AWS SD: Do not crash on serverless MSK clusters or MSK clusters without Open Monitoring. [#19194](prometheus/prometheus#19194)
- \[BUGFIX] AWS SD: Do not panic when the ElastiCache API omits optional fields of a serverless cache or cache cluster. [#19435](prometheus/prometheus#19435)
- \[BUGFIX] AWS SD: Reject non-positive `request_concurrency` instead of hanging service discovery indefinitely. [#19524](prometheus/prometheus#19524)
- \[BUGFIX] Discovery/AWS: Avoid a panic when discovering standalone ECS tasks with custom task groups. [#19302](prometheus/prometheus#19302)
- \[BUGFIX] Discovery: Do not panic in AWS Lightsail service discovery when an instance is missing optional fields such as availability zone, blueprint, bundle, name, state or support code. [#19324](prometheus/prometheus#19324)
- \[BUGFIX] Discovery: delete the stale `prometheus_sd_last_update_timestamp_seconds` series for a config that is removed on reload. [#19131](prometheus/prometheus#19131)
- \[BUGFIX] HTTP: Avoid truncating compressed responses when handlers set Content-Length. [#19661](prometheus/prometheus#19661)
- \[BUGFIX] IONOS SD: Do not panic when the API response omits the server, NIC or volume collections, or a server's properties. [#19438](prometheus/prometheus#19438)
- \[BUGFIX] Metadata will not affect the number of Remote Write v2 shards. [#19218](prometheus/prometheus#19218)
- \[BUGFIX] Mixins: Fix label mismatches that prevented PrometheusHAGroupNotIngestingSamples and PrometheusHAGroupCrashlooping from firing. [#19444](prometheus/prometheus#19444)
- \[BUGFIX] Native histograms: `DetectReset` no longer misses a counter reset when a populated bucket behind an empty one disappears, which could make histogram `rate()`/`increase()` undercount. [#19367](prometheus/prometheus#19367)
- \[BUGFIX] Never skip histogram buckets for histogram\_stddev and histogram\_stdvar functions. [#19521](prometheus/prometheus#19521)
- \[BUGFIX] OTLP: Do not abort an entire OTLP payload ingestion if one metric has no datapoints. [#19343](prometheus/prometheus#19343)
- \[BUGFIX] PromQL: Fix FastRegexMatcher false-positive match when a capturing group is directly adjacent to a literal (e.g. `.*\|(foo)\|.*`). [#19516](prometheus/prometheus#19516)
- \[BUGFIX] PromQL: Fix a panic in range selectors using the experimental `anchored` or `smoothed` modifier when the selected series has no samples inside the query window, for example a query evaluated inside a scrape gap. [#19431](prometheus/prometheus#19431)
- \[BUGFIX] PromQL: Fix empty results when a subquery with `@` is used as the matrix argument of a call that is not step-invariant (for example `quantile_over_time(scalar(x), metric[...:...] @ T)`). [#19187](prometheus/prometheus#19187)
- \[BUGFIX] PromQL: Make the "found duplicate series for the match group" many-to-many matching error message deterministic by sorting the two duplicate labels. [#18810](prometheus/prometheus#18810)
- \[BUGFIX] PromQL: Preserve parentheses around duration literals on Expr.String() round-trip. [#19403](prometheus/prometheus#19403)
- \[BUGFIX] PromQL: Reject duration-expression offsets and `@ start()` / `@ end()` before range selectors, matching the existing rejection of literal offsets and `@ <timestamp>`. [#19406](prometheus/prometheus#19406)
- \[BUGFIX] PromQL: Report the position of the histogram argument rather than of a scalar argument in the native histogram NaN observation annotations of `histogram_quantile` and `histogram_fraction`. [#19330](prometheus/prometheus#19330)
- \[BUGFIX] PromQL: info() now applies the @ modifier/offset when evaluating the info series, so info(v @ T) enriches as of T consistently instead of depending on the query start/eval time. [#19266](prometheus/prometheus#19266)
- \[BUGFIX] Rules: Fix a panic when a rule manager created without a logger loads a rule file containing multiple YAML documents. [#19433](prometheus/prometheus#19433)
- \[BUGFIX] Scrape: Do not append a stale marker for a series that is still exposed when the storage returns a new series reference for it. [#19328](prometheus/prometheus#19328)
- \[BUGFIX] Scrape: JSON log formatter correctly format scrape target info. [#19472](prometheus/prometheus#19472)
- \[BUGFIX] TSDB: Fix in-order chunk ID overflow by wrapping head chunk IDs modulo 2^23 so they never collide with the out-of-order flag bit. [#19450](prometheus/prometheus#19450)
- \[BUGFIX] TSDB: Fix out-of-order chunk ID overflow by wrapping `firstOOOChunkID` modulo 2^23 instead of growing unbounded. [#19216](prometheus/prometheus#19216)
- \[BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. [#19326](prometheus/prometheus#19326)
- \[BUGFIX] discovery/aws: Avoid a panic when an ElastiCache ARN is missing its resource ID. [#19333](prometheus/prometheus#19333)
- \[BUGFIX] discovery/aws: Do not panic on MSK clusters whose optional API fields are absent. [#19584](prometheus/prometheus#19584)
- \[BUGFIX] discovery/aws: Do not panic when the EC2 API omits optional instance fields. [#19512](prometheus/prometheus#19512)
- \[BUGFIX] discovery/aws: Don't panic on ECS tasks with absent optional fields. [#19396](prometheus/prometheus#19396)
- \[BUGFIX] discovery/aws: MSK Optional Custom Configuration Fields [#19422](prometheus/prometheus#19422). [#19422](prometheus/prometheus#19422)
- \[BUGFIX] discovery/ionos: Fix panic when the IONOS API omits optional server fields. [#19418](prometheus/prometheus#19418)
- \[BUGFIX] discovery/kubernetes: Populate `__meta_kubernetes_service_loadbalancer_ip` from `status.loadBalancer.ingress`, falling back to deprecated `spec.loadBalancerIP`. [#19404](prometheus/prometheus#19404)
- \[BUGFIX] histogram: Fix Compact moving buckets to wrong indices, and producing negative bucket counts for integer histograms, when more than one span is merged in the same pass. [#19312](prometheus/prometheus#19312)
- \[BUGFIX] promtool: Fixed `tsdb dump` silently dropping native histogram samples. [#18051](prometheus/prometheus#18051)
- \[BUGFIX] scrape: fix data race in `Manager.TargetsDroppedCounts` to avoid miscounting dropped targets. [#19304](prometheus/prometheus#19304)
- \[BUGFIX] scrape: fix nil histogram when native and classic histograms are mixed in one metric family. [#19452](prometheus/prometheus#19452)
- \[BUGFIX] TSDB: Fix WAL and GC log messages to emit human-readable duration strings instead of nanosecond integers. [#19307](prometheus/prometheus#19307)
- \[BUGFIX] TSDB: Avoid WAL corruption after a failed WAL write in Agent mode. [#19700](prometheus/prometheus#19700)
- \[BUGFIX] Agent: Ignore unknown WAL record types, to help users rolling back. [#19814](prometheus/prometheus#19814)
renovate Bot added a commit to sdwilsh/ansible-playbooks that referenced this pull request Sep 26, 2026
##### [\`v3.15.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.15.0)

- \[CHANGE] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. Add tests to prevent regression of the fix made in [#18081](prometheus/prometheus#18081). [#18598](prometheus/prometheus#18598)
- \[CHANGE] PromQL: Do not register a start timestamp reset if the start timestamp hasn't changed between subsequent samples. [#19454](prometheus/prometheus#19454)
- \[CHANGE] Logging: Deprecate `--log.level`; use `runtime.log_level` configuration to supply the default level. [#19511](prometheus/prometheus#19511)
- \[FEATURE] Configuration: Allow changing the process log level through `runtime.log_level` on configuration reload. [#19511](prometheus/prometheus#19511)
- \[FEATURE] Prometheus: Add `--auto-gomemlimit.refresh-interval` flag to periodically re-detect the container or system memory limit and update `GOMEMLIMIT` at runtime. [#18843](prometheus/prometheus#18843)
- \[FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets. [#12024](prometheus/prometheus#12024). [#18091](prometheus/prometheus#18091)
- \[FEATURE] scrape: Implement OM2.0 scrape format. [#18606](prometheus/prometheus#18606)
- \[ENHANCEMENT] Reduce TSDB head CPU utilization when initializing. [#18001](prometheus/prometheus#18001)
- \[ENHANCEMENT] Docker SD: Add labels `__meta_docker_container_image` and `__meta_docker_container_image_id`. [#19386](prometheus/prometheus#19386)
- \[ENHANCEMENT] Mixin: Add a p95/p99 remote-write send-batch latency panel to the remote-write dashboard. [#19500](prometheus/prometheus#19500)
- \[ENHANCEMENT] Mixin: Support native histograms in the remote-write send-batch latency panel. [#19522](prometheus/prometheus#19522)
- \[ENHANCEMENT] PromQL/TSDB: The `--enable-feature=st-storage` flag now automatically enables XOR2 float chunk encoding and ST-capable histogram chunk encoding, so you no longer need to pass `xor2-encoding` and `histograms-st-encoding` alongside it. [#19518](prometheus/prometheus#19518)
- \[ENHANCEMENT] Remote write / Alertmanager: upgrade sigv4 to v0.5.0, adding `session_name` and `tags` fields for STS AssumeRole sessions. The previously undocumented `service_name` field is now also documented. [#19569](prometheus/prometheus#19569)
- \[ENHANCEMENT] Scraping: Support zstd-compressed scrape responses, enabled via feature flag `zstd-scrape`. [#19502](prometheus/prometheus#19502)
- \[ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. `--enable-feature=xor2-encoding` is deprecated; use `storage.tsdb.chunk_encoding.floats: xor2` instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. [#19461](prometheus/prometheus#19461)
- \[ENHANCEMENT] TSDB: add `prometheus_tsdb_head_appenders_created_total` metric. [#19411](prometheus/prometheus#19411)
- \[ENHANCEMENT] Tracing: add more spans to scrapes, API queries and rule evaluations. [#19410](prometheus/prometheus#19410)
- \[ENHANCEMENT] UI: Show the effective configuration for each scrape pool on the Targets and Service Discovery pages. [#19384](prometheus/prometheus#19384)
- \[ENHANCEMENT] scrape: Enable start time synthesis for summary `_count` and `_sum` series in scrape appender v2. [#19323](prometheus/prometheus#19323)
- \[ENHANCEMENT] scrape: stop all pools in parallel for faster shutdowns. [#19295](prometheus/prometheus#19295)
- \[ENHANCEMENT] storage/remote: Add undocumented failed\_request\_logging config field to debug log remote write V2 requests on send errors. [#19249](prometheus/prometheus#19249)
- \[ENHANCEMENT] TSDB: Add fast path for XOR chunk decompression to speed up queries. [#18049](prometheus/prometheus#18049)
- \[ENHANCEMENT] UI: Improve native histogram table formatting and add a background bar indicating the bucket count. [#19332](prometheus/prometheus#19332)
- \[ENHANCEMENT] TSDB: Add `prometheus_tsdb_head_series_pending_commit_underflow_total` to report pending-sample reservation underflows. [#19470](prometheus/prometheus#19470)
- \[PERF] AWS SD: Build RDS cluster labels once per cluster instead of once per instance. [#19504](prometheus/prometheus#19504)
- \[PERF] AWS SD: Describe RDS instances of different clusters concurrently, bounded by `request_concurrency`. [#19506](prometheus/prometheus#19506)
- \[PERF] AWS SD: Describe each ElastiCache resource once per refresh instead of twice. [#19585](prometheus/prometheus#19585)
- \[PERF] Remote read: Avoid cloning labels for sampled reads when no external labels are configured. [#19503](prometheus/prometheus#19503)
- \[PERF] Remote write: Reuse OTLP converter scratch state between requests. [#19388](prometheus/prometheus#19388)
- \[PERF] scrape: conversion from classic to native histograms should only parse start times when enabled. [#19446](prometheus/prometheus#19446)
- \[BUGFIX] PromQL: Fix `info()` enrichment for composite expressions with mixed `@`/offset references or selector-free vector branches, preventing metadata from being evaluated at an unrelated timestamp. [#19387](prometheus/prometheus#19387)
- \[BUGFIX] PromQL: Fix info() enrichment when input series use different subsets of identifying labels. [#19557](prometheus/prometheus#19557)
- \[BUGFIX] PromQL: Preserve metric-name dropping through the info function when delayed name removal is enabled. [#19413](prometheus/prometheus#19413)
- \[BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to. [#19272](prometheus/prometheus#19272)
- \[BUGFIX] TSDB: Keep series with uncommitted samples during selected- and stale-series compaction, including when appenders overlap. [#19470](prometheus/prometheus#19470)
- \[BUGFIX] TSDB: Do not retain head series after a synthetic start-timestamp zero sample is rejected. [#19470](prometheus/prometheus#19470)
- \[BUGFIX] TSDB: Prevent query panics during series eviction after WAL replay. [#19664](prometheus/prometheus#19664)
- \[BUGFIX] TSDB: fix potential deadlock between mmapSeriesChunks and gcSeries. [#19460](prometheus/prometheus#19460)
- \[BUGFIX] TSDB: fix default block reload interval for custom options. [#19368](prometheus/prometheus#19368)
- \[BUGFIX] AWS SD: Do not crash on serverless MSK clusters or MSK clusters without Open Monitoring. [#19194](prometheus/prometheus#19194)
- \[BUGFIX] AWS SD: Do not panic when the ElastiCache API omits optional fields of a serverless cache or cache cluster. [#19435](prometheus/prometheus#19435)
- \[BUGFIX] AWS SD: Reject non-positive `request_concurrency` instead of hanging service discovery indefinitely. [#19524](prometheus/prometheus#19524)
- \[BUGFIX] Discovery/AWS: Avoid a panic when discovering standalone ECS tasks with custom task groups. [#19302](prometheus/prometheus#19302)
- \[BUGFIX] Discovery: Do not panic in AWS Lightsail service discovery when an instance is missing optional fields such as availability zone, blueprint, bundle, name, state or support code. [#19324](prometheus/prometheus#19324)
- \[BUGFIX] Discovery: delete the stale `prometheus_sd_last_update_timestamp_seconds` series for a config that is removed on reload. [#19131](prometheus/prometheus#19131)
- \[BUGFIX] HTTP: Avoid truncating compressed responses when handlers set Content-Length. [#19661](prometheus/prometheus#19661)
- \[BUGFIX] IONOS SD: Do not panic when the API response omits the server, NIC or volume collections, or a server's properties. [#19438](prometheus/prometheus#19438)
- \[BUGFIX] Metadata will not affect the number of Remote Write v2 shards. [#19218](prometheus/prometheus#19218)
- \[BUGFIX] Mixins: Fix label mismatches that prevented PrometheusHAGroupNotIngestingSamples and PrometheusHAGroupCrashlooping from firing. [#19444](prometheus/prometheus#19444)
- \[BUGFIX] Native histograms: `DetectReset` no longer misses a counter reset when a populated bucket behind an empty one disappears, which could make histogram `rate()`/`increase()` undercount. [#19367](prometheus/prometheus#19367)
- \[BUGFIX] Never skip histogram buckets for histogram\_stddev and histogram\_stdvar functions. [#19521](prometheus/prometheus#19521)
- \[BUGFIX] OTLP: Do not abort an entire OTLP payload ingestion if one metric has no datapoints. [#19343](prometheus/prometheus#19343)
- \[BUGFIX] PromQL: Fix FastRegexMatcher false-positive match when a capturing group is directly adjacent to a literal (e.g. `.*\|(foo)\|.*`). [#19516](prometheus/prometheus#19516)
- \[BUGFIX] PromQL: Fix a panic in range selectors using the experimental `anchored` or `smoothed` modifier when the selected series has no samples inside the query window, for example a query evaluated inside a scrape gap. [#19431](prometheus/prometheus#19431)
- \[BUGFIX] PromQL: Fix empty results when a subquery with `@` is used as the matrix argument of a call that is not step-invariant (for example `quantile_over_time(scalar(x), metric[...:...] @ T)`). [#19187](prometheus/prometheus#19187)
- \[BUGFIX] PromQL: Make the "found duplicate series for the match group" many-to-many matching error message deterministic by sorting the two duplicate labels. [#18810](prometheus/prometheus#18810)
- \[BUGFIX] PromQL: Preserve parentheses around duration literals on Expr.String() round-trip. [#19403](prometheus/prometheus#19403)
- \[BUGFIX] PromQL: Reject duration-expression offsets and `@ start()` / `@ end()` before range selectors, matching the existing rejection of literal offsets and `@ <timestamp>`. [#19406](prometheus/prometheus#19406)
- \[BUGFIX] PromQL: Report the position of the histogram argument rather than of a scalar argument in the native histogram NaN observation annotations of `histogram_quantile` and `histogram_fraction`. [#19330](prometheus/prometheus#19330)
- \[BUGFIX] PromQL: info() now applies the @ modifier/offset when evaluating the info series, so info(v @ T) enriches as of T consistently instead of depending on the query start/eval time. [#19266](prometheus/prometheus#19266)
- \[BUGFIX] Rules: Fix a panic when a rule manager created without a logger loads a rule file containing multiple YAML documents. [#19433](prometheus/prometheus#19433)
- \[BUGFIX] Scrape: Do not append a stale marker for a series that is still exposed when the storage returns a new series reference for it. [#19328](prometheus/prometheus#19328)
- \[BUGFIX] Scrape: JSON log formatter correctly format scrape target info. [#19472](prometheus/prometheus#19472)
- \[BUGFIX] TSDB: Fix in-order chunk ID overflow by wrapping head chunk IDs modulo 2^23 so they never collide with the out-of-order flag bit. [#19450](prometheus/prometheus#19450)
- \[BUGFIX] TSDB: Fix out-of-order chunk ID overflow by wrapping `firstOOOChunkID` modulo 2^23 instead of growing unbounded. [#19216](prometheus/prometheus#19216)
- \[BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. [#19326](prometheus/prometheus#19326)
- \[BUGFIX] discovery/aws: Avoid a panic when an ElastiCache ARN is missing its resource ID. [#19333](prometheus/prometheus#19333)
- \[BUGFIX] discovery/aws: Do not panic on MSK clusters whose optional API fields are absent. [#19584](prometheus/prometheus#19584)
- \[BUGFIX] discovery/aws: Do not panic when the EC2 API omits optional instance fields. [#19512](prometheus/prometheus#19512)
- \[BUGFIX] discovery/aws: Don't panic on ECS tasks with absent optional fields. [#19396](prometheus/prometheus#19396)
- \[BUGFIX] discovery/aws: MSK Optional Custom Configuration Fields [#19422](prometheus/prometheus#19422). [#19422](prometheus/prometheus#19422)
- \[BUGFIX] discovery/ionos: Fix panic when the IONOS API omits optional server fields. [#19418](prometheus/prometheus#19418)
- \[BUGFIX] discovery/kubernetes: Populate `__meta_kubernetes_service_loadbalancer_ip` from `status.loadBalancer.ingress`, falling back to deprecated `spec.loadBalancerIP`. [#19404](prometheus/prometheus#19404)
- \[BUGFIX] histogram: Fix Compact moving buckets to wrong indices, and producing negative bucket counts for integer histograms, when more than one span is merged in the same pass. [#19312](prometheus/prometheus#19312)
- \[BUGFIX] promtool: Fixed `tsdb dump` silently dropping native histogram samples. [#18051](prometheus/prometheus#18051)
- \[BUGFIX] scrape: fix data race in `Manager.TargetsDroppedCounts` to avoid miscounting dropped targets. [#19304](prometheus/prometheus#19304)
- \[BUGFIX] scrape: fix nil histogram when native and classic histograms are mixed in one metric family. [#19452](prometheus/prometheus#19452)
- \[BUGFIX] TSDB: Fix WAL and GC log messages to emit human-readable duration strings instead of nanosecond integers. [#19307](prometheus/prometheus#19307)
- \[BUGFIX] TSDB: Avoid WAL corruption after a failed WAL write in Agent mode. [#19700](prometheus/prometheus#19700)
- \[BUGFIX] Agent: Ignore unknown WAL record types, to help users rolling back. [#19814](prometheus/prometheus#19814)
renovate Bot added a commit to sdwilsh/ansible-playbooks that referenced this pull request Sep 26, 2026
##### [\`v3.15.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.15.0)

- \[CHANGE] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. Add tests to prevent regression of the fix made in [#18081](prometheus/prometheus#18081). [#18598](prometheus/prometheus#18598)
- \[CHANGE] PromQL: Do not register a start timestamp reset if the start timestamp hasn't changed between subsequent samples. [#19454](prometheus/prometheus#19454)
- \[CHANGE] Logging: Deprecate `--log.level`; use `runtime.log_level` configuration to supply the default level. [#19511](prometheus/prometheus#19511)
- \[FEATURE] Configuration: Allow changing the process log level through `runtime.log_level` on configuration reload. [#19511](prometheus/prometheus#19511)
- \[FEATURE] Prometheus: Add `--auto-gomemlimit.refresh-interval` flag to periodically re-detect the container or system memory limit and update `GOMEMLIMIT` at runtime. [#18843](prometheus/prometheus#18843)
- \[FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets. [#12024](prometheus/prometheus#12024). [#18091](prometheus/prometheus#18091)
- \[FEATURE] scrape: Implement OM2.0 scrape format. [#18606](prometheus/prometheus#18606)
- \[ENHANCEMENT] Reduce TSDB head CPU utilization when initializing. [#18001](prometheus/prometheus#18001)
- \[ENHANCEMENT] Docker SD: Add labels `__meta_docker_container_image` and `__meta_docker_container_image_id`. [#19386](prometheus/prometheus#19386)
- \[ENHANCEMENT] Mixin: Add a p95/p99 remote-write send-batch latency panel to the remote-write dashboard. [#19500](prometheus/prometheus#19500)
- \[ENHANCEMENT] Mixin: Support native histograms in the remote-write send-batch latency panel. [#19522](prometheus/prometheus#19522)
- \[ENHANCEMENT] PromQL/TSDB: The `--enable-feature=st-storage` flag now automatically enables XOR2 float chunk encoding and ST-capable histogram chunk encoding, so you no longer need to pass `xor2-encoding` and `histograms-st-encoding` alongside it. [#19518](prometheus/prometheus#19518)
- \[ENHANCEMENT] Remote write / Alertmanager: upgrade sigv4 to v0.5.0, adding `session_name` and `tags` fields for STS AssumeRole sessions. The previously undocumented `service_name` field is now also documented. [#19569](prometheus/prometheus#19569)
- \[ENHANCEMENT] Scraping: Support zstd-compressed scrape responses, enabled via feature flag `zstd-scrape`. [#19502](prometheus/prometheus#19502)
- \[ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. `--enable-feature=xor2-encoding` is deprecated; use `storage.tsdb.chunk_encoding.floats: xor2` instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. [#19461](prometheus/prometheus#19461)
- \[ENHANCEMENT] TSDB: add `prometheus_tsdb_head_appenders_created_total` metric. [#19411](prometheus/prometheus#19411)
- \[ENHANCEMENT] Tracing: add more spans to scrapes, API queries and rule evaluations. [#19410](prometheus/prometheus#19410)
- \[ENHANCEMENT] UI: Show the effective configuration for each scrape pool on the Targets and Service Discovery pages. [#19384](prometheus/prometheus#19384)
- \[ENHANCEMENT] scrape: Enable start time synthesis for summary `_count` and `_sum` series in scrape appender v2. [#19323](prometheus/prometheus#19323)
- \[ENHANCEMENT] scrape: stop all pools in parallel for faster shutdowns. [#19295](prometheus/prometheus#19295)
- \[ENHANCEMENT] storage/remote: Add undocumented failed\_request\_logging config field to debug log remote write V2 requests on send errors. [#19249](prometheus/prometheus#19249)
- \[ENHANCEMENT] TSDB: Add fast path for XOR chunk decompression to speed up queries. [#18049](prometheus/prometheus#18049)
- \[ENHANCEMENT] UI: Improve native histogram table formatting and add a background bar indicating the bucket count. [#19332](prometheus/prometheus#19332)
- \[ENHANCEMENT] TSDB: Add `prometheus_tsdb_head_series_pending_commit_underflow_total` to report pending-sample reservation underflows. [#19470](prometheus/prometheus#19470)
- \[PERF] AWS SD: Build RDS cluster labels once per cluster instead of once per instance. [#19504](prometheus/prometheus#19504)
- \[PERF] AWS SD: Describe RDS instances of different clusters concurrently, bounded by `request_concurrency`. [#19506](prometheus/prometheus#19506)
- \[PERF] AWS SD: Describe each ElastiCache resource once per refresh instead of twice. [#19585](prometheus/prometheus#19585)
- \[PERF] Remote read: Avoid cloning labels for sampled reads when no external labels are configured. [#19503](prometheus/prometheus#19503)
- \[PERF] Remote write: Reuse OTLP converter scratch state between requests. [#19388](prometheus/prometheus#19388)
- \[PERF] scrape: conversion from classic to native histograms should only parse start times when enabled. [#19446](prometheus/prometheus#19446)
- \[BUGFIX] PromQL: Fix `info()` enrichment for composite expressions with mixed `@`/offset references or selector-free vector branches, preventing metadata from being evaluated at an unrelated timestamp. [#19387](prometheus/prometheus#19387)
- \[BUGFIX] PromQL: Fix info() enrichment when input series use different subsets of identifying labels. [#19557](prometheus/prometheus#19557)
- \[BUGFIX] PromQL: Preserve metric-name dropping through the info function when delayed name removal is enabled. [#19413](prometheus/prometheus#19413)
- \[BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to. [#19272](prometheus/prometheus#19272)
- \[BUGFIX] TSDB: Keep series with uncommitted samples during selected- and stale-series compaction, including when appenders overlap. [#19470](prometheus/prometheus#19470)
- \[BUGFIX] TSDB: Do not retain head series after a synthetic start-timestamp zero sample is rejected. [#19470](prometheus/prometheus#19470)
- \[BUGFIX] TSDB: Prevent query panics during series eviction after WAL replay. [#19664](prometheus/prometheus#19664)
- \[BUGFIX] TSDB: fix potential deadlock between mmapSeriesChunks and gcSeries. [#19460](prometheus/prometheus#19460)
- \[BUGFIX] TSDB: fix default block reload interval for custom options. [#19368](prometheus/prometheus#19368)
- \[BUGFIX] AWS SD: Do not crash on serverless MSK clusters or MSK clusters without Open Monitoring. [#19194](prometheus/prometheus#19194)
- \[BUGFIX] AWS SD: Do not panic when the ElastiCache API omits optional fields of a serverless cache or cache cluster. [#19435](prometheus/prometheus#19435)
- \[BUGFIX] AWS SD: Reject non-positive `request_concurrency` instead of hanging service discovery indefinitely. [#19524](prometheus/prometheus#19524)
- \[BUGFIX] Discovery/AWS: Avoid a panic when discovering standalone ECS tasks with custom task groups. [#19302](prometheus/prometheus#19302)
- \[BUGFIX] Discovery: Do not panic in AWS Lightsail service discovery when an instance is missing optional fields such as availability zone, blueprint, bundle, name, state or support code. [#19324](prometheus/prometheus#19324)
- \[BUGFIX] Discovery: delete the stale `prometheus_sd_last_update_timestamp_seconds` series for a config that is removed on reload. [#19131](prometheus/prometheus#19131)
- \[BUGFIX] HTTP: Avoid truncating compressed responses when handlers set Content-Length. [#19661](prometheus/prometheus#19661)
- \[BUGFIX] IONOS SD: Do not panic when the API response omits the server, NIC or volume collections, or a server's properties. [#19438](prometheus/prometheus#19438)
- \[BUGFIX] Metadata will not affect the number of Remote Write v2 shards. [#19218](prometheus/prometheus#19218)
- \[BUGFIX] Mixins: Fix label mismatches that prevented PrometheusHAGroupNotIngestingSamples and PrometheusHAGroupCrashlooping from firing. [#19444](prometheus/prometheus#19444)
- \[BUGFIX] Native histograms: `DetectReset` no longer misses a counter reset when a populated bucket behind an empty one disappears, which could make histogram `rate()`/`increase()` undercount. [#19367](prometheus/prometheus#19367)
- \[BUGFIX] Never skip histogram buckets for histogram\_stddev and histogram\_stdvar functions. [#19521](prometheus/prometheus#19521)
- \[BUGFIX] OTLP: Do not abort an entire OTLP payload ingestion if one metric has no datapoints. [#19343](prometheus/prometheus#19343)
- \[BUGFIX] PromQL: Fix FastRegexMatcher false-positive match when a capturing group is directly adjacent to a literal (e.g. `.*\|(foo)\|.*`). [#19516](prometheus/prometheus#19516)
- \[BUGFIX] PromQL: Fix a panic in range selectors using the experimental `anchored` or `smoothed` modifier when the selected series has no samples inside the query window, for example a query evaluated inside a scrape gap. [#19431](prometheus/prometheus#19431)
- \[BUGFIX] PromQL: Fix empty results when a subquery with `@` is used as the matrix argument of a call that is not step-invariant (for example `quantile_over_time(scalar(x), metric[...:...] @ T)`). [#19187](prometheus/prometheus#19187)
- \[BUGFIX] PromQL: Make the "found duplicate series for the match group" many-to-many matching error message deterministic by sorting the two duplicate labels. [#18810](prometheus/prometheus#18810)
- \[BUGFIX] PromQL: Preserve parentheses around duration literals on Expr.String() round-trip. [#19403](prometheus/prometheus#19403)
- \[BUGFIX] PromQL: Reject duration-expression offsets and `@ start()` / `@ end()` before range selectors, matching the existing rejection of literal offsets and `@ <timestamp>`. [#19406](prometheus/prometheus#19406)
- \[BUGFIX] PromQL: Report the position of the histogram argument rather than of a scalar argument in the native histogram NaN observation annotations of `histogram_quantile` and `histogram_fraction`. [#19330](prometheus/prometheus#19330)
- \[BUGFIX] PromQL: info() now applies the @ modifier/offset when evaluating the info series, so info(v @ T) enriches as of T consistently instead of depending on the query start/eval time. [#19266](prometheus/prometheus#19266)
- \[BUGFIX] Rules: Fix a panic when a rule manager created without a logger loads a rule file containing multiple YAML documents. [#19433](prometheus/prometheus#19433)
- \[BUGFIX] Scrape: Do not append a stale marker for a series that is still exposed when the storage returns a new series reference for it. [#19328](prometheus/prometheus#19328)
- \[BUGFIX] Scrape: JSON log formatter correctly format scrape target info. [#19472](prometheus/prometheus#19472)
- \[BUGFIX] TSDB: Fix in-order chunk ID overflow by wrapping head chunk IDs modulo 2^23 so they never collide with the out-of-order flag bit. [#19450](prometheus/prometheus#19450)
- \[BUGFIX] TSDB: Fix out-of-order chunk ID overflow by wrapping `firstOOOChunkID` modulo 2^23 instead of growing unbounded. [#19216](prometheus/prometheus#19216)
- \[BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. [#19326](prometheus/prometheus#19326)
- \[BUGFIX] discovery/aws: Avoid a panic when an ElastiCache ARN is missing its resource ID. [#19333](prometheus/prometheus#19333)
- \[BUGFIX] discovery/aws: Do not panic on MSK clusters whose optional API fields are absent. [#19584](prometheus/prometheus#19584)
- \[BUGFIX] discovery/aws: Do not panic when the EC2 API omits optional instance fields. [#19512](prometheus/prometheus#19512)
- \[BUGFIX] discovery/aws: Don't panic on ECS tasks with absent optional fields. [#19396](prometheus/prometheus#19396)
- \[BUGFIX] discovery/aws: MSK Optional Custom Configuration Fields [#19422](prometheus/prometheus#19422). [#19422](prometheus/prometheus#19422)
- \[BUGFIX] discovery/ionos: Fix panic when the IONOS API omits optional server fields. [#19418](prometheus/prometheus#19418)
- \[BUGFIX] discovery/kubernetes: Populate `__meta_kubernetes_service_loadbalancer_ip` from `status.loadBalancer.ingress`, falling back to deprecated `spec.loadBalancerIP`. [#19404](prometheus/prometheus#19404)
- \[BUGFIX] histogram: Fix Compact moving buckets to wrong indices, and producing negative bucket counts for integer histograms, when more than one span is merged in the same pass. [#19312](prometheus/prometheus#19312)
- \[BUGFIX] promtool: Fixed `tsdb dump` silently dropping native histogram samples. [#18051](prometheus/prometheus#18051)
- \[BUGFIX] scrape: fix data race in `Manager.TargetsDroppedCounts` to avoid miscounting dropped targets. [#19304](prometheus/prometheus#19304)
- \[BUGFIX] scrape: fix nil histogram when native and classic histograms are mixed in one metric family. [#19452](prometheus/prometheus#19452)
- \[BUGFIX] TSDB: Fix WAL and GC log messages to emit human-readable duration strings instead of nanosecond integers. [#19307](prometheus/prometheus#19307)
- \[BUGFIX] TSDB: Avoid WAL corruption after a failed WAL write in Agent mode. [#19700](prometheus/prometheus#19700)
- \[BUGFIX] Agent: Ignore unknown WAL record types, to help users rolling back. [#19814](prometheus/prometheus#19814)
renovate Bot added a commit to sdwilsh/ansible-playbooks that referenced this pull request Sep 26, 2026
##### [\`v3.15.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.15.0)

- \[CHANGE] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. Add tests to prevent regression of the fix made in [#18081](prometheus/prometheus#18081). [#18598](prometheus/prometheus#18598)
- \[CHANGE] PromQL: Do not register a start timestamp reset if the start timestamp hasn't changed between subsequent samples. [#19454](prometheus/prometheus#19454)
- \[CHANGE] Logging: Deprecate `--log.level`; use `runtime.log_level` configuration to supply the default level. [#19511](prometheus/prometheus#19511)
- \[FEATURE] Configuration: Allow changing the process log level through `runtime.log_level` on configuration reload. [#19511](prometheus/prometheus#19511)
- \[FEATURE] Prometheus: Add `--auto-gomemlimit.refresh-interval` flag to periodically re-detect the container or system memory limit and update `GOMEMLIMIT` at runtime. [#18843](prometheus/prometheus#18843)
- \[FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets. [#12024](prometheus/prometheus#12024). [#18091](prometheus/prometheus#18091)
- \[FEATURE] scrape: Implement OM2.0 scrape format. [#18606](prometheus/prometheus#18606)
- \[ENHANCEMENT] Reduce TSDB head CPU utilization when initializing. [#18001](prometheus/prometheus#18001)
- \[ENHANCEMENT] Docker SD: Add labels `__meta_docker_container_image` and `__meta_docker_container_image_id`. [#19386](prometheus/prometheus#19386)
- \[ENHANCEMENT] Mixin: Add a p95/p99 remote-write send-batch latency panel to the remote-write dashboard. [#19500](prometheus/prometheus#19500)
- \[ENHANCEMENT] Mixin: Support native histograms in the remote-write send-batch latency panel. [#19522](prometheus/prometheus#19522)
- \[ENHANCEMENT] PromQL/TSDB: The `--enable-feature=st-storage` flag now automatically enables XOR2 float chunk encoding and ST-capable histogram chunk encoding, so you no longer need to pass `xor2-encoding` and `histograms-st-encoding` alongside it. [#19518](prometheus/prometheus#19518)
- \[ENHANCEMENT] Remote write / Alertmanager: upgrade sigv4 to v0.5.0, adding `session_name` and `tags` fields for STS AssumeRole sessions. The previously undocumented `service_name` field is now also documented. [#19569](prometheus/prometheus#19569)
- \[ENHANCEMENT] Scraping: Support zstd-compressed scrape responses, enabled via feature flag `zstd-scrape`. [#19502](prometheus/prometheus#19502)
- \[ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. `--enable-feature=xor2-encoding` is deprecated; use `storage.tsdb.chunk_encoding.floats: xor2` instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. [#19461](prometheus/prometheus#19461)
- \[ENHANCEMENT] TSDB: add `prometheus_tsdb_head_appenders_created_total` metric. [#19411](prometheus/prometheus#19411)
- \[ENHANCEMENT] Tracing: add more spans to scrapes, API queries and rule evaluations. [#19410](prometheus/prometheus#19410)
- \[ENHANCEMENT] UI: Show the effective configuration for each scrape pool on the Targets and Service Discovery pages. [#19384](prometheus/prometheus#19384)
- \[ENHANCEMENT] scrape: Enable start time synthesis for summary `_count` and `_sum` series in scrape appender v2. [#19323](prometheus/prometheus#19323)
- \[ENHANCEMENT] scrape: stop all pools in parallel for faster shutdowns. [#19295](prometheus/prometheus#19295)
- \[ENHANCEMENT] storage/remote: Add undocumented failed\_request\_logging config field to debug log remote write V2 requests on send errors. [#19249](prometheus/prometheus#19249)
- \[ENHANCEMENT] TSDB: Add fast path for XOR chunk decompression to speed up queries. [#18049](prometheus/prometheus#18049)
- \[ENHANCEMENT] UI: Improve native histogram table formatting and add a background bar indicating the bucket count. [#19332](prometheus/prometheus#19332)
- \[ENHANCEMENT] TSDB: Add `prometheus_tsdb_head_series_pending_commit_underflow_total` to report pending-sample reservation underflows. [#19470](prometheus/prometheus#19470)
- \[PERF] AWS SD: Build RDS cluster labels once per cluster instead of once per instance. [#19504](prometheus/prometheus#19504)
- \[PERF] AWS SD: Describe RDS instances of different clusters concurrently, bounded by `request_concurrency`. [#19506](prometheus/prometheus#19506)
- \[PERF] AWS SD: Describe each ElastiCache resource once per refresh instead of twice. [#19585](prometheus/prometheus#19585)
- \[PERF] Remote read: Avoid cloning labels for sampled reads when no external labels are configured. [#19503](prometheus/prometheus#19503)
- \[PERF] Remote write: Reuse OTLP converter scratch state between requests. [#19388](prometheus/prometheus#19388)
- \[PERF] scrape: conversion from classic to native histograms should only parse start times when enabled. [#19446](prometheus/prometheus#19446)
- \[BUGFIX] PromQL: Fix `info()` enrichment for composite expressions with mixed `@`/offset references or selector-free vector branches, preventing metadata from being evaluated at an unrelated timestamp. [#19387](prometheus/prometheus#19387)
- \[BUGFIX] PromQL: Fix info() enrichment when input series use different subsets of identifying labels. [#19557](prometheus/prometheus#19557)
- \[BUGFIX] PromQL: Preserve metric-name dropping through the info function when delayed name removal is enabled. [#19413](prometheus/prometheus#19413)
- \[BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to. [#19272](prometheus/prometheus#19272)
- \[BUGFIX] TSDB: Keep series with uncommitted samples during selected- and stale-series compaction, including when appenders overlap. [#19470](prometheus/prometheus#19470)
- \[BUGFIX] TSDB: Do not retain head series after a synthetic start-timestamp zero sample is rejected. [#19470](prometheus/prometheus#19470)
- \[BUGFIX] TSDB: Prevent query panics during series eviction after WAL replay. [#19664](prometheus/prometheus#19664)
- \[BUGFIX] TSDB: fix potential deadlock between mmapSeriesChunks and gcSeries. [#19460](prometheus/prometheus#19460)
- \[BUGFIX] TSDB: fix default block reload interval for custom options. [#19368](prometheus/prometheus#19368)
- \[BUGFIX] AWS SD: Do not crash on serverless MSK clusters or MSK clusters without Open Monitoring. [#19194](prometheus/prometheus#19194)
- \[BUGFIX] AWS SD: Do not panic when the ElastiCache API omits optional fields of a serverless cache or cache cluster. [#19435](prometheus/prometheus#19435)
- \[BUGFIX] AWS SD: Reject non-positive `request_concurrency` instead of hanging service discovery indefinitely. [#19524](prometheus/prometheus#19524)
- \[BUGFIX] Discovery/AWS: Avoid a panic when discovering standalone ECS tasks with custom task groups. [#19302](prometheus/prometheus#19302)
- \[BUGFIX] Discovery: Do not panic in AWS Lightsail service discovery when an instance is missing optional fields such as availability zone, blueprint, bundle, name, state or support code. [#19324](prometheus/prometheus#19324)
- \[BUGFIX] Discovery: delete the stale `prometheus_sd_last_update_timestamp_seconds` series for a config that is removed on reload. [#19131](prometheus/prometheus#19131)
- \[BUGFIX] HTTP: Avoid truncating compressed responses when handlers set Content-Length. [#19661](prometheus/prometheus#19661)
- \[BUGFIX] IONOS SD: Do not panic when the API response omits the server, NIC or volume collections, or a server's properties. [#19438](prometheus/prometheus#19438)
- \[BUGFIX] Metadata will not affect the number of Remote Write v2 shards. [#19218](prometheus/prometheus#19218)
- \[BUGFIX] Mixins: Fix label mismatches that prevented PrometheusHAGroupNotIngestingSamples and PrometheusHAGroupCrashlooping from firing. [#19444](prometheus/prometheus#19444)
- \[BUGFIX] Native histograms: `DetectReset` no longer misses a counter reset when a populated bucket behind an empty one disappears, which could make histogram `rate()`/`increase()` undercount. [#19367](prometheus/prometheus#19367)
- \[BUGFIX] Never skip histogram buckets for histogram\_stddev and histogram\_stdvar functions. [#19521](prometheus/prometheus#19521)
- \[BUGFIX] OTLP: Do not abort an entire OTLP payload ingestion if one metric has no datapoints. [#19343](prometheus/prometheus#19343)
- \[BUGFIX] PromQL: Fix FastRegexMatcher false-positive match when a capturing group is directly adjacent to a literal (e.g. `.*\|(foo)\|.*`). [#19516](prometheus/prometheus#19516)
- \[BUGFIX] PromQL: Fix a panic in range selectors using the experimental `anchored` or `smoothed` modifier when the selected series has no samples inside the query window, for example a query evaluated inside a scrape gap. [#19431](prometheus/prometheus#19431)
- \[BUGFIX] PromQL: Fix empty results when a subquery with `@` is used as the matrix argument of a call that is not step-invariant (for example `quantile_over_time(scalar(x), metric[...:...] @ T)`). [#19187](prometheus/prometheus#19187)
- \[BUGFIX] PromQL: Make the "found duplicate series for the match group" many-to-many matching error message deterministic by sorting the two duplicate labels. [#18810](prometheus/prometheus#18810)
- \[BUGFIX] PromQL: Preserve parentheses around duration literals on Expr.String() round-trip. [#19403](prometheus/prometheus#19403)
- \[BUGFIX] PromQL: Reject duration-expression offsets and `@ start()` / `@ end()` before range selectors, matching the existing rejection of literal offsets and `@ <timestamp>`. [#19406](prometheus/prometheus#19406)
- \[BUGFIX] PromQL: Report the position of the histogram argument rather than of a scalar argument in the native histogram NaN observation annotations of `histogram_quantile` and `histogram_fraction`. [#19330](prometheus/prometheus#19330)
- \[BUGFIX] PromQL: info() now applies the @ modifier/offset when evaluating the info series, so info(v @ T) enriches as of T consistently instead of depending on the query start/eval time. [#19266](prometheus/prometheus#19266)
- \[BUGFIX] Rules: Fix a panic when a rule manager created without a logger loads a rule file containing multiple YAML documents. [#19433](prometheus/prometheus#19433)
- \[BUGFIX] Scrape: Do not append a stale marker for a series that is still exposed when the storage returns a new series reference for it. [#19328](prometheus/prometheus#19328)
- \[BUGFIX] Scrape: JSON log formatter correctly format scrape target info. [#19472](prometheus/prometheus#19472)
- \[BUGFIX] TSDB: Fix in-order chunk ID overflow by wrapping head chunk IDs modulo 2^23 so they never collide with the out-of-order flag bit. [#19450](prometheus/prometheus#19450)
- \[BUGFIX] TSDB: Fix out-of-order chunk ID overflow by wrapping `firstOOOChunkID` modulo 2^23 instead of growing unbounded. [#19216](prometheus/prometheus#19216)
- \[BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. [#19326](prometheus/prometheus#19326)
- \[BUGFIX] discovery/aws: Avoid a panic when an ElastiCache ARN is missing its resource ID. [#19333](prometheus/prometheus#19333)
- \[BUGFIX] discovery/aws: Do not panic on MSK clusters whose optional API fields are absent. [#19584](prometheus/prometheus#19584)
- \[BUGFIX] discovery/aws: Do not panic when the EC2 API omits optional instance fields. [#19512](prometheus/prometheus#19512)
- \[BUGFIX] discovery/aws: Don't panic on ECS tasks with absent optional fields. [#19396](prometheus/prometheus#19396)
- \[BUGFIX] discovery/aws: MSK Optional Custom Configuration Fields [#19422](prometheus/prometheus#19422). [#19422](prometheus/prometheus#19422)
- \[BUGFIX] discovery/ionos: Fix panic when the IONOS API omits optional server fields. [#19418](prometheus/prometheus#19418)
- \[BUGFIX] discovery/kubernetes: Populate `__meta_kubernetes_service_loadbalancer_ip` from `status.loadBalancer.ingress`, falling back to deprecated `spec.loadBalancerIP`. [#19404](prometheus/prometheus#19404)
- \[BUGFIX] histogram: Fix Compact moving buckets to wrong indices, and producing negative bucket counts for integer histograms, when more than one span is merged in the same pass. [#19312](prometheus/prometheus#19312)
- \[BUGFIX] promtool: Fixed `tsdb dump` silently dropping native histogram samples. [#18051](prometheus/prometheus#18051)
- \[BUGFIX] scrape: fix data race in `Manager.TargetsDroppedCounts` to avoid miscounting dropped targets. [#19304](prometheus/prometheus#19304)
- \[BUGFIX] scrape: fix nil histogram when native and classic histograms are mixed in one metric family. [#19452](prometheus/prometheus#19452)
- \[BUGFIX] TSDB: Fix WAL and GC log messages to emit human-readable duration strings instead of nanosecond integers. [#19307](prometheus/prometheus#19307)
- \[BUGFIX] TSDB: Avoid WAL corruption after a failed WAL write in Agent mode. [#19700](prometheus/prometheus#19700)
- \[BUGFIX] Agent: Ignore unknown WAL record types, to help users rolling back. [#19814](prometheus/prometheus#19814)
renovate Bot added a commit to sdwilsh/ansible-playbooks that referenced this pull request Sep 26, 2026
##### [\`v3.15.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.15.0)

- \[CHANGE] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. Add tests to prevent regression of the fix made in [#18081](prometheus/prometheus#18081). [#18598](prometheus/prometheus#18598)
- \[CHANGE] PromQL: Do not register a start timestamp reset if the start timestamp hasn't changed between subsequent samples. [#19454](prometheus/prometheus#19454)
- \[CHANGE] Logging: Deprecate `--log.level`; use `runtime.log_level` configuration to supply the default level. [#19511](prometheus/prometheus#19511)
- \[FEATURE] Configuration: Allow changing the process log level through `runtime.log_level` on configuration reload. [#19511](prometheus/prometheus#19511)
- \[FEATURE] Prometheus: Add `--auto-gomemlimit.refresh-interval` flag to periodically re-detect the container or system memory limit and update `GOMEMLIMIT` at runtime. [#18843](prometheus/prometheus#18843)
- \[FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets. [#12024](prometheus/prometheus#12024). [#18091](prometheus/prometheus#18091)
- \[FEATURE] scrape: Implement OM2.0 scrape format. [#18606](prometheus/prometheus#18606)
- \[ENHANCEMENT] Reduce TSDB head CPU utilization when initializing. [#18001](prometheus/prometheus#18001)
- \[ENHANCEMENT] Docker SD: Add labels `__meta_docker_container_image` and `__meta_docker_container_image_id`. [#19386](prometheus/prometheus#19386)
- \[ENHANCEMENT] Mixin: Add a p95/p99 remote-write send-batch latency panel to the remote-write dashboard. [#19500](prometheus/prometheus#19500)
- \[ENHANCEMENT] Mixin: Support native histograms in the remote-write send-batch latency panel. [#19522](prometheus/prometheus#19522)
- \[ENHANCEMENT] PromQL/TSDB: The `--enable-feature=st-storage` flag now automatically enables XOR2 float chunk encoding and ST-capable histogram chunk encoding, so you no longer need to pass `xor2-encoding` and `histograms-st-encoding` alongside it. [#19518](prometheus/prometheus#19518)
- \[ENHANCEMENT] Remote write / Alertmanager: upgrade sigv4 to v0.5.0, adding `session_name` and `tags` fields for STS AssumeRole sessions. The previously undocumented `service_name` field is now also documented. [#19569](prometheus/prometheus#19569)
- \[ENHANCEMENT] Scraping: Support zstd-compressed scrape responses, enabled via feature flag `zstd-scrape`. [#19502](prometheus/prometheus#19502)
- \[ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. `--enable-feature=xor2-encoding` is deprecated; use `storage.tsdb.chunk_encoding.floats: xor2` instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. [#19461](prometheus/prometheus#19461)
- \[ENHANCEMENT] TSDB: add `prometheus_tsdb_head_appenders_created_total` metric. [#19411](prometheus/prometheus#19411)
- \[ENHANCEMENT] Tracing: add more spans to scrapes, API queries and rule evaluations. [#19410](prometheus/prometheus#19410)
- \[ENHANCEMENT] UI: Show the effective configuration for each scrape pool on the Targets and Service Discovery pages. [#19384](prometheus/prometheus#19384)
- \[ENHANCEMENT] scrape: Enable start time synthesis for summary `_count` and `_sum` series in scrape appender v2. [#19323](prometheus/prometheus#19323)
- \[ENHANCEMENT] scrape: stop all pools in parallel for faster shutdowns. [#19295](prometheus/prometheus#19295)
- \[ENHANCEMENT] storage/remote: Add undocumented failed\_request\_logging config field to debug log remote write V2 requests on send errors. [#19249](prometheus/prometheus#19249)
- \[ENHANCEMENT] TSDB: Add fast path for XOR chunk decompression to speed up queries. [#18049](prometheus/prometheus#18049)
- \[ENHANCEMENT] UI: Improve native histogram table formatting and add a background bar indicating the bucket count. [#19332](prometheus/prometheus#19332)
- \[ENHANCEMENT] TSDB: Add `prometheus_tsdb_head_series_pending_commit_underflow_total` to report pending-sample reservation underflows. [#19470](prometheus/prometheus#19470)
- \[PERF] AWS SD: Build RDS cluster labels once per cluster instead of once per instance. [#19504](prometheus/prometheus#19504)
- \[PERF] AWS SD: Describe RDS instances of different clusters concurrently, bounded by `request_concurrency`. [#19506](prometheus/prometheus#19506)
- \[PERF] AWS SD: Describe each ElastiCache resource once per refresh instead of twice. [#19585](prometheus/prometheus#19585)
- \[PERF] Remote read: Avoid cloning labels for sampled reads when no external labels are configured. [#19503](prometheus/prometheus#19503)
- \[PERF] Remote write: Reuse OTLP converter scratch state between requests. [#19388](prometheus/prometheus#19388)
- \[PERF] scrape: conversion from classic to native histograms should only parse start times when enabled. [#19446](prometheus/prometheus#19446)
- \[BUGFIX] PromQL: Fix `info()` enrichment for composite expressions with mixed `@`/offset references or selector-free vector branches, preventing metadata from being evaluated at an unrelated timestamp. [#19387](prometheus/prometheus#19387)
- \[BUGFIX] PromQL: Fix info() enrichment when input series use different subsets of identifying labels. [#19557](prometheus/prometheus#19557)
- \[BUGFIX] PromQL: Preserve metric-name dropping through the info function when delayed name removal is enabled. [#19413](prometheus/prometheus#19413)
- \[BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to. [#19272](prometheus/prometheus#19272)
- \[BUGFIX] TSDB: Keep series with uncommitted samples during selected- and stale-series compaction, including when appenders overlap. [#19470](prometheus/prometheus#19470)
- \[BUGFIX] TSDB: Do not retain head series after a synthetic start-timestamp zero sample is rejected. [#19470](prometheus/prometheus#19470)
- \[BUGFIX] TSDB: Prevent query panics during series eviction after WAL replay. [#19664](prometheus/prometheus#19664)
- \[BUGFIX] TSDB: fix potential deadlock between mmapSeriesChunks and gcSeries. [#19460](prometheus/prometheus#19460)
- \[BUGFIX] TSDB: fix default block reload interval for custom options. [#19368](prometheus/prometheus#19368)
- \[BUGFIX] AWS SD: Do not crash on serverless MSK clusters or MSK clusters without Open Monitoring. [#19194](prometheus/prometheus#19194)
- \[BUGFIX] AWS SD: Do not panic when the ElastiCache API omits optional fields of a serverless cache or cache cluster. [#19435](prometheus/prometheus#19435)
- \[BUGFIX] AWS SD: Reject non-positive `request_concurrency` instead of hanging service discovery indefinitely. [#19524](prometheus/prometheus#19524)
- \[BUGFIX] Discovery/AWS: Avoid a panic when discovering standalone ECS tasks with custom task groups. [#19302](prometheus/prometheus#19302)
- \[BUGFIX] Discovery: Do not panic in AWS Lightsail service discovery when an instance is missing optional fields such as availability zone, blueprint, bundle, name, state or support code. [#19324](prometheus/prometheus#19324)
- \[BUGFIX] Discovery: delete the stale `prometheus_sd_last_update_timestamp_seconds` series for a config that is removed on reload. [#19131](prometheus/prometheus#19131)
- \[BUGFIX] HTTP: Avoid truncating compressed responses when handlers set Content-Length. [#19661](prometheus/prometheus#19661)
- \[BUGFIX] IONOS SD: Do not panic when the API response omits the server, NIC or volume collections, or a server's properties. [#19438](prometheus/prometheus#19438)
- \[BUGFIX] Metadata will not affect the number of Remote Write v2 shards. [#19218](prometheus/prometheus#19218)
- \[BUGFIX] Mixins: Fix label mismatches that prevented PrometheusHAGroupNotIngestingSamples and PrometheusHAGroupCrashlooping from firing. [#19444](prometheus/prometheus#19444)
- \[BUGFIX] Native histograms: `DetectReset` no longer misses a counter reset when a populated bucket behind an empty one disappears, which could make histogram `rate()`/`increase()` undercount. [#19367](prometheus/prometheus#19367)
- \[BUGFIX] Never skip histogram buckets for histogram\_stddev and histogram\_stdvar functions. [#19521](prometheus/prometheus#19521)
- \[BUGFIX] OTLP: Do not abort an entire OTLP payload ingestion if one metric has no datapoints. [#19343](prometheus/prometheus#19343)
- \[BUGFIX] PromQL: Fix FastRegexMatcher false-positive match when a capturing group is directly adjacent to a literal (e.g. `.*\|(foo)\|.*`). [#19516](prometheus/prometheus#19516)
- \[BUGFIX] PromQL: Fix a panic in range selectors using the experimental `anchored` or `smoothed` modifier when the selected series has no samples inside the query window, for example a query evaluated inside a scrape gap. [#19431](prometheus/prometheus#19431)
- \[BUGFIX] PromQL: Fix empty results when a subquery with `@` is used as the matrix argument of a call that is not step-invariant (for example `quantile_over_time(scalar(x), metric[...:...] @ T)`). [#19187](prometheus/prometheus#19187)
- \[BUGFIX] PromQL: Make the "found duplicate series for the match group" many-to-many matching error message deterministic by sorting the two duplicate labels. [#18810](prometheus/prometheus#18810)
- \[BUGFIX] PromQL: Preserve parentheses around duration literals on Expr.String() round-trip. [#19403](prometheus/prometheus#19403)
- \[BUGFIX] PromQL: Reject duration-expression offsets and `@ start()` / `@ end()` before range selectors, matching the existing rejection of literal offsets and `@ <timestamp>`. [#19406](prometheus/prometheus#19406)
- \[BUGFIX] PromQL: Report the position of the histogram argument rather than of a scalar argument in the native histogram NaN observation annotations of `histogram_quantile` and `histogram_fraction`. [#19330](prometheus/prometheus#19330)
- \[BUGFIX] PromQL: info() now applies the @ modifier/offset when evaluating the info series, so info(v @ T) enriches as of T consistently instead of depending on the query start/eval time. [#19266](prometheus/prometheus#19266)
- \[BUGFIX] Rules: Fix a panic when a rule manager created without a logger loads a rule file containing multiple YAML documents. [#19433](prometheus/prometheus#19433)
- \[BUGFIX] Scrape: Do not append a stale marker for a series that is still exposed when the storage returns a new series reference for it. [#19328](prometheus/prometheus#19328)
- \[BUGFIX] Scrape: JSON log formatter correctly format scrape target info. [#19472](prometheus/prometheus#19472)
- \[BUGFIX] TSDB: Fix in-order chunk ID overflow by wrapping head chunk IDs modulo 2^23 so they never collide with the out-of-order flag bit. [#19450](prometheus/prometheus#19450)
- \[BUGFIX] TSDB: Fix out-of-order chunk ID overflow by wrapping `firstOOOChunkID` modulo 2^23 instead of growing unbounded. [#19216](prometheus/prometheus#19216)
- \[BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. [#19326](prometheus/prometheus#19326)
- \[BUGFIX] discovery/aws: Avoid a panic when an ElastiCache ARN is missing its resource ID. [#19333](prometheus/prometheus#19333)
- \[BUGFIX] discovery/aws: Do not panic on MSK clusters whose optional API fields are absent. [#19584](prometheus/prometheus#19584)
- \[BUGFIX] discovery/aws: Do not panic when the EC2 API omits optional instance fields. [#19512](prometheus/prometheus#19512)
- \[BUGFIX] discovery/aws: Don't panic on ECS tasks with absent optional fields. [#19396](prometheus/prometheus#19396)
- \[BUGFIX] discovery/aws: MSK Optional Custom Configuration Fields [#19422](prometheus/prometheus#19422). [#19422](prometheus/prometheus#19422)
- \[BUGFIX] discovery/ionos: Fix panic when the IONOS API omits optional server fields. [#19418](prometheus/prometheus#19418)
- \[BUGFIX] discovery/kubernetes: Populate `__meta_kubernetes_service_loadbalancer_ip` from `status.loadBalancer.ingress`, falling back to deprecated `spec.loadBalancerIP`. [#19404](prometheus/prometheus#19404)
- \[BUGFIX] histogram: Fix Compact moving buckets to wrong indices, and producing negative bucket counts for integer histograms, when more than one span is merged in the same pass. [#19312](prometheus/prometheus#19312)
- \[BUGFIX] promtool: Fixed `tsdb dump` silently dropping native histogram samples. [#18051](prometheus/prometheus#18051)
- \[BUGFIX] scrape: fix data race in `Manager.TargetsDroppedCounts` to avoid miscounting dropped targets. [#19304](prometheus/prometheus#19304)
- \[BUGFIX] scrape: fix nil histogram when native and classic histograms are mixed in one metric family. [#19452](prometheus/prometheus#19452)
- \[BUGFIX] TSDB: Fix WAL and GC log messages to emit human-readable duration strings instead of nanosecond integers. [#19307](prometheus/prometheus#19307)
- \[BUGFIX] TSDB: Avoid WAL corruption after a failed WAL write in Agent mode. [#19700](prometheus/prometheus#19700)
- \[BUGFIX] Agent: Ignore unknown WAL record types, to help users rolling back. [#19814](prometheus/prometheus#19814)
hbjydev pushed a commit to hbjydev/phoebe that referenced this pull request Sep 28, 2026
…→ v3.15.0) (#769)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [quay.io/prometheus/prometheus](https://github.com/prometheus/prometheus) | minor | `v3.14.0` → `v3.15.0` |

---

### Release Notes

<details>
<summary>prometheus/prometheus (quay.io/prometheus/prometheus)</summary>

### [`v3.15.0`](https://github.com/prometheus/prometheus/releases/tag/v3.15.0): 3.15.0 / 2026-09-24

[Compare Source](prometheus/prometheus@v3.14.0...v3.15.0)

- \[CHANGE] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. Add tests to prevent regression of the fix made in [#&#8203;18081](prometheus/prometheus#18081). [#&#8203;18598](prometheus/prometheus#18598)
- \[CHANGE] PromQL: Do not register a start timestamp reset if the start timestamp hasn't changed between subsequent samples. [#&#8203;19454](prometheus/prometheus#19454)
- \[CHANGE] Logging: Deprecate `--log.level`; use `runtime.log_level` configuration to supply the default level. [#&#8203;19511](prometheus/prometheus#19511)
- \[FEATURE] Configuration: Allow changing the process log level through `runtime.log_level` on configuration reload. [#&#8203;19511](prometheus/prometheus#19511)
- \[FEATURE] Prometheus: Add `--auto-gomemlimit.refresh-interval` flag to periodically re-detect the container or system memory limit and update `GOMEMLIMIT` at runtime. [#&#8203;18843](prometheus/prometheus#18843)
- \[FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets. [#&#8203;12024](prometheus/prometheus#12024). [#&#8203;18091](prometheus/prometheus#18091)
- \[FEATURE] scrape: Implement OM2.0 scrape format. [#&#8203;18606](prometheus/prometheus#18606)
- \[ENHANCEMENT] Reduce TSDB head CPU utilization when initializing. [#&#8203;18001](prometheus/prometheus#18001)
- \[ENHANCEMENT] Docker SD: Add labels `__meta_docker_container_image` and `__meta_docker_container_image_id`. [#&#8203;19386](prometheus/prometheus#19386)
- \[ENHANCEMENT] Mixin: Add a p95/p99 remote-write send-batch latency panel to the remote-write dashboard. [#&#8203;19500](prometheus/prometheus#19500)
- \[ENHANCEMENT] Mixin: Support native histograms in the remote-write send-batch latency panel. [#&#8203;19522](prometheus/prometheus#19522)
- \[ENHANCEMENT] PromQL/TSDB: The `--enable-feature=st-storage` flag now automatically enables XOR2 float chunk encoding and ST-capable histogram chunk encoding, so you no longer need to pass `xor2-encoding` and `histograms-st-encoding` alongside it. [#&#8203;19518](prometheus/prometheus#19518)
- \[ENHANCEMENT] Remote write / Alertmanager: upgrade sigv4 to v0.5.0, adding `session_name` and `tags` fields for STS AssumeRole sessions. The previously undocumented `service_name` field is now also documented. [#&#8203;19569](prometheus/prometheus#19569)
- \[ENHANCEMENT] Scraping: Support zstd-compressed scrape responses, enabled via feature flag `zstd-scrape`. [#&#8203;19502](prometheus/prometheus#19502)
- \[ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. `--enable-feature=xor2-encoding` is deprecated; use `storage.tsdb.chunk_encoding.floats: xor2` instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. [#&#8203;19461](prometheus/prometheus#19461)
- \[ENHANCEMENT] TSDB: add `prometheus_tsdb_head_appenders_created_total` metric. [#&#8203;19411](prometheus/prometheus#19411)
- \[ENHANCEMENT] Tracing: add more spans to scrapes, API queries and rule evaluations. [#&#8203;19410](prometheus/prometheus#19410)
- \[ENHANCEMENT] UI: Show the effective configuration for each scrape pool on the Targets and Service Discovery pages. [#&#8203;19384](prometheus/prometheus#19384)
- \[ENHANCEMENT] scrape: Enable start time synthesis for summary `_count` and `_sum` series in scrape appender v2. [#&#8203;19323](prometheus/prometheus#19323)
- \[ENHANCEMENT] scrape: stop all pools in parallel for faster shutdowns. [#&#8203;19295](prometheus/prometheus#19295)
- \[ENHANCEMENT] storage/remote: Add undocumented failed\_request\_logging config field to debug log remote write V2 requests on send errors. [#&#8203;19249](prometheus/prometheus#19249)
- \[ENHANCEMENT] TSDB: Add fast path for XOR chunk decompression to speed up queries. [#&#8203;18049](prometheus/prometheus#18049)
- \[ENHANCEMENT] UI: Improve native histogram table formatting and add a background bar indicating the bucket count. [#&#8203;19332](prometheus/prometheus#19332)
- \[ENHANCEMENT] TSDB: Add `prometheus_tsdb_head_series_pending_commit_underflow_total` to report pending-sample reservation underflows. [#&#8203;19470](prometheus/prometheus#19470)
- \[PERF] AWS SD: Build RDS cluster labels once per cluster instead of once per instance. [#&#8203;19504](prometheus/prometheus#19504)
- \[PERF] AWS SD: Describe RDS instances of different clusters concurrently, bounded by `request_concurrency`. [#&#8203;19506](prometheus/prometheus#19506)
- \[PERF] AWS SD: Describe each ElastiCache resource once per refresh instead of twice. [#&#8203;19585](prometheus/prometheus#19585)
- \[PERF] Remote read: Avoid cloning labels for sampled reads when no external labels are configured. [#&#8203;19503](prometheus/prometheus#19503)
- \[PERF] Remote write: Reuse OTLP converter scratch state between requests. [#&#8203;19388](prometheus/prometheus#19388)
- \[PERF] scrape: conversion from classic to native histograms should only parse start times when enabled. [#&#8203;19446](prometheus/prometheus#19446)
- \[BUGFIX] PromQL: Fix `info()` enrichment for composite expressions with mixed `@`/offset references or selector-free vector branches, preventing metadata from being evaluated at an unrelated timestamp. [#&#8203;19387](prometheus/prometheus#19387)
- \[BUGFIX] PromQL: Fix info() enrichment when input series use different subsets of identifying labels. [#&#8203;19557](prometheus/prometheus#19557)
- \[BUGFIX] PromQL: Preserve metric-name dropping through the info function when delayed name removal is enabled. [#&#8203;19413](prometheus/prometheus#19413)
- \[BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to. [#&#8203;19272](prometheus/prometheus#19272)
- \[BUGFIX] TSDB: Keep series with uncommitted samples during selected- and stale-series compaction, including when appenders overlap. [#&#8203;19470](prometheus/prometheus#19470)
- \[BUGFIX] TSDB: Do not retain head series after a synthetic start-timestamp zero sample is rejected. [#&#8203;19470](prometheus/prometheus#19470)
- \[BUGFIX] TSDB: Prevent query panics during series eviction after WAL replay. [#&#8203;19664](prometheus/prometheus#19664)
- \[BUGFIX] TSDB: fix potential deadlock between mmapSeriesChunks and gcSeries. [#&#8203;19460](prometheus/prometheus#19460)
- \[BUGFIX] TSDB: fix default block reload interval for custom options. [#&#8203;19368](prometheus/prometheus#19368)
- \[BUGFIX] AWS SD: Do not crash on serverless MSK clusters or MSK clusters without Open Monitoring. [#&#8203;19194](prometheus/prometheus#19194)
- \[BUGFIX] AWS SD: Do not panic when the ElastiCache API omits optional fields of a serverless cache or cache cluster. [#&#8203;19435](prometheus/prometheus#19435)
- \[BUGFIX] AWS SD: Reject non-positive `request_concurrency` instead of hanging service discovery indefinitely. [#&#8203;19524](prometheus/prometheus#19524)
- \[BUGFIX] Discovery/AWS: Avoid a panic when discovering standalone ECS tasks with custom task groups. [#&#8203;19302](prometheus/prometheus#19302)
- \[BUGFIX] Discovery: Do not panic in AWS Lightsail service discovery when an instance is missing optional fields such as availability zone, blueprint, bundle, name, state or support code. [#&#8203;19324](prometheus/prometheus#19324)
- \[BUGFIX] Discovery: delete the stale `prometheus_sd_last_update_timestamp_seconds` series for a config that is removed on reload. [#&#8203;19131](prometheus/prometheus#19131)
- \[BUGFIX] HTTP: Avoid truncating compressed responses when handlers set Content-Length. [#&#8203;19661](prometheus/prometheus#19661)
- \[BUGFIX] IONOS SD: Do not panic when the API response omits the server, NIC or volume collections, or a server's properties. [#&#8203;19438](prometheus/prometheus#19438)
- \[BUGFIX] Metadata will not affect the number of Remote Write v2 shards. [#&#8203;19218](prometheus/prometheus#19218)
- \[BUGFIX] Mixins: Fix label mismatches that prevented PrometheusHAGroupNotIngestingSamples and PrometheusHAGroupCrashlooping from firing. [#&#8203;19444](prometheus/prometheus#19444)
- \[BUGFIX] Native histograms: `DetectReset` no longer misses a counter reset when a populated bucket behind an empty one disappears, which could make histogram `rate()`/`increase()` undercount. [#&#8203;19367](prometheus/prometheus#19367)
- \[BUGFIX] Never skip histogram buckets for histogram\_stddev and histogram\_stdvar functions. [#&#8203;19521](prometheus/prometheus#19521)
- \[BUGFIX] OTLP: Do not abort an entire OTLP payload ingestion if one metric has no datapoints. [#&#8203;19343](prometheus/prometheus#19343)
- \[BUGFIX] PromQL: Fix FastRegexMatcher false-positive match when a capturing group is directly adjacent to a literal (e.g. `.*\|(foo)\|.*`). [#&#8203;19516](prometheus/prometheus#19516)
- \[BUGFIX] PromQL: Fix a panic in range selectors using the experimental `anchored` or `smoothed` modifier when the selected series has no samples inside the query window, for example a query evaluated inside a scrape gap. [#&#8203;19431](prometheus/prometheus#19431)
- \[BUGFIX] PromQL: Fix empty results when a subquery with `@` is used as the matrix argument of a call that is not step-invariant (for example `quantile_over_time(scalar(x), metric[...:...] @ T)`). [#&#8203;19187](prometheus/prometheus#19187)
- \[BUGFIX] PromQL: Make the "found duplicate series for the match group" many-to-many matching error message deterministic by sorting the two duplicate labels. [#&#8203;18810](prometheus/prometheus#18810)
- \[BUGFIX] PromQL: Preserve parentheses around duration literals on Expr.String() round-trip. [#&#8203;19403](prometheus/prometheus#19403)
- \[BUGFIX] PromQL: Reject duration-expression offsets and `@ start()` / `@ end()` before range selectors, matching the existing rejection of literal offsets and `@ <timestamp>`. [#&#8203;19406](prometheus/prometheus#19406)
- \[BUGFIX] PromQL: Report the position of the histogram argument rather than of a scalar argument in the native histogram NaN observation annotations of `histogram_quantile` and `histogram_fraction`. [#&#8203;19330](prometheus/prometheus#19330)
- \[BUGFIX] PromQL: info() now applies the @&#8203; modifier/offset when evaluating the info series, so info(v @&#8203; T) enriches as of T consistently instead of depending on the query start/eval time. [#&#8203;19266](prometheus/prometheus#19266)
- \[BUGFIX] Rules: Fix a panic when a rule manager created without a logger loads a rule file containing multiple YAML documents. [#&#8203;19433](prometheus/prometheus#19433)
- \[BUGFIX] Scrape: Do not append a stale marker for a series that is still exposed when the storage returns a new series reference for it. [#&#8203;19328](prometheus/prometheus#19328)
- \[BUGFIX] Scrape: JSON log formatter correctly format scrape target info. [#&#8203;19472](prometheus/prometheus#19472)
- \[BUGFIX] TSDB: Fix in-order chunk ID overflow by wrapping head chunk IDs modulo 2^23 so they never collide with the out-of-order flag bit. [#&#8203;19450](prometheus/prometheus#19450)
- \[BUGFIX] TSDB: Fix out-of-order chunk ID overflow by wrapping `firstOOOChunkID` modulo 2^23 instead of growing unbounded. [#&#8203;19216](prometheus/prometheus#19216)
- \[BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. [#&#8203;19326](prometheus/prometheus#19326)
- \[BUGFIX] discovery/aws: Avoid a panic when an ElastiCache ARN is missing its resource ID. [#&#8203;19333](prometheus/prometheus#19333)
- \[BUGFIX] discovery/aws: Do not panic on MSK clusters whose optional API fields are absent. [#&#8203;19584](prometheus/prometheus#19584)
- \[BUGFIX] discovery/aws: Do not panic when the EC2 API omits optional instance fields. [#&#8203;19512](prometheus/prometheus#19512)
- \[BUGFIX] discovery/aws: Don't panic on ECS tasks with absent optional fields. [#&#8203;19396](prometheus/prometheus#19396)
- \[BUGFIX] discovery/aws: MSK Optional Custom Configuration Fields [#&#8203;19422](prometheus/prometheus#19422). [#&#8203;19422](prometheus/prometheus#19422)
- \[BUGFIX] discovery/ionos: Fix panic when the IONOS API omits optional server fields. [#&#8203;19418](prometheus/prometheus#19418)
- \[BUGFIX] discovery/kubernetes: Populate `__meta_kubernetes_service_loadbalancer_ip` from `status.loadBalancer.ingress`, falling back to deprecated `spec.loadBalancerIP`. [#&#8203;19404](prometheus/prometheus#19404)
- \[BUGFIX] histogram: Fix Compact moving buckets to wrong indices, and producing negative bucket counts for integer histograms, when more than one span is merged in the same pass. [#&#8203;19312](prometheus/prometheus#19312)
- \[BUGFIX] promtool: Fixed `tsdb dump` silently dropping native histogram samples. [#&#8203;18051](prometheus/prometheus#18051)
- \[BUGFIX] scrape: fix data race in `Manager.TargetsDroppedCounts` to avoid miscounting dropped targets. [#&#8203;19304](prometheus/prometheus#19304)
- \[BUGFIX] scrape: fix nil histogram when native and classic histograms are mixed in one metric family. [#&#8203;19452](prometheus/prometheus#19452)
- \[BUGFIX] TSDB: Fix WAL and GC log messages to emit human-readable duration strings instead of nanosecond integers. [#&#8203;19307](prometheus/prometheus#19307)
- \[BUGFIX] TSDB: Avoid WAL corruption after a failed WAL write in Agent mode. [#&#8203;19700](prometheus/prometheus#19700)
- \[BUGFIX] Agent: Ignore unknown WAL record types, to help users rolling back. [#&#8203;19814](prometheus/prometheus#19814)

</details>

---

### Configuration

📅 **Schedule**: (in timezone Europe/London)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xMTUuMSIsInVwZGF0ZWRJblZlciI6IjQ0LjExNS4xMyIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsicmVub3ZhdGUvY29udGFpbmVyIiwidHlwZS9taW5vciJdfQ==-->

Reviewed-on: https://git.hayden.moe/hayden/phoebe/pulls/769
eleboucher pushed a commit to eleboucher/homelab that referenced this pull request Oct 3, 2026
…➔ v3.15.0) (#2199)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [quay.io/prometheus/prometheus](https://github.com/prometheus/prometheus) | minor | `v3.14.0` → `v3.15.0` |

---

> ⚠️ **Warning**
>
> Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/567) for more information.

---

### Release Notes

<details>
<summary>prometheus/prometheus (quay.io/prometheus/prometheus)</summary>

### [`v3.15.0`](https://github.com/prometheus/prometheus/releases/tag/v3.15.0): 3.15.0 / 2026-09-24

[Compare Source](prometheus/prometheus@v3.14.0...v3.15.0)

- \[CHANGE] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. Add tests to prevent regression of the fix made in [#&#8203;18081](prometheus/prometheus#18081). [#&#8203;18598](prometheus/prometheus#18598)
- \[CHANGE] PromQL: Do not register a start timestamp reset if the start timestamp hasn't changed between subsequent samples. [#&#8203;19454](prometheus/prometheus#19454)
- \[CHANGE] Logging: Deprecate `--log.level`; use `runtime.log_level` configuration to supply the default level. [#&#8203;19511](prometheus/prometheus#19511)
- \[FEATURE] Configuration: Allow changing the process log level through `runtime.log_level` on configuration reload. [#&#8203;19511](prometheus/prometheus#19511)
- \[FEATURE] Prometheus: Add `--auto-gomemlimit.refresh-interval` flag to periodically re-detect the container or system memory limit and update `GOMEMLIMIT` at runtime. [#&#8203;18843](prometheus/prometheus#18843)
- \[FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets. [#&#8203;12024](prometheus/prometheus#12024). [#&#8203;18091](prometheus/prometheus#18091)
- \[FEATURE] scrape: Implement OM2.0 scrape format. [#&#8203;18606](prometheus/prometheus#18606)
- \[ENHANCEMENT] Reduce TSDB head CPU utilization when initializing. [#&#8203;18001](prometheus/prometheus#18001)
- \[ENHANCEMENT] Docker SD: Add labels `__meta_docker_container_image` and `__meta_docker_container_image_id`. [#&#8203;19386](prometheus/prometheus#19386)
- \[ENHANCEMENT] Mixin: Add a p95/p99 remote-write send-batch latency panel to the remote-write dashboard. [#&#8203;19500](prometheus/prometheus#19500)
- \[ENHANCEMENT] Mixin: Support native histograms in the remote-write send-batch latency panel. [#&#8203;19522](prometheus/prometheus#19522)
- \[ENHANCEMENT] PromQL/TSDB: The `--enable-feature=st-storage` flag now automatically enables XOR2 float chunk encoding and ST-capable histogram chunk encoding, so you no longer need to pass `xor2-encoding` and `histograms-st-encoding` alongside it. [#&#8203;19518](prometheus/prometheus#19518)
- \[ENHANCEMENT] Remote write / Alertmanager: upgrade sigv4 to v0.5.0, adding `session_name` and `tags` fields for STS AssumeRole sessions. The previously undocumented `service_name` field is now also documented. [#&#8203;19569](prometheus/prometheus#19569)
- \[ENHANCEMENT] Scraping: Support zstd-compressed scrape responses, enabled via feature flag `zstd-scrape`. [#&#8203;19502](prometheus/prometheus#19502)
- \[ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. `--enable-feature=xor2-encoding` is deprecated; use `storage.tsdb.chunk_encoding.floats: xor2` instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. [#&#8203;19461](prometheus/prometheus#19461)
- \[ENHANCEMENT] TSDB: add `prometheus_tsdb_head_appenders_created_total` metric. [#&#8203;19411](prometheus/prometheus#19411)
- \[ENHANCEMENT] Tracing: add more spans to scrapes, API queries and rule evaluations. [#&#8203;19410](prometheus/prometheus#19410)
- \[ENHANCEMENT] UI: Show the effective configuration for each scrape pool on the Targets and Service Discovery pages. [#&#8203;19384](prometheus/prometheus#19384)
- \[ENHANCEMENT] scrape: Enable start time synthesis for summary `_count` and `_sum` series in scrape appender v2. [#&#8203;19323](prometheus/prometheus#19323)
- \[ENHANCEMENT] scrape: stop all pools in parallel for faster shutdowns. [#&#8203;19295](prometheus/prometheus#19295)
- \[ENHANCEMENT] storage/remote: Add undocumented failed\_request\_logging config field to debug log remote write V2 requests on send errors. [#&#8203;19249](prometheus/prometheus#19249)
- \[ENHANCEMENT] TSDB: Add fast path for XOR chunk decompression to speed up queries. [#&#8203;18049](prometheus/prometheus#18049)
- \[ENHANCEMENT] UI: Improve native histogram table formatting and add a background bar indicating the bucket count. [#&#8203;19332](prometheus/prometheus#19332)
- \[ENHANCEMENT] TSDB: Add `prometheus_tsdb_head_series_pending_commit_underflow_total` to report pending-sample reservation underflows. [#&#8203;19470](prometheus/prometheus#19470)
- \[PERF] AWS SD: Build RDS cluster labels once per cluster instead of once per instance. [#&#8203;19504](prometheus/prometheus#19504)
- \[PERF] AWS SD: Describe RDS instances of different clusters concurrently, bounded by `request_concurrency`. [#&#8203;19506](prometheus/prometheus#19506)
- \[PERF] AWS SD: Describe each ElastiCache resource once per refresh instead of twice. [#&#8203;19585](prometheus/prometheus#19585)
- \[PERF] Remote read: Avoid cloning labels for sampled reads when no external labels are configured. [#&#8203;19503](prometheus/prometheus#19503)
- \[PERF] Remote write: Reuse OTLP converter scratch state between requests. [#&#8203;19388](prometheus/prometheus#19388)
- \[PERF] scrape: conversion from classic to native histograms should only parse start times when enabled. [#&#8203;19446](prometheus/prometheus#19446)
- \[BUGFIX] PromQL: Fix `info()` enrichment for composite expressions with mixed `@`/offset references or selector-free vector branches, preventing metadata from being evaluated at an unrelated timestamp. [#&#8203;19387](prometheus/prometheus#19387)
- \[BUGFIX] PromQL: Fix info() enrichment when input series use different subsets of identifying labels. [#&#8203;19557](prometheus/prometheus#19557)
- \[BUGFIX] PromQL: Preserve metric-name dropping through the info function when delayed name removal is enabled. [#&#8203;19413](prometheus/prometheus#19413)
- \[BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to. [#&#8203;19272](prometheus/prometheus#19272)
- \[BUGFIX] TSDB: Keep series with uncommitted samples during selected- and stale-series compaction, including when appenders overlap. [#&#8203;19470](prometheus/prometheus#19470)
- \[BUGFIX] TSDB: Do not retain head series after a synthetic start-timestamp zero sample is rejected. [#&#8203;19470](prometheus/prometheus#19470)
- \[BUGFIX] TSDB: Prevent query panics during series eviction after WAL replay. [#&#8203;19664](prometheus/prometheus#19664)
- \[BUGFIX] TSDB: fix potential deadlock between mmapSeriesChunks and gcSeries. [#&#8203;19460](prometheus/prometheus#19460)
- \[BUGFIX] TSDB: fix default block reload interval for custom options. [#&#8203;19368](prometheus/prometheus#19368)
- \[BUGFIX] AWS SD: Do not crash on serverless MSK clusters or MSK clusters without Open Monitoring. [#&#8203;19194](prometheus/prometheus#19194)
- \[BUGFIX] AWS SD: Do not panic when the ElastiCache API omits optional fields of a serverless cache or cache cluster. [#&#8203;19435](prometheus/prometheus#19435)
- \[BUGFIX] AWS SD: Reject non-positive `request_concurrency` instead of hanging service discovery indefinitely. [#&#8203;19524](prometheus/prometheus#19524)
- \[BUGFIX] Discovery/AWS: Avoid a panic when discovering standalone ECS tasks with custom task groups. [#&#8203;19302](prometheus/prometheus#19302)
- \[BUGFIX] Discovery: Do not panic in AWS Lightsail service discovery when an instance is missing optional fields such as availability zone, blueprint, bundle, name, state or support code. [#&#8203;19324](prometheus/prometheus#19324)
- \[BUGFIX] Discovery: delete the stale `prometheus_sd_last_update_timestamp_seconds` series for a config that is removed on reload. [#&#8203;19131](prometheus/prometheus#19131)
- \[BUGFIX] HTTP: Avoid truncating compressed responses when handlers set Content-Length. [#&#8203;19661](prometheus/prometheus#19661)
- \[BUGFIX] IONOS SD: Do not panic when the API response omits the server, NIC or volume collections, or a server's properties. [#&#8203;19438](prometheus/prometheus#19438)
- \[BUGFIX] Metadata will not affect the number of Remote Write v2 shards. [#&#8203;19218](prometheus/prometheus#19218)
- \[BUGFIX] Mixins: Fix label mismatches that prevented PrometheusHAGroupNotIngestingSamples and PrometheusHAGroupCrashlooping from firing. [#&#8203;19444](prometheus/prometheus#19444)
- \[BUGFIX] Native histograms: `DetectReset` no longer misses a counter reset when a populated bucket behind an empty one disappears, which could make histogram `rate()`/`increase()` undercount. [#&#8203;19367](prometheus/prometheus#19367)
- \[BUGFIX] Never skip histogram buckets for histogram\_stddev and histogram\_stdvar functions. [#&#8203;19521](prometheus/prometheus#19521)
- \[BUGFIX] OTLP: Do not abort an entire OTLP payload ingestion if one metric has no datapoints. [#&#8203;19343](prometheus/prometheus#19343)
- \[BUGFIX] PromQL: Fix FastRegexMatcher false-positive match when a capturing group is directly adjacent to a literal (e.g. `.*\|(foo)\|.*`). [#&#8203;19516](prometheus/prometheus#19516)
- \[BUGFIX] PromQL: Fix a panic in range selectors using the experimental `anchored` or `smoothed` modifier when the selected series has no samples inside the query window, for example a query evaluated inside a scrape gap. [#&#8203;19431](prometheus/prometheus#19431)
- \[BUGFIX] PromQL: Fix empty results when a subquery with `@` is used as the matrix argument of a call that is not step-invariant (for example `quantile_over_time(scalar(x), metric[...:...] @ T)`). [#&#8203;19187](prometheus/prometheus#19187)
- \[BUGFIX] PromQL: Make the "found duplicate series for the match group" many-to-many matching error message deterministic by sorting the two duplicate labels. [#&#8203;18810](prometheus/prometheus#18810)
- \[BUGFIX] PromQL: Preserve parentheses around duration literals on Expr.String() round-trip. [#&#8203;19403](prometheus/prometheus#19403)
- \[BUGFIX] PromQL: Reject duration-expression offsets and `@ start()` / `@ end()` before range selectors, matching the existing rejection of literal offsets and `@ <timestamp>`. [#&#8203;19406](prometheus/prometheus#19406)
- \[BUGFIX] PromQL: Report the position of the histogram argument rather than of a scalar argument in the native histogram NaN observation annotations of `histogram_quantile` and `histogram_fraction`. [#&#8203;19330](prometheus/prometheus#19330)
- \[BUGFIX] PromQL: info() now applies the @&#8203; modifier/offset when evaluating the info series, so info(v @&#8203; T) enriches as of T consistently instead of depending on the query start/eval time. [#&#8203;19266](prometheus/prometheus#19266)
- \[BUGFIX] Rules: Fix a panic when a rule manager created without a logger loads a rule file containing multiple YAML documents. [#&#8203;19433](prometheus/prometheus#19433)
- \[BUGFIX] Scrape: Do not append a stale marker for a series that is still exposed when the storage returns a new series reference for it. [#&#8203;19328](prometheus/prometheus#19328)
- \[BUGFIX] Scrape: JSON log formatter correctly format scrape target info. [#&#8203;19472](prometheus/prometheus#19472)
- \[BUGFIX] TSDB: Fix in-order chunk ID overflow by wrapping head chunk IDs modulo 2^23 so they never collide with the out-of-order flag bit. [#&#8203;19450](prometheus/prometheus#19450)
- \[BUGFIX] TSDB: Fix out-of-order chunk ID overflow by wrapping `firstOOOChunkID` modulo 2^23 instead of growing unbounded. [#&#8203;19216](prometheus/prometheus#19216)
- \[BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. [#&#8203;19326](prometheus/prometheus#19326)
- \[BUGFIX] discovery/aws: Avoid a panic when an ElastiCache ARN is missing its resource ID. [#&#8203;19333](prometheus/prometheus#19333)
- \[BUGFIX] discovery/aws: Do not panic on MSK clusters whose optional API fields are absent. [#&#8203;19584](prometheus/prometheus#19584)
- \[BUGFIX] discovery/aws: Do not panic when the EC2 API omits optional instance fields. [#&#8203;19512](prometheus/prometheus#19512)
- \[BUGFIX] discovery/aws: Don't panic on ECS tasks with absent optional fields. [#&#8203;19396](prometheus/prometheus#19396)
- \[BUGFIX] discovery/aws: MSK Optional Custom Configuration Fields [#&#8203;19422](prometheus/prometheus#19422). [#&#8203;19422](prometheus/prometheus#19422)
- \[BUGFIX] discovery/ionos: Fix panic when the IONOS API omits optional server fields. [#&#8203;19418](prometheus/prometheus#19418)
- \[BUGFIX] discovery/kubernetes: Populate `__meta_kubernetes_service_loadbalancer_ip` from `status.loadBalancer.ingress`, falling back to deprecated `spec.loadBalancerIP`. [#&#8203;19404](prometheus/prometheus#19404)
- \[BUGFIX] histogram: Fix Compact moving buckets to wrong indices, and producing negative bucket counts for integer histograms, when more than one span is merged in the same pass. [#&#8203;19312](prometheus/prometheus#19312)
- \[BUGFIX] promtool: Fixed `tsdb dump` silently dropping native histogram samples. [#&#8203;18051](prometheus/prometheus#18051)
- \[BUGFIX] scrape: fix data race in `Manager.TargetsDroppedCounts` to avoid miscounting dropped targets. [#&#8203;19304](prometheus/prometheus#19304)
- \[BUGFIX] scrape: fix nil histogram when native and classic histograms are mixed in one metric family. [#&#8203;19452](prometheus/prometheus#19452)
- \[BUGFIX] TSDB: Fix WAL and GC log messages to emit human-readable duration strings instead of nanosecond integers. [#&#8203;19307](prometheus/prometheus#19307)
- \[BUGFIX] TSDB: Avoid WAL corruption after a failed WAL write in Agent mode. [#&#8203;19700](prometheus/prometheus#19700)
- \[BUGFIX] Agent: Ignore unknown WAL record types, to help users rolling back. [#&#8203;19814](prometheus/prometheus#19814)

</details>

---

### Configuration

📅 **Schedule**: (in timezone Europe/Paris)

- Branch creation
  - At any time (no schedule defined)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xMDQuMiIsInVwZGF0ZWRJblZlciI6IjQ0LjEwNC4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL21pbm9yIl19-->

Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/2199
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants