tsdb: don't lose samples appended to a series removed by head GC by locker95 · Pull Request #19272 · prometheus/prometheus

locker95 · 2026-07-25T20:33:13Z

Description

Head garbage collection can remove a series from the head index while an appender is
already holding a pointer to it. The appender then appends to a memSeries that is no
longer reachable from the head: Append() and Commit() both report success, but the
samples are gone. As @brian-brazil put it on the issue, "it's never okay to silently lose
a write".

Fixes #8365

Relation to #16333

@colega fixed one half of this in #16333 (the race between series creation and gc), and
noted in the PR description and in
#16333 (comment) that he was
deliberately only marking newly created series, leaving the #8365 case — a series that has
existed for a while and gets collected just as more samples arrive — unsolved.

This PR covers that remaining half, building on the same pendingCommit-at-creation
mechanism #16333 introduced.

grafana/mimir#10582 is a production incident report for exactly this half; the analysis by
@alexweav and @colega there is what this PR follows.

Root cause

Looking a series up only holds the stripe lock for the duration of the lookup:

s := a.head.series.getByID(chunks.HeadSeriesRef(ref))   // stripe RLock ... RUnlock
...
s.Lock()                                                // <-- gc() can run in between
...
s.pendingCommit = true                                  // too late

stripeSeries.gc() decides whether a series is still alive by looking at, among other
things, series.pendingCommit. In the window above that flag is still false, so gc
unlinks the series from both the by-ref and the by-hash map. The appender then carries on
with the now-orphaned memSeries, and Commit() writes the samples into a chunk nobody
can reach. Nothing reports an error.

For series that the appender created itself, #16333 closed this by setting pendingCommit
in the constructor, which runs under the stripe lock. That does not help when
getByID()/getByHash() return a pre-existing series, because nothing marks it before
the stripe lock is dropped.

Fix

The locking is deliberately not widened: @codesome
ruled out
holding the stripeSeries lock for the whole append, because it depends on an external
caller invoking Commit()/Rollback() and a missing call would block gc forever.

Instead the loser of the race is made able to detect that it lost. gc already holds the
series lock while it both decides and performs the removal, so it can record that decision
on the series itself:

memSeries gets a gced flag, set by stripeSeries.gc() and stripeSeries.gcSeries()
when the series is unlinked from the head index. Both already hold the series lock
there, so no new locking and no new lock ordering is introduced. The flag fits in the
struct's existing padding — unsafe.Sizeof(memSeries{}) is 168 bytes before and after —
so there is no per-series memory cost.
Appenders take the series lock through the new headAppenderBase.lockForAppend(), which
checks the flag. If the series was collected in the meantime, it obtains a live series
for the same labels through the existing getOrCreate() path (so the WAL series record
and the pendingCommit bookkeeping from fix: race condition between series creation and garbage collection #16333 are taken care of) and returns that one,
locked.

Because gc holds the series lock across "check pendingCommit" and "unlink", both
orderings are now correct:

the appender gets the lock first: it sets pendingCommit = true, gc then sees it and
keeps the series;
gc gets the lock first: it unlinks the series and sets gced, the appender then sees
the flag and appends to a fresh series instead.

The second case ends up in exactly the same state as gc winning the race outright, i.e.
getByID() returning nil — a case the code already handles today. Retrying rather than
returning an error matters here: the scrape loop breaks out of the sample loop on
unexpected append errors, so failing the sample would turn one racy series into a gap for
the whole target.

This is applied to every sample-appending entry point of both appenders: Append,
AppendSTZeroSample, AppendHistogram and AppendHistogramSTZeroSample on
headAppender, and appendFloat, appendHistogram and appendFloatHistogram on
headAppenderV2 — the latter now return the series the sample landed on, so the caller
reports the right SeriesRef.

Deliberately left out of scope: AppendExemplar() and UpdateMetadata() also resolve a
series without marking it. Exemplars are stored keyed by labels, so they are not lost; a
metadata update can be. Both document that they assume the series already exists and
return an error otherwise, so making them create series is a behaviour change that is
better made separately.

Tests

TestHead_RaceBetweenExistingSeriesAppendAndGC is the counterpart of
TestHead_RaceBetweenSeriesCreationAndGC that came with #16333. It appends by reference —
the way the scrape loop does for series it has seen before — while gc() runs, with the
head's min time moved past the previous round's samples so that the existing series really
are collectable, as they are after a head truncation in production.

Before the fix — 10 out of 10 runs failed:

$ go test ./tsdb/ -run 'TestHead_RaceBetweenExistingSeriesAppendAndGC$' -v
=== RUN   TestHead_RaceBetweenExistingSeriesAppendAndGC
    head_test.go:1499:
        	Error Trace:	/.../tsdb/head_test.go:1499
        	Error:      	Not equal:
        	            	expected: 10000
        	            	actual  : 9999
        	Test:       	TestHead_RaceBetweenExistingSeriesAppendAndGC
        	Messages:   	round 4
--- FAIL: TestHead_RaceBetweenExistingSeriesAppendAndGC (0.13s)
FAIL
FAIL	github.com/prometheus/prometheus/tsdb	0.912s

Every Append() in that round returned success, so the missing series is a silently
dropped sample.

After the fix — 0 out of 10 runs failed:

$ go test ./tsdb/ -run 'TestHead_RaceBetweenExistingSeriesAppendAndGC$' -v
=== RUN   TestHead_RaceBetweenExistingSeriesAppendAndGC
--- PASS: TestHead_RaceBetweenExistingSeriesAppendAndGC (0.71s)
PASS

Benchmarks

BenchmarkHeadAppender_AppendCommit restricted to the cases this touches. The old (main)
and new (this PR) test binaries were built once and then run alternately, so both sides saw
the same machine conditions; 6 samples each, -benchtime 2s -cpu 2 -benchmem.

goos: darwin
goarch: arm64
pkg: github.com/prometheus/prometheus/tsdb
cpu: Apple M4
                                                                                                       │     old      │                new                 │
                                                                                                       │    sec/op    │    sec/op     vs base              │
HeadAppender_AppendCommit/appender=v1/case=floats/series=100/samples_per_append=1-2                      166.0µ ± 16%   175.9µ ± 16%       ~ (p=0.937 n=6)
HeadAppender_AppendCommit/appender=v1/case=floats/series=100/samples_per_append=100-2                    4.126m ± 11%   4.120m ± 11%       ~ (p=0.818 n=6)
HeadAppender_AppendCommit/appender=v2/case=floats/series=100/samples_per_append=1-2                      172.3µ ± 11%   177.0µ ± 14%       ~ (p=0.937 n=6)
HeadAppender_AppendCommit/appender=v2/case=floats/series=100/samples_per_append=100-2                    3.940m ±  9%   3.978m ±  9%       ~ (p=0.937 n=6)
HeadAppender_AppendCommit/appender=v1/case=floatsHistogramsExemplars/series=100/samples_per_append=1-2   1.513m ± 17%   1.504m ± 15%       ~ (p=0.699 n=6)
HeadAppender_AppendCommit/appender=v2/case=floatsHistogramsExemplars/series=100/samples_per_append=1-2   1.420m ± 16%   1.504m ± 17%       ~ (p=0.699 n=6)
geomean                                                                                                  999.8µ         1.024m        +2.43%

                                                                                                       │     old      │                new                 │
                                                                                                       │     B/op     │     B/op      vs base              │
HeadAppender_AppendCommit/appender=v1/case=floats/series=100/samples_per_append=1-2                      2.921Ki ± 0%   2.917Ki ± 0%       ~ (p=0.117 n=6)
HeadAppender_AppendCommit/appender=v1/case=floats/series=100/samples_per_append=100-2                    54.00Ki ± 6%   53.80Ki ± 5%       ~ (p=1.000 n=6)
HeadAppender_AppendCommit/appender=v2/case=floats/series=100/samples_per_append=1-2                      2.918Ki ± 0%   2.922Ki ± 0%       ~ (p=0.985 n=6)
HeadAppender_AppendCommit/appender=v2/case=floats/series=100/samples_per_append=100-2                    54.75Ki ± 3%   54.51Ki ± 3%       ~ (p=0.974 n=6)
HeadAppender_AppendCommit/appender=v1/case=floatsHistogramsExemplars/series=100/samples_per_append=1-2   34.39Ki ± 0%   34.39Ki ± 0%       ~ (p=0.900 n=6)
HeadAppender_AppendCommit/appender=v2/case=floatsHistogramsExemplars/series=100/samples_per_append=1-2   34.49Ki ± 0%   34.50Ki ± 0%       ~ (p=0.623 n=6)
geomean                                                                                                  17.62Ki        17.59Ki       -0.13%

                                                                                                       │    old     │                new                 │
                                                                                                       │ allocs/op  │ allocs/op   vs base                │
HeadAppender_AppendCommit/appender=v1/case=floats/series=100/samples_per_append=1-2                      44.00 ± 0%   44.00 ± 0%       ~ (p=1.000 n=6) ¹
HeadAppender_AppendCommit/appender=v1/case=floats/series=100/samples_per_append=100-2                    512.0 ± 0%   512.0 ± 0%       ~ (p=1.000 n=6) ¹
HeadAppender_AppendCommit/appender=v2/case=floats/series=100/samples_per_append=1-2                      44.00 ± 0%   44.00 ± 0%       ~ (p=1.000 n=6) ¹
HeadAppender_AppendCommit/appender=v2/case=floats/series=100/samples_per_append=100-2                    512.5 ± 0%   512.0 ± 0%       ~ (p=0.545 n=6)
HeadAppender_AppendCommit/appender=v1/case=floatsHistogramsExemplars/series=100/samples_per_append=1-2   615.0 ± 0%   615.0 ± 0%       ~ (p=1.000 n=6) ¹
HeadAppender_AppendCommit/appender=v2/case=floatsHistogramsExemplars/series=100/samples_per_append=1-2   616.0 ± 0%   616.0 ± 0%       ~ (p=1.000 n=6) ¹
geomean                                                                                                  240.3        240.2       -0.02%
¹ all samples are equal

Nothing moves outside the noise: every case is ~ and allocs/op is identical. That is
what the change should do — on the hot path it only adds one bool check under a lock the
appender was already taking. The geomean sec/op delta sits inside the ±9-17% run-to-run
spread these benchmarks show on a laptop, and no individual case is significant (p >= 0.69).
Happy to have a maintainer trigger /prombench for a more representative measurement, as
was done for #16333.

Verification

go test ./tsdb/ -race -run 'TestHead_RaceBetween|TestHead_CanGarbagecollect|TestHead_Truncate|TestHead_ActiveAppenders|TestHead_Append|TestHeadAppender|TestAppend|TestHead_GC|TestGCSeriesAccess|TestGCChunkAccess|TestWALReplayRaceWithStaleSeriesCompaction' - passes, no races reported.
go test ./scrape/ ./storage/ - passes.
golangci-lint run ./tsdb/..., gofmt -l, go build -tags dedupelabels ./tsdb/... - clean.
The full go test ./tsdb/... run takes well over an hour on the machine I have here and
I was not able to get a clean end-to-end run, so I verified the affected tests (above)
rather than the whole package. CI will cover the rest; please tell me if anything shows
up there.

[BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to

Head garbage collection can remove a series from the head index between the moment an appender looks the series up and the moment it marks it as pending commit. The appender then appends to a memSeries that is no longer reachable from the head, so the samples are lost even though Append() and Commit() both report success. TestHead_RaceBetweenSeriesCreationAndGC covers that race for series the appender creates itself. This test covers series that already exist in the head, which is the case reported in issue 8365 and explicitly left unsolved by PR 16333. The test fails before the following commit and passes after it. Signed-off-by: Dean Chen <[email protected]>

Looking a series up only holds the stripe lock for the duration of the lookup, so garbage collection can unlink the series before the appender takes the series lock and sets pendingCommit. The appender then appends to an orphaned memSeries and the samples are silently lost. Widening the locking is not an option: holding the stripeSeries lock for the whole append would let a caller that never calls Commit() or Rollback() block garbage collection forever. Instead, record the removal on the series itself. gc already holds the series lock while it both decides and performs the removal, so it can set a gced flag there. Appenders now take the series lock through lockForAppend(), which checks that flag and, when the series was collected, appends to a freshly obtained series for the same labels. That is the same outcome as garbage collection winning the race outright, which is a case the code already handles. The new flag fits in the existing struct padding, so memSeries does not grow. Fixes prometheus#8365 Signed-off-by: Dean Chen <[email protected]>

krajorama · 2026-07-29T05:27:50Z

Thank you, the solution sounds plausible. Checking with prombench.

krajorama · 2026-07-29T05:53:00Z

/prombench main

prombot · 2026-07-29T05:53:02Z

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-19272 and main

After the successful deployment (check status here), the benchmarking results can be viewed at:

Available Commands:

To restart benchmark: /prombench restart main
To stop benchmark: /prombench cancel
To print help: /prombench help

bboreham · 2026-08-23T10:15:12Z

/prombench cancel

prombot · 2026-08-23T10:15:15Z

Benchmark cancel is in progress.

bboreham · 2026-08-23T10:26:41Z

Looks like this Prombench run maybe got forgotten so I canceled it.

Broadly the CPU difference was small enough to ignore, although it did bounce to a 20%+ penalty at times, including a steady period over the last 30 hours.

The full go test ./tsdb/... run takes well over an hour

FYI there is an option to run go test -short which should skip the slowest tests.

locker95 · 2026-08-23T21:40:23Z

yeah, sorry about leaving that bench running. good to hear the cpu delta isn't a blocker — I'll use -short locally next time.

krajorama · 2026-08-25T09:47:12Z

8365

It was my bad, sorry, got lost in the noise of many incoming PRs.

aknuds1

This PR fixes a bug I found independently, and it looks like the right approach. Could you consider including the following from #19469?

TestCompactSelectedSeries_RetriesAfterSeriesEvicted
A seam corresponding to Head.testAfterSeriesLookup for testability
The extension to TestStripeSeries_gc

The append-ID watermark does not apply until the series is locked, so an appender that has only resolved the series can still write into an evicted pointer. Pause after getByID, compact the series away, and check the retry lands on a live series for both appender versions. Also assert stripeSeries.gc marks the unlinked series. Signed-off-by: Dean Chen <[email protected]>

locker95 · 2026-08-25T21:48:55Z

yeah, that race is a bit earlier than the watermark. added TestCompactSelectedSeries_RetriesAfterSeriesEvicted for v1+v2, a hook after getByID, and a gced check on TestStripeSeries_gc. kept our gced flag rather than renaming to retired.

aknuds1

LGTM, thank you!

prometheus#19272 prevents an appender that resolved a series immediately before garbage collection from writing into the unlinked memSeries. It does not protect a series once an appender has already queued an uncommitted sample: gcSeries ignored pending state, and the boolean pendingCommit let one overlapping appender clear another appender's protection. Count one reservation per accepted queued sample, plus a separate reservation for each newly indexed series. Pack the 30-bit count with the histogram end-time and garbage-collected flags so memSeries does not grow. Both GC paths now retain a series while any reservation remains, and commit or rollback releases each reservation. AppendSTZeroSample reserves only after its error and out-of-order rejection checks. An unmatched release would underflow the count and corrupt the packed flags, so it is clamped at zero, reported through prometheus_tsdb_head_series_pending_commit_underflow_total and logged while the commit continues. Overflow panics only when attempting the 2^30th outstanding reservation for one series. Signed-off-by: Arve Knudsen <[email protected]>

##### [\`v3.15.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.15.0) - \[CHANGE] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. Add tests to prevent regression of the fix made in [#18081](prometheus/prometheus#18081). [#18598](prometheus/prometheus#18598) - \[CHANGE] PromQL: Do not register a start timestamp reset if the start timestamp hasn't changed between subsequent samples. [#19454](prometheus/prometheus#19454) - \[CHANGE] Logging: Deprecate `--log.level`; use `runtime.log_level` configuration to supply the default level. [#19511](prometheus/prometheus#19511) - \[FEATURE] Configuration: Allow changing the process log level through `runtime.log_level` on configuration reload. [#19511](prometheus/prometheus#19511) - \[FEATURE] Prometheus: Add `--auto-gomemlimit.refresh-interval` flag to periodically re-detect the container or system memory limit and update `GOMEMLIMIT` at runtime. [#18843](prometheus/prometheus#18843) - \[FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets. [#12024](prometheus/prometheus#12024). [#18091](prometheus/prometheus#18091) - \[FEATURE] scrape: Implement OM2.0 scrape format. [#18606](prometheus/prometheus#18606) - \[ENHANCEMENT] Reduce TSDB head CPU utilization when initializing. [#18001](prometheus/prometheus#18001) - \[ENHANCEMENT] Docker SD: Add labels `__meta_docker_container_image` and `__meta_docker_container_image_id`. [#19386](prometheus/prometheus#19386) - \[ENHANCEMENT] Mixin: Add a p95/p99 remote-write send-batch latency panel to the remote-write dashboard. [#19500](prometheus/prometheus#19500) - \[ENHANCEMENT] Mixin: Support native histograms in the remote-write send-batch latency panel. [#19522](prometheus/prometheus#19522) - \[ENHANCEMENT] PromQL/TSDB: The `--enable-feature=st-storage` flag now automatically enables XOR2 float chunk encoding and ST-capable histogram chunk encoding, so you no longer need to pass `xor2-encoding` and `histograms-st-encoding` alongside it. [#19518](prometheus/prometheus#19518) - \[ENHANCEMENT] Remote write / Alertmanager: upgrade sigv4 to v0.5.0, adding `session_name` and `tags` fields for STS AssumeRole sessions. The previously undocumented `service_name` field is now also documented. [#19569](prometheus/prometheus#19569) - \[ENHANCEMENT] Scraping: Support zstd-compressed scrape responses, enabled via feature flag `zstd-scrape`. [#19502](prometheus/prometheus#19502) - \[ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. `--enable-feature=xor2-encoding` is deprecated; use `storage.tsdb.chunk_encoding.floats: xor2` instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. [#19461](prometheus/prometheus#19461) - \[ENHANCEMENT] TSDB: add `prometheus_tsdb_head_appenders_created_total` metric. [#19411](prometheus/prometheus#19411) - \[ENHANCEMENT] Tracing: add more spans to scrapes, API queries and rule evaluations. [#19410](prometheus/prometheus#19410) - \[ENHANCEMENT] UI: Show the effective configuration for each scrape pool on the Targets and Service Discovery pages. [#19384](prometheus/prometheus#19384) - \[ENHANCEMENT] scrape: Enable start time synthesis for summary `_count` and `_sum` series in scrape appender v2. [#19323](prometheus/prometheus#19323) - \[ENHANCEMENT] scrape: stop all pools in parallel for faster shutdowns. [#19295](prometheus/prometheus#19295) - \[ENHANCEMENT] storage/remote: Add undocumented failed\_request\_logging config field to debug log remote write V2 requests on send errors. [#19249](prometheus/prometheus#19249) - \[ENHANCEMENT] TSDB: Add fast path for XOR chunk decompression to speed up queries. [#18049](prometheus/prometheus#18049) - \[ENHANCEMENT] UI: Improve native histogram table formatting and add a background bar indicating the bucket count. [#19332](prometheus/prometheus#19332) - \[ENHANCEMENT] TSDB: Add `prometheus_tsdb_head_series_pending_commit_underflow_total` to report pending-sample reservation underflows. [#19470](prometheus/prometheus#19470) - \[PERF] AWS SD: Build RDS cluster labels once per cluster instead of once per instance. [#19504](prometheus/prometheus#19504) - \[PERF] AWS SD: Describe RDS instances of different clusters concurrently, bounded by `request_concurrency`. [#19506](prometheus/prometheus#19506) - \[PERF] AWS SD: Describe each ElastiCache resource once per refresh instead of twice. [#19585](prometheus/prometheus#19585) - \[PERF] Remote read: Avoid cloning labels for sampled reads when no external labels are configured. [#19503](prometheus/prometheus#19503) - \[PERF] Remote write: Reuse OTLP converter scratch state between requests. [#19388](prometheus/prometheus#19388) - \[PERF] scrape: conversion from classic to native histograms should only parse start times when enabled. [#19446](prometheus/prometheus#19446) - \[BUGFIX] PromQL: Fix `info()` enrichment for composite expressions with mixed `@`/offset references or selector-free vector branches, preventing metadata from being evaluated at an unrelated timestamp. [#19387](prometheus/prometheus#19387) - \[BUGFIX] PromQL: Fix info() enrichment when input series use different subsets of identifying labels. [#19557](prometheus/prometheus#19557) - \[BUGFIX] PromQL: Preserve metric-name dropping through the info function when delayed name removal is enabled. [#19413](prometheus/prometheus#19413) - \[BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to. [#19272](prometheus/prometheus#19272) - \[BUGFIX] TSDB: Keep series with uncommitted samples during selected- and stale-series compaction, including when appenders overlap. [#19470](prometheus/prometheus#19470) - \[BUGFIX] TSDB: Do not retain head series after a synthetic start-timestamp zero sample is rejected. [#19470](prometheus/prometheus#19470) - \[BUGFIX] TSDB: Prevent query panics during series eviction after WAL replay. [#19664](prometheus/prometheus#19664) - \[BUGFIX] TSDB: fix potential deadlock between mmapSeriesChunks and gcSeries. [#19460](prometheus/prometheus#19460) - \[BUGFIX] TSDB: fix default block reload interval for custom options. [#19368](prometheus/prometheus#19368) - \[BUGFIX] AWS SD: Do not crash on serverless MSK clusters or MSK clusters without Open Monitoring. [#19194](prometheus/prometheus#19194) - \[BUGFIX] AWS SD: Do not panic when the ElastiCache API omits optional fields of a serverless cache or cache cluster. [#19435](prometheus/prometheus#19435) - \[BUGFIX] AWS SD: Reject non-positive `request_concurrency` instead of hanging service discovery indefinitely. [#19524](prometheus/prometheus#19524) - \[BUGFIX] Discovery/AWS: Avoid a panic when discovering standalone ECS tasks with custom task groups. [#19302](prometheus/prometheus#19302) - \[BUGFIX] Discovery: Do not panic in AWS Lightsail service discovery when an instance is missing optional fields such as availability zone, blueprint, bundle, name, state or support code. [#19324](prometheus/prometheus#19324) - \[BUGFIX] Discovery: delete the stale `prometheus_sd_last_update_timestamp_seconds` series for a config that is removed on reload. [#19131](prometheus/prometheus#19131) - \[BUGFIX] HTTP: Avoid truncating compressed responses when handlers set Content-Length. [#19661](prometheus/prometheus#19661) - \[BUGFIX] IONOS SD: Do not panic when the API response omits the server, NIC or volume collections, or a server's properties. [#19438](prometheus/prometheus#19438) - \[BUGFIX] Metadata will not affect the number of Remote Write v2 shards. [#19218](prometheus/prometheus#19218) - \[BUGFIX] Mixins: Fix label mismatches that prevented PrometheusHAGroupNotIngestingSamples and PrometheusHAGroupCrashlooping from firing. [#19444](prometheus/prometheus#19444) - \[BUGFIX] Native histograms: `DetectReset` no longer misses a counter reset when a populated bucket behind an empty one disappears, which could make histogram `rate()`/`increase()` undercount. [#19367](prometheus/prometheus#19367) - \[BUGFIX] Never skip histogram buckets for histogram\_stddev and histogram\_stdvar functions. [#19521](prometheus/prometheus#19521) - \[BUGFIX] OTLP: Do not abort an entire OTLP payload ingestion if one metric has no datapoints. [#19343](prometheus/prometheus#19343) - \[BUGFIX] PromQL: Fix FastRegexMatcher false-positive match when a capturing group is directly adjacent to a literal (e.g. `.*\|(foo)\|.*`). [#19516](prometheus/prometheus#19516) - \[BUGFIX] PromQL: Fix a panic in range selectors using the experimental `anchored` or `smoothed` modifier when the selected series has no samples inside the query window, for example a query evaluated inside a scrape gap. [#19431](prometheus/prometheus#19431) - \[BUGFIX] PromQL: Fix empty results when a subquery with `@` is used as the matrix argument of a call that is not step-invariant (for example `quantile_over_time(scalar(x), metric[...:...] @ T)`). [#19187](prometheus/prometheus#19187) - \[BUGFIX] PromQL: Make the "found duplicate series for the match group" many-to-many matching error message deterministic by sorting the two duplicate labels. [#18810](prometheus/prometheus#18810) - \[BUGFIX] PromQL: Preserve parentheses around duration literals on Expr.String() round-trip. [#19403](prometheus/prometheus#19403) - \[BUGFIX] PromQL: Reject duration-expression offsets and `@ start()` / `@ end()` before range selectors, matching the existing rejection of literal offsets and `@ <timestamp>`. [#19406](prometheus/prometheus#19406) - \[BUGFIX] PromQL: Report the position of the histogram argument rather than of a scalar argument in the native histogram NaN observation annotations of `histogram_quantile` and `histogram_fraction`. [#19330](prometheus/prometheus#19330) - \[BUGFIX] PromQL: info() now applies the @ modifier/offset when evaluating the info series, so info(v @ T) enriches as of T consistently instead of depending on the query start/eval time. [#19266](prometheus/prometheus#19266) - \[BUGFIX] Rules: Fix a panic when a rule manager created without a logger loads a rule file containing multiple YAML documents. [#19433](prometheus/prometheus#19433) - \[BUGFIX] Scrape: Do not append a stale marker for a series that is still exposed when the storage returns a new series reference for it. [#19328](prometheus/prometheus#19328) - \[BUGFIX] Scrape: JSON log formatter correctly format scrape target info. [#19472](prometheus/prometheus#19472) - \[BUGFIX] TSDB: Fix in-order chunk ID overflow by wrapping head chunk IDs modulo 2^23 so they never collide with the out-of-order flag bit. [#19450](prometheus/prometheus#19450) - \[BUGFIX] TSDB: Fix out-of-order chunk ID overflow by wrapping `firstOOOChunkID` modulo 2^23 instead of growing unbounded. [#19216](prometheus/prometheus#19216) - \[BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. [#19326](prometheus/prometheus#19326) - \[BUGFIX] discovery/aws: Avoid a panic when an ElastiCache ARN is missing its resource ID. [#19333](prometheus/prometheus#19333) - \[BUGFIX] discovery/aws: Do not panic on MSK clusters whose optional API fields are absent. [#19584](prometheus/prometheus#19584) - \[BUGFIX] discovery/aws: Do not panic when the EC2 API omits optional instance fields. [#19512](prometheus/prometheus#19512) - \[BUGFIX] discovery/aws: Don't panic on ECS tasks with absent optional fields. [#19396](prometheus/prometheus#19396) - \[BUGFIX] discovery/aws: MSK Optional Custom Configuration Fields [#19422](prometheus/prometheus#19422). [#19422](prometheus/prometheus#19422) - \[BUGFIX] discovery/ionos: Fix panic when the IONOS API omits optional server fields. [#19418](prometheus/prometheus#19418) - \[BUGFIX] discovery/kubernetes: Populate `__meta_kubernetes_service_loadbalancer_ip` from `status.loadBalancer.ingress`, falling back to deprecated `spec.loadBalancerIP`. [#19404](prometheus/prometheus#19404) - \[BUGFIX] histogram: Fix Compact moving buckets to wrong indices, and producing negative bucket counts for integer histograms, when more than one span is merged in the same pass. [#19312](prometheus/prometheus#19312) - \[BUGFIX] promtool: Fixed `tsdb dump` silently dropping native histogram samples. [#18051](prometheus/prometheus#18051) - \[BUGFIX] scrape: fix data race in `Manager.TargetsDroppedCounts` to avoid miscounting dropped targets. [#19304](prometheus/prometheus#19304) - \[BUGFIX] scrape: fix nil histogram when native and classic histograms are mixed in one metric family. [#19452](prometheus/prometheus#19452) - \[BUGFIX] TSDB: Fix WAL and GC log messages to emit human-readable duration strings instead of nanosecond integers. [#19307](prometheus/prometheus#19307) - \[BUGFIX] TSDB: Avoid WAL corruption after a failed WAL write in Agent mode. [#19700](prometheus/prometheus#19700) - \[BUGFIX] Agent: Ignore unknown WAL record types, to help users rolling back. [#19814](prometheus/prometheus#19814)

…→ v3.15.0) (#769) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [quay.io/prometheus/prometheus](https://github.com/prometheus/prometheus) | minor | `v3.14.0` → `v3.15.0` | --- ### Release Notes <details> <summary>prometheus/prometheus (quay.io/prometheus/prometheus)</summary> ### [`v3.15.0`](https://github.com/prometheus/prometheus/releases/tag/v3.15.0): 3.15.0 / 2026-09-24 [Compare Source](prometheus/prometheus@v3.14.0...v3.15.0) - \[CHANGE] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. Add tests to prevent regression of the fix made in [#18081](prometheus/prometheus#18081). [#18598](prometheus/prometheus#18598) - \[CHANGE] PromQL: Do not register a start timestamp reset if the start timestamp hasn't changed between subsequent samples. [#19454](prometheus/prometheus#19454) - \[CHANGE] Logging: Deprecate `--log.level`; use `runtime.log_level` configuration to supply the default level. [#19511](prometheus/prometheus#19511) - \[FEATURE] Configuration: Allow changing the process log level through `runtime.log_level` on configuration reload. [#19511](prometheus/prometheus#19511) - \[FEATURE] Prometheus: Add `--auto-gomemlimit.refresh-interval` flag to periodically re-detect the container or system memory limit and update `GOMEMLIMIT` at runtime. [#18843](prometheus/prometheus#18843) - \[FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets. [#12024](prometheus/prometheus#12024). [#18091](prometheus/prometheus#18091) - \[FEATURE] scrape: Implement OM2.0 scrape format. [#18606](prometheus/prometheus#18606) - \[ENHANCEMENT] Reduce TSDB head CPU utilization when initializing. [#18001](prometheus/prometheus#18001) - \[ENHANCEMENT] Docker SD: Add labels `__meta_docker_container_image` and `__meta_docker_container_image_id`. [#19386](prometheus/prometheus#19386) - \[ENHANCEMENT] Mixin: Add a p95/p99 remote-write send-batch latency panel to the remote-write dashboard. [#19500](prometheus/prometheus#19500) - \[ENHANCEMENT] Mixin: Support native histograms in the remote-write send-batch latency panel. [#19522](prometheus/prometheus#19522) - \[ENHANCEMENT] PromQL/TSDB: The `--enable-feature=st-storage` flag now automatically enables XOR2 float chunk encoding and ST-capable histogram chunk encoding, so you no longer need to pass `xor2-encoding` and `histograms-st-encoding` alongside it. [#19518](prometheus/prometheus#19518) - \[ENHANCEMENT] Remote write / Alertmanager: upgrade sigv4 to v0.5.0, adding `session_name` and `tags` fields for STS AssumeRole sessions. The previously undocumented `service_name` field is now also documented. [#19569](prometheus/prometheus#19569) - \[ENHANCEMENT] Scraping: Support zstd-compressed scrape responses, enabled via feature flag `zstd-scrape`. [#19502](prometheus/prometheus#19502) - \[ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. `--enable-feature=xor2-encoding` is deprecated; use `storage.tsdb.chunk_encoding.floats: xor2` instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. [#19461](prometheus/prometheus#19461) - \[ENHANCEMENT] TSDB: add `prometheus_tsdb_head_appenders_created_total` metric. [#19411](prometheus/prometheus#19411) - \[ENHANCEMENT] Tracing: add more spans to scrapes, API queries and rule evaluations. [#19410](prometheus/prometheus#19410) - \[ENHANCEMENT] UI: Show the effective configuration for each scrape pool on the Targets and Service Discovery pages. [#19384](prometheus/prometheus#19384) - \[ENHANCEMENT] scrape: Enable start time synthesis for summary `_count` and `_sum` series in scrape appender v2. [#19323](prometheus/prometheus#19323) - \[ENHANCEMENT] scrape: stop all pools in parallel for faster shutdowns. [#19295](prometheus/prometheus#19295) - \[ENHANCEMENT] storage/remote: Add undocumented failed\_request\_logging config field to debug log remote write V2 requests on send errors. [#19249](prometheus/prometheus#19249) - \[ENHANCEMENT] TSDB: Add fast path for XOR chunk decompression to speed up queries. [#18049](prometheus/prometheus#18049) - \[ENHANCEMENT] UI: Improve native histogram table formatting and add a background bar indicating the bucket count. [#19332](prometheus/prometheus#19332) - \[ENHANCEMENT] TSDB: Add `prometheus_tsdb_head_series_pending_commit_underflow_total` to report pending-sample reservation underflows. [#19470](prometheus/prometheus#19470) - \[PERF] AWS SD: Build RDS cluster labels once per cluster instead of once per instance. [#19504](prometheus/prometheus#19504) - \[PERF] AWS SD: Describe RDS instances of different clusters concurrently, bounded by `request_concurrency`. [#19506](prometheus/prometheus#19506) - \[PERF] AWS SD: Describe each ElastiCache resource once per refresh instead of twice. [#19585](prometheus/prometheus#19585) - \[PERF] Remote read: Avoid cloning labels for sampled reads when no external labels are configured. [#19503](prometheus/prometheus#19503) - \[PERF] Remote write: Reuse OTLP converter scratch state between requests. [#19388](prometheus/prometheus#19388) - \[PERF] scrape: conversion from classic to native histograms should only parse start times when enabled. [#19446](prometheus/prometheus#19446) - \[BUGFIX] PromQL: Fix `info()` enrichment for composite expressions with mixed `@`/offset references or selector-free vector branches, preventing metadata from being evaluated at an unrelated timestamp. [#19387](prometheus/prometheus#19387) - \[BUGFIX] PromQL: Fix info() enrichment when input series use different subsets of identifying labels. [#19557](prometheus/prometheus#19557) - \[BUGFIX] PromQL: Preserve metric-name dropping through the info function when delayed name removal is enabled. [#19413](prometheus/prometheus#19413) - \[BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to. [#19272](prometheus/prometheus#19272) - \[BUGFIX] TSDB: Keep series with uncommitted samples during selected- and stale-series compaction, including when appenders overlap. [#19470](prometheus/prometheus#19470) - \[BUGFIX] TSDB: Do not retain head series after a synthetic start-timestamp zero sample is rejected. [#19470](prometheus/prometheus#19470) - \[BUGFIX] TSDB: Prevent query panics during series eviction after WAL replay. [#19664](prometheus/prometheus#19664) - \[BUGFIX] TSDB: fix potential deadlock between mmapSeriesChunks and gcSeries. [#19460](prometheus/prometheus#19460) - \[BUGFIX] TSDB: fix default block reload interval for custom options. [#19368](prometheus/prometheus#19368) - \[BUGFIX] AWS SD: Do not crash on serverless MSK clusters or MSK clusters without Open Monitoring. [#19194](prometheus/prometheus#19194) - \[BUGFIX] AWS SD: Do not panic when the ElastiCache API omits optional fields of a serverless cache or cache cluster. [#19435](prometheus/prometheus#19435) - \[BUGFIX] AWS SD: Reject non-positive `request_concurrency` instead of hanging service discovery indefinitely. [#19524](prometheus/prometheus#19524) - \[BUGFIX] Discovery/AWS: Avoid a panic when discovering standalone ECS tasks with custom task groups. [#19302](prometheus/prometheus#19302) - \[BUGFIX] Discovery: Do not panic in AWS Lightsail service discovery when an instance is missing optional fields such as availability zone, blueprint, bundle, name, state or support code. [#19324](prometheus/prometheus#19324) - \[BUGFIX] Discovery: delete the stale `prometheus_sd_last_update_timestamp_seconds` series for a config that is removed on reload. [#19131](prometheus/prometheus#19131) - \[BUGFIX] HTTP: Avoid truncating compressed responses when handlers set Content-Length. [#19661](prometheus/prometheus#19661) - \[BUGFIX] IONOS SD: Do not panic when the API response omits the server, NIC or volume collections, or a server's properties. [#19438](prometheus/prometheus#19438) - \[BUGFIX] Metadata will not affect the number of Remote Write v2 shards. [#19218](prometheus/prometheus#19218) - \[BUGFIX] Mixins: Fix label mismatches that prevented PrometheusHAGroupNotIngestingSamples and PrometheusHAGroupCrashlooping from firing. [#19444](prometheus/prometheus#19444) - \[BUGFIX] Native histograms: `DetectReset` no longer misses a counter reset when a populated bucket behind an empty one disappears, which could make histogram `rate()`/`increase()` undercount. [#19367](prometheus/prometheus#19367) - \[BUGFIX] Never skip histogram buckets for histogram\_stddev and histogram\_stdvar functions. [#19521](prometheus/prometheus#19521) - \[BUGFIX] OTLP: Do not abort an entire OTLP payload ingestion if one metric has no datapoints. [#19343](prometheus/prometheus#19343) - \[BUGFIX] PromQL: Fix FastRegexMatcher false-positive match when a capturing group is directly adjacent to a literal (e.g. `.*\|(foo)\|.*`). [#19516](prometheus/prometheus#19516) - \[BUGFIX] PromQL: Fix a panic in range selectors using the experimental `anchored` or `smoothed` modifier when the selected series has no samples inside the query window, for example a query evaluated inside a scrape gap. [#19431](prometheus/prometheus#19431) - \[BUGFIX] PromQL: Fix empty results when a subquery with `@` is used as the matrix argument of a call that is not step-invariant (for example `quantile_over_time(scalar(x), metric[...:...] @ T)`). [#19187](prometheus/prometheus#19187) - \[BUGFIX] PromQL: Make the "found duplicate series for the match group" many-to-many matching error message deterministic by sorting the two duplicate labels. [#18810](prometheus/prometheus#18810) - \[BUGFIX] PromQL: Preserve parentheses around duration literals on Expr.String() round-trip. [#19403](prometheus/prometheus#19403) - \[BUGFIX] PromQL: Reject duration-expression offsets and `@ start()` / `@ end()` before range selectors, matching the existing rejection of literal offsets and `@ <timestamp>`. [#19406](prometheus/prometheus#19406) - \[BUGFIX] PromQL: Report the position of the histogram argument rather than of a scalar argument in the native histogram NaN observation annotations of `histogram_quantile` and `histogram_fraction`. [#19330](prometheus/prometheus#19330) - \[BUGFIX] PromQL: info() now applies the @ modifier/offset when evaluating the info series, so info(v @ T) enriches as of T consistently instead of depending on the query start/eval time. [#19266](prometheus/prometheus#19266) - \[BUGFIX] Rules: Fix a panic when a rule manager created without a logger loads a rule file containing multiple YAML documents. [#19433](prometheus/prometheus#19433) - \[BUGFIX] Scrape: Do not append a stale marker for a series that is still exposed when the storage returns a new series reference for it. [#19328](prometheus/prometheus#19328) - \[BUGFIX] Scrape: JSON log formatter correctly format scrape target info. [#19472](prometheus/prometheus#19472) - \[BUGFIX] TSDB: Fix in-order chunk ID overflow by wrapping head chunk IDs modulo 2^23 so they never collide with the out-of-order flag bit. [#19450](prometheus/prometheus#19450) - \[BUGFIX] TSDB: Fix out-of-order chunk ID overflow by wrapping `firstOOOChunkID` modulo 2^23 instead of growing unbounded. [#19216](prometheus/prometheus#19216) - \[BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. [#19326](prometheus/prometheus#19326) - \[BUGFIX] discovery/aws: Avoid a panic when an ElastiCache ARN is missing its resource ID. [#19333](prometheus/prometheus#19333) - \[BUGFIX] discovery/aws: Do not panic on MSK clusters whose optional API fields are absent. [#19584](prometheus/prometheus#19584) - \[BUGFIX] discovery/aws: Do not panic when the EC2 API omits optional instance fields. [#19512](prometheus/prometheus#19512) - \[BUGFIX] discovery/aws: Don't panic on ECS tasks with absent optional fields. [#19396](prometheus/prometheus#19396) - \[BUGFIX] discovery/aws: MSK Optional Custom Configuration Fields [#19422](prometheus/prometheus#19422). [#19422](prometheus/prometheus#19422) - \[BUGFIX] discovery/ionos: Fix panic when the IONOS API omits optional server fields. [#19418](prometheus/prometheus#19418) - \[BUGFIX] discovery/kubernetes: Populate `__meta_kubernetes_service_loadbalancer_ip` from `status.loadBalancer.ingress`, falling back to deprecated `spec.loadBalancerIP`. [#19404](prometheus/prometheus#19404) - \[BUGFIX] histogram: Fix Compact moving buckets to wrong indices, and producing negative bucket counts for integer histograms, when more than one span is merged in the same pass. [#19312](prometheus/prometheus#19312) - \[BUGFIX] promtool: Fixed `tsdb dump` silently dropping native histogram samples. [#18051](prometheus/prometheus#18051) - \[BUGFIX] scrape: fix data race in `Manager.TargetsDroppedCounts` to avoid miscounting dropped targets. [#19304](prometheus/prometheus#19304) - \[BUGFIX] scrape: fix nil histogram when native and classic histograms are mixed in one metric family. [#19452](prometheus/prometheus#19452) - \[BUGFIX] TSDB: Fix WAL and GC log messages to emit human-readable duration strings instead of nanosecond integers. [#19307](prometheus/prometheus#19307) - \[BUGFIX] TSDB: Avoid WAL corruption after a failed WAL write in Agent mode. [#19700](prometheus/prometheus#19700) - \[BUGFIX] Agent: Ignore unknown WAL record types, to help users rolling back. [#19814](prometheus/prometheus#19814) </details> --- ### Configuration 📅 **Schedule**: (in timezone Europe/London) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.hayden.moe/hayden/phoebe/pulls/769

…➔ v3.15.0) (#2199) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [quay.io/prometheus/prometheus](https://github.com/prometheus/prometheus) | minor | `v3.14.0` → `v3.15.0` | --- > ⚠️ **Warning** > > Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/567) for more information. --- ### Release Notes <details> <summary>prometheus/prometheus (quay.io/prometheus/prometheus)</summary> ### [`v3.15.0`](https://github.com/prometheus/prometheus/releases/tag/v3.15.0): 3.15.0 / 2026-09-24 [Compare Source](prometheus/prometheus@v3.14.0...v3.15.0) - \[CHANGE] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. Add tests to prevent regression of the fix made in [#18081](prometheus/prometheus#18081). [#18598](prometheus/prometheus#18598) - \[CHANGE] PromQL: Do not register a start timestamp reset if the start timestamp hasn't changed between subsequent samples. [#19454](prometheus/prometheus#19454) - \[CHANGE] Logging: Deprecate `--log.level`; use `runtime.log_level` configuration to supply the default level. [#19511](prometheus/prometheus#19511) - \[FEATURE] Configuration: Allow changing the process log level through `runtime.log_level` on configuration reload. [#19511](prometheus/prometheus#19511) - \[FEATURE] Prometheus: Add `--auto-gomemlimit.refresh-interval` flag to periodically re-detect the container or system memory limit and update `GOMEMLIMIT` at runtime. [#18843](prometheus/prometheus#18843) - \[FEATURE] Scraping: Add support for scraping targets via Unix Domain Sockets. [#12024](prometheus/prometheus#12024). [#18091](prometheus/prometheus#18091) - \[FEATURE] scrape: Implement OM2.0 scrape format. [#18606](prometheus/prometheus#18606) - \[ENHANCEMENT] Reduce TSDB head CPU utilization when initializing. [#18001](prometheus/prometheus#18001) - \[ENHANCEMENT] Docker SD: Add labels `__meta_docker_container_image` and `__meta_docker_container_image_id`. [#19386](prometheus/prometheus#19386) - \[ENHANCEMENT] Mixin: Add a p95/p99 remote-write send-batch latency panel to the remote-write dashboard. [#19500](prometheus/prometheus#19500) - \[ENHANCEMENT] Mixin: Support native histograms in the remote-write send-batch latency panel. [#19522](prometheus/prometheus#19522) - \[ENHANCEMENT] PromQL/TSDB: The `--enable-feature=st-storage` flag now automatically enables XOR2 float chunk encoding and ST-capable histogram chunk encoding, so you no longer need to pass `xor2-encoding` and `histograms-st-encoding` alongside it. [#19518](prometheus/prometheus#19518) - \[ENHANCEMENT] Remote write / Alertmanager: upgrade sigv4 to v0.5.0, adding `session_name` and `tags` fields for STS AssumeRole sessions. The previously undocumented `service_name` field is now also documented. [#19569](prometheus/prometheus#19569) - \[ENHANCEMENT] Scraping: Support zstd-compressed scrape responses, enabled via feature flag `zstd-scrape`. [#19502](prometheus/prometheus#19502) - \[ENHANCEMENT] TSDB: Stabilize the XOR2 float chunk encoding. `--enable-feature=xor2-encoding` is deprecated; use `storage.tsdb.chunk_encoding.floats: xor2` instead. Check that other software reading the TSDB directly (e.g. Thanos sidecar) supports XOR2 before enabling. [#19461](prometheus/prometheus#19461) - \[ENHANCEMENT] TSDB: add `prometheus_tsdb_head_appenders_created_total` metric. [#19411](prometheus/prometheus#19411) - \[ENHANCEMENT] Tracing: add more spans to scrapes, API queries and rule evaluations. [#19410](prometheus/prometheus#19410) - \[ENHANCEMENT] UI: Show the effective configuration for each scrape pool on the Targets and Service Discovery pages. [#19384](prometheus/prometheus#19384) - \[ENHANCEMENT] scrape: Enable start time synthesis for summary `_count` and `_sum` series in scrape appender v2. [#19323](prometheus/prometheus#19323) - \[ENHANCEMENT] scrape: stop all pools in parallel for faster shutdowns. [#19295](prometheus/prometheus#19295) - \[ENHANCEMENT] storage/remote: Add undocumented failed\_request\_logging config field to debug log remote write V2 requests on send errors. [#19249](prometheus/prometheus#19249) - \[ENHANCEMENT] TSDB: Add fast path for XOR chunk decompression to speed up queries. [#18049](prometheus/prometheus#18049) - \[ENHANCEMENT] UI: Improve native histogram table formatting and add a background bar indicating the bucket count. [#19332](prometheus/prometheus#19332) - \[ENHANCEMENT] TSDB: Add `prometheus_tsdb_head_series_pending_commit_underflow_total` to report pending-sample reservation underflows. [#19470](prometheus/prometheus#19470) - \[PERF] AWS SD: Build RDS cluster labels once per cluster instead of once per instance. [#19504](prometheus/prometheus#19504) - \[PERF] AWS SD: Describe RDS instances of different clusters concurrently, bounded by `request_concurrency`. [#19506](prometheus/prometheus#19506) - \[PERF] AWS SD: Describe each ElastiCache resource once per refresh instead of twice. [#19585](prometheus/prometheus#19585) - \[PERF] Remote read: Avoid cloning labels for sampled reads when no external labels are configured. [#19503](prometheus/prometheus#19503) - \[PERF] Remote write: Reuse OTLP converter scratch state between requests. [#19388](prometheus/prometheus#19388) - \[PERF] scrape: conversion from classic to native histograms should only parse start times when enabled. [#19446](prometheus/prometheus#19446) - \[BUGFIX] PromQL: Fix `info()` enrichment for composite expressions with mixed `@`/offset references or selector-free vector branches, preventing metadata from being evaluated at an unrelated timestamp. [#19387](prometheus/prometheus#19387) - \[BUGFIX] PromQL: Fix info() enrichment when input series use different subsets of identifying labels. [#19557](prometheus/prometheus#19557) - \[BUGFIX] PromQL: Preserve metric-name dropping through the info function when delayed name removal is enabled. [#19413](prometheus/prometheus#19413) - \[BUGFIX] TSDB: Don't silently drop samples when head garbage collection removes a series while it is being appended to. [#19272](prometheus/prometheus#19272) - \[BUGFIX] TSDB: Keep series with uncommitted samples during selected- and stale-series compaction, including when appenders overlap. [#19470](prometheus/prometheus#19470) - \[BUGFIX] TSDB: Do not retain head series after a synthetic start-timestamp zero sample is rejected. [#19470](prometheus/prometheus#19470) - \[BUGFIX] TSDB: Prevent query panics during series eviction after WAL replay. [#19664](prometheus/prometheus#19664) - \[BUGFIX] TSDB: fix potential deadlock between mmapSeriesChunks and gcSeries. [#19460](prometheus/prometheus#19460) - \[BUGFIX] TSDB: fix default block reload interval for custom options. [#19368](prometheus/prometheus#19368) - \[BUGFIX] AWS SD: Do not crash on serverless MSK clusters or MSK clusters without Open Monitoring. [#19194](prometheus/prometheus#19194) - \[BUGFIX] AWS SD: Do not panic when the ElastiCache API omits optional fields of a serverless cache or cache cluster. [#19435](prometheus/prometheus#19435) - \[BUGFIX] AWS SD: Reject non-positive `request_concurrency` instead of hanging service discovery indefinitely. [#19524](prometheus/prometheus#19524) - \[BUGFIX] Discovery/AWS: Avoid a panic when discovering standalone ECS tasks with custom task groups. [#19302](prometheus/prometheus#19302) - \[BUGFIX] Discovery: Do not panic in AWS Lightsail service discovery when an instance is missing optional fields such as availability zone, blueprint, bundle, name, state or support code. [#19324](prometheus/prometheus#19324) - \[BUGFIX] Discovery: delete the stale `prometheus_sd_last_update_timestamp_seconds` series for a config that is removed on reload. [#19131](prometheus/prometheus#19131) - \[BUGFIX] HTTP: Avoid truncating compressed responses when handlers set Content-Length. [#19661](prometheus/prometheus#19661) - \[BUGFIX] IONOS SD: Do not panic when the API response omits the server, NIC or volume collections, or a server's properties. [#19438](prometheus/prometheus#19438) - \[BUGFIX] Metadata will not affect the number of Remote Write v2 shards. [#19218](prometheus/prometheus#19218) - \[BUGFIX] Mixins: Fix label mismatches that prevented PrometheusHAGroupNotIngestingSamples and PrometheusHAGroupCrashlooping from firing. [#19444](prometheus/prometheus#19444) - \[BUGFIX] Native histograms: `DetectReset` no longer misses a counter reset when a populated bucket behind an empty one disappears, which could make histogram `rate()`/`increase()` undercount. [#19367](prometheus/prometheus#19367) - \[BUGFIX] Never skip histogram buckets for histogram\_stddev and histogram\_stdvar functions. [#19521](prometheus/prometheus#19521) - \[BUGFIX] OTLP: Do not abort an entire OTLP payload ingestion if one metric has no datapoints. [#19343](prometheus/prometheus#19343) - \[BUGFIX] PromQL: Fix FastRegexMatcher false-positive match when a capturing group is directly adjacent to a literal (e.g. `.*\|(foo)\|.*`). [#19516](prometheus/prometheus#19516) - \[BUGFIX] PromQL: Fix a panic in range selectors using the experimental `anchored` or `smoothed` modifier when the selected series has no samples inside the query window, for example a query evaluated inside a scrape gap. [#19431](prometheus/prometheus#19431) - \[BUGFIX] PromQL: Fix empty results when a subquery with `@` is used as the matrix argument of a call that is not step-invariant (for example `quantile_over_time(scalar(x), metric[...:...] @ T)`). [#19187](prometheus/prometheus#19187) - \[BUGFIX] PromQL: Make the "found duplicate series for the match group" many-to-many matching error message deterministic by sorting the two duplicate labels. [#18810](prometheus/prometheus#18810) - \[BUGFIX] PromQL: Preserve parentheses around duration literals on Expr.String() round-trip. [#19403](prometheus/prometheus#19403) - \[BUGFIX] PromQL: Reject duration-expression offsets and `@ start()` / `@ end()` before range selectors, matching the existing rejection of literal offsets and `@ <timestamp>`. [#19406](prometheus/prometheus#19406) - \[BUGFIX] PromQL: Report the position of the histogram argument rather than of a scalar argument in the native histogram NaN observation annotations of `histogram_quantile` and `histogram_fraction`. [#19330](prometheus/prometheus#19330) - \[BUGFIX] PromQL: info() now applies the @ modifier/offset when evaluating the info series, so info(v @ T) enriches as of T consistently instead of depending on the query start/eval time. [#19266](prometheus/prometheus#19266) - \[BUGFIX] Rules: Fix a panic when a rule manager created without a logger loads a rule file containing multiple YAML documents. [#19433](prometheus/prometheus#19433) - \[BUGFIX] Scrape: Do not append a stale marker for a series that is still exposed when the storage returns a new series reference for it. [#19328](prometheus/prometheus#19328) - \[BUGFIX] Scrape: JSON log formatter correctly format scrape target info. [#19472](prometheus/prometheus#19472) - \[BUGFIX] TSDB: Fix in-order chunk ID overflow by wrapping head chunk IDs modulo 2^23 so they never collide with the out-of-order flag bit. [#19450](prometheus/prometheus#19450) - \[BUGFIX] TSDB: Fix out-of-order chunk ID overflow by wrapping `firstOOOChunkID` modulo 2^23 instead of growing unbounded. [#19216](prometheus/prometheus#19216) - \[BUGFIX] UI: Remove an extraneous X-axis tick mark in the native histogram chart when using the "linear" display mode. [#19326](prometheus/prometheus#19326) - \[BUGFIX] discovery/aws: Avoid a panic when an ElastiCache ARN is missing its resource ID. [#19333](prometheus/prometheus#19333) - \[BUGFIX] discovery/aws: Do not panic on MSK clusters whose optional API fields are absent. [#19584](prometheus/prometheus#19584) - \[BUGFIX] discovery/aws: Do not panic when the EC2 API omits optional instance fields. [#19512](prometheus/prometheus#19512) - \[BUGFIX] discovery/aws: Don't panic on ECS tasks with absent optional fields. [#19396](prometheus/prometheus#19396) - \[BUGFIX] discovery/aws: MSK Optional Custom Configuration Fields [#19422](prometheus/prometheus#19422). [#19422](prometheus/prometheus#19422) - \[BUGFIX] discovery/ionos: Fix panic when the IONOS API omits optional server fields. [#19418](prometheus/prometheus#19418) - \[BUGFIX] discovery/kubernetes: Populate `__meta_kubernetes_service_loadbalancer_ip` from `status.loadBalancer.ingress`, falling back to deprecated `spec.loadBalancerIP`. [#19404](prometheus/prometheus#19404) - \[BUGFIX] histogram: Fix Compact moving buckets to wrong indices, and producing negative bucket counts for integer histograms, when more than one span is merged in the same pass. [#19312](prometheus/prometheus#19312) - \[BUGFIX] promtool: Fixed `tsdb dump` silently dropping native histogram samples. [#18051](prometheus/prometheus#18051) - \[BUGFIX] scrape: fix data race in `Manager.TargetsDroppedCounts` to avoid miscounting dropped targets. [#19304](prometheus/prometheus#19304) - \[BUGFIX] scrape: fix nil histogram when native and classic histograms are mixed in one metric family. [#19452](prometheus/prometheus#19452) - \[BUGFIX] TSDB: Fix WAL and GC log messages to emit human-readable duration strings instead of nanosecond integers. [#19307](prometheus/prometheus#19307) - \[BUGFIX] TSDB: Avoid WAL corruption after a failed WAL write in Agent mode. [#19700](prometheus/prometheus#19700) - \[BUGFIX] Agent: Ignore unknown WAL record types, to help users rolling back. [#19814](prometheus/prometheus#19814) </details> --- ### Configuration 📅 **Schedule**: (in timezone Europe/Paris) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/2199

locker95 added 2 commits July 26, 2026 01:32

locker95 requested review from a team, bwplotka, codesome, jesusvazquez and krajorama as code owners July 25, 2026 20:33

prombot added the prombench label Jul 29, 2026

krajorama mentioned this pull request Aug 25, 2026

tsdb: retry appends when GC retires the resolved series #19469

Closed

krajorama requested a review from aknuds1 August 25, 2026 10:18

krajorama mentioned this pull request Aug 25, 2026

tsdb: fix gcSeries and mmapHeadChunksInStripe deadlocking #19460

Merged

aknuds1 reviewed Aug 25, 2026

View reviewed changes

aknuds1 approved these changes Aug 26, 2026

View reviewed changes

aknuds1 added kind/bug component/tsdb labels Aug 26, 2026

aknuds1 merged commit e4e086c into prometheus:main Aug 26, 2026
67 of 69 checks passed

aknuds1 mentioned this pull request Aug 26, 2026

tsdb: keep series with in-flight appends during eviction #19470

Merged

github-actions Bot mentioned this pull request Oct 19, 2026

Update prom/prometheus Docker tag to v3.15.0 claytono/infra#3239

Merged

1 task

Navigation Menu

Repository navigation

tsdb: don't lose samples appended to a series removed by head GC - #19272

tsdb: don't lose samples appended to a series removed by head GC#19272
aknuds1 merged 3 commits into
prometheus:mainfrom
locker95:fix/head-gc-append-race-8365

locker95 commented Jul 25, 2026 •

edited by aknuds1

Loading

Uh oh!

krajorama commented Jul 29, 2026

Uh oh!

krajorama commented Jul 29, 2026

Uh oh!

prombot commented Jul 29, 2026

Uh oh!

bboreham commented Aug 23, 2026

Uh oh!

prombot commented Aug 23, 2026

Uh oh!

bboreham commented Aug 23, 2026

Uh oh!

locker95 commented Aug 23, 2026

Uh oh!

krajorama commented Aug 25, 2026

Uh oh!

aknuds1 left a comment

Uh oh!

locker95 commented Aug 25, 2026

Uh oh!

aknuds1 left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

locker95 commented Jul 25, 2026 • edited by aknuds1 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Relation to #16333

Root cause

Fix

Tests

Benchmarks

Verification

Uh oh!

krajorama commented Jul 29, 2026

Uh oh!

krajorama commented Jul 29, 2026

Uh oh!

prombot commented Jul 29, 2026

Uh oh!

bboreham commented Aug 23, 2026

Uh oh!

prombot commented Aug 23, 2026

Uh oh!

bboreham commented Aug 23, 2026

Uh oh!

locker95 commented Aug 23, 2026

Uh oh!

krajorama commented Aug 25, 2026

Uh oh!

aknuds1 left a comment

Choose a reason for hiding this comment

Uh oh!

locker95 commented Aug 25, 2026

Uh oh!

aknuds1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

locker95 commented Jul 25, 2026 •

edited by aknuds1

Loading