tsdb/agent: Prevent duplicate SeriesRefs from being lost in stripeSeries by kgeckhart · Pull Request #17538 · prometheus/prometheus

kgeckhart · 2025-11-14T22:06:10Z

Which issue(s) does the PR fix:

This PR applies a fix which allows SeriesRefs to be held forever that impacts tsdb/agent described in grafana/alloy#3926.

The first commit is a test that shows duplicate SeriesRefs are able to be referenced if the db is closed before a SeriesRef is fully removed,

=== RUN   TestDuplicateSeriesRefsByHash
    db_test.go:1396: 
        	Error Trace:	/Users/keckhart/src/prometheus/tsdb/agent/db_test.go:1396
        	Error:      	Expected nil, but got: &agent.memSeries{Mutex:sync.Mutex{_:sync.noCopy{}, mu:sync.Mutex{state:0, sema:0x0}}, ref:0x5, lset:labels.Labels{data:"\b__name__\x03foo"}, lastTs:20}
        	Test:       	TestDuplicateSeriesRefsByHash
--- FAIL: TestDuplicateSeriesRefsByHash (0.05s)

The fix is similar to how the head_wal handles this same use case https://github.com/prometheus/prometheus/blob/main/tsdb/head_wal.go#L257-L434,

New series are created only if they represent a unique label hash through the new stripeSeries.GetOrSet
Duplicate series are tracked in the WAL as deleted with the lastSegment where we found a sample (I'm pretty sure the issue described in WAL checkpoint can drop series references still needed for samples kept in checkpoint #16859 applies here as well)
Duplicate SeriesRefs are rewritten to a single SeriesRef when they are loaded using duplicateRefToValidRef aka multiRef

Does this PR introduce a user-facing change?

[BUGFIX] Prevent duplicate SeriesRefs from being loaded as active series in tsdb/agent allowing them to be held forever.

krajorama · 2026-02-10T11:28:14Z

Hello from the bug scrub!

@krajorama volunteered to review.

krajorama

Looks good in general, @kgeckhart please rebase so tests can run again.
I've added a couple of comments.

Signed-off-by: Kyle Eckhart <[email protected]>

##### [\`v3.11.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.11.0) - \[CHANGE] Hetzner SD: The `__meta_hetzner_datacenter` label is deprecated for the role `robot` but kept for backward compatibility, use the `__meta_hetzner_robot_datacenter` label instead. For the role `hcloud`, the label is deprecated and will stop working after the 1 July 2026. [#17850](prometheus/prometheus#17850) - \[CHANGE] Hetzner SD: The `__meta_hetzner_hcloud_datacenter_location` and `__meta_hetzner_hcloud_datacenter_location_network_zone` labels are deprecated, use the `__meta_hetzner_hcloud_location` and `__meta_hetzner_hcloud_location_network_zone` labels instead. [#17850](prometheus/prometheus#17850) - \[CHANGE] Promtool: Redirect debug output to stderr to avoid interfering with stdout-based tool output. [#18346](prometheus/prometheus#18346) - \[FEATURE] AWS SD: Add Elasticache Role. [#18099](prometheus/prometheus#18099) - \[FEATURE] AWS SD: Add RDS Role. [#18206](prometheus/prometheus#18206) - \[FEATURE] Azure SD: Add support for Azure Workload Identity authentication method. [#17207](prometheus/prometheus#17207) - \[FEATURE] Discovery: Introduce `prometheus_sd_last_update_timestamp_seconds` metric to track the last time a service discovery update was sent to consumers. [#18194](prometheus/prometheus#18194) - \[FEATURE] Kubernetes SD: Add support for node role selectors for pod roles. [#18006](prometheus/prometheus#18006) - \[FEATURE] Kubernetes SD: Introduce pod-based labels for deployment, cronjob, and job controller names: `__meta_kubernetes_pod_deployment_name`, `__meta_kubernetes_pod_cronjob_name` and `__meta_kubernetes_pod_job_name`, respectively. [#17774](prometheus/prometheus#17774) - \[FEATURE] PromQL: Add `</` and `>/` operators for trimming observations from native histograms. [#17904](prometheus/prometheus#17904) - \[FEATURE] PromQL: Add experimental `histogram_quantiles` variadic function for computing multiple quantiles at once. [#17285](prometheus/prometheus#17285) - \[FEATURE] TSDB: Add `storage.tsdb.retention.percentage` configuration to configure the maximum percent of disk usable for TSDB storage. [#18080](prometheus/prometheus#18080) - \[FEATURE] TSDB: Add an experimental `fast-startup` feature flag that writes a `series_state.json` file to the WAL directory to track active series state across restarts. [#18303](prometheus/prometheus#18303) - \[FEATURE] TSDB: Add an experimental `st-storage` feature flag. When enabled, Prometheus stores ingested start timestamps (ST, previously called Created Timestamp) from scrape or OTLP in the TSDB and Agent WAL, and exposes them via Remote Write 2. [#18062](prometheus/prometheus#18062) - \[FEATURE] TSDB: Add an experimental `xor2-encoding` feature flag for the new TSDB block float sample chunk encoding that is optimized for scraped data and allows encoding start timestamps. [#18062](prometheus/prometheus#18062) - \[ENHANCEMENT] HTTP client: Add AWS `external_id` support for sigv4. [#17916](prometheus/prometheus#17916) - \[ENHANCEMENT] Kubernetes SD: Deduplicate deprecation warning logs from the Kubernetes API to reduce noise. [#17829](prometheus/prometheus#17829) - \[ENHANCEMENT] TSDB: Remove old temporary checkpoints when creating a Checkpoint. [#17598](prometheus/prometheus#17598) - \[ENHANCEMENT] UI: Add autocomplete support for experimental `first_over_time` and `ts_of_first_over_time` PromQL functions. [#18318](prometheus/prometheus#18318) - \[ENHANCEMENT] Vultr SD: Upgrade govultr library from v2 to v3 for continued security patches and maintenance. [#18347](prometheus/prometheus#18347) - \[PERF] PromQL: Improve performance and reduce heap allocations in joins (VectorBinop)/And/Or/Unless. [#17159](prometheus/prometheus#17159) - \[PERF] PromQL: Partially address performance regression in native histogram aggregations due to using `KahanAdd`. [#18252](prometheus/prometheus#18252) - \[PERF] Remote write: Optimize WAL watching used for RW sending to reuse internal buffers. [#18250](prometheus/prometheus#18250) - \[PERF] TSDB: Optimize LabelValues intersection performance for matchers. [#18069](prometheus/prometheus#18069) - \[PERF] UI: Skip restacking on hover in stacked series charts. [#18230](prometheus/prometheus#18230) - \[BUGFIX] AWS SD: Fix EC2 SD ignoring the configured `endpoint` option, a regression from the AWS SDK v2 migration. [#18133](prometheus/prometheus#18133) - \[BUGFIX] AWS SD: Fix panic in EC2 SD when DescribeAvailabilityZones returns nil ZoneName or ZoneId. [#18133](prometheus/prometheus#18133) - \[BUGFIX] Agent: Fix memory leak caused by duplicate SeriesRefs being loaded as active series. [#17538](prometheus/prometheus#17538) - \[BUGFIX] Alerting: Fix alert state incorrectly resetting to pending when the FOR period is increased in the config file. [#18244](prometheus/prometheus#18244) - \[BUGFIX] Azure SD: Fix system-assigned managed identity not working when `client_id` is empty. [#18323](prometheus/prometheus#18323) - \[BUGFIX] Consul SD: Fix filter parameter not being applied to health service endpoint, causing Node and Node.Meta filters to be ignored. [#17349](prometheus/prometheus#17349) - \[BUGFIX] Kubernetes SD: Fix duplicate targets generated by `*DualStack` EndpointSlices policies. [#18192](prometheus/prometheus#18192) - \[BUGFIX] OTLP: Fix ErrTooOldSample being returned as HTTP 500 instead of 400 in PRW v2 histogram write paths, preventing infinite client retry loops. [#18084](prometheus/prometheus#18084) - \[BUGFIX] OTLP: Fix exemplars getting mixed between incorrect parts of a histogram. [#18056](prometheus/prometheus#18056) - \[BUGFIX] PromQL: Do not skip histogram buckets in queries where histogram trimming is used. [#18263](prometheus/prometheus#18263) - \[BUGFIX] Remote write: Fix `prometheus_remote_storage_sent_batch_duration_seconds` measuring before the request was sent. [#18214](prometheus/prometheus#18214) - \[BUGFIX] Rules: Fix alert state restoration when rule labels contain Go template expressions. [#18375](prometheus/prometheus#18375) - \[BUGFIX] Scrape: Fix panic when parsing bare label names without an equal sign in brace-only metric notation. [#18229](prometheus/prometheus#18229) - \[BUGFIX] TSDB: Fail early when `use-uncached-io` feature flag is set on unsupported environments. [#18219](prometheus/prometheus#18219) - \[BUGFIX] TSDB: Fall back to CLI flag values when retention is removed from config file. [#18200](prometheus/prometheus#18200) - \[BUGFIX] TSDB: Fix memory leaks in buffer pools by clearing reference fields before returning buffers to pools. [#17895](prometheus/prometheus#17895) - \[BUGFIX] TSDB: Fix missing mmap of histogram chunks during WAL replay. [#18306](prometheus/prometheus#18306) - \[BUGFIX] TSDB: Fix storage.tsdb.retention.time unit mismatch in file causing retention to be 1e6 times longer than configured. [#18200](prometheus/prometheus#18200) - \[BUGFIX] Tracing: Fix missing traceID in query log when tracing is enabled, previously only spanID was emitted. [#18189](prometheus/prometheus#18189) - \[BUGFIX] UI: Fix tooltip Y-offset drift when using multiple graph panels. [#18228](prometheus/prometheus#18228) - \[BUGFIX] UI: Update retention display in runtime info when config is reloaded. [#18200](prometheus/prometheus#18200)

kgeckhart requested a review from jesusvazquez as a code owner November 14, 2025 22:06

github-actions Bot added the stale label Jan 14, 2026

krajorama self-assigned this Feb 10, 2026

krajorama self-requested a review February 10, 2026 11:27

krajorama reviewed Feb 10, 2026

View reviewed changes

Comment thread tsdb/agent/db.go Outdated

Comment thread tsdb/agent/db.go

Comment thread tsdb/agent/db.go Outdated

github-actions Bot removed the stale label Feb 10, 2026

kgeckhart mentioned this pull request Feb 11, 2026

Replace our thin fork with the upstream implementation grafana/alloy#5507

Open

kgeckhart requested review from a team, bwplotka and codesome as code owners February 23, 2026 20:35

kgeckhart force-pushed the kgeckhart/agent-db-fix-series-leak branch from 3044c61 to 9e1e960 Compare February 23, 2026 20:37

kgeckhart added 6 commits February 25, 2026 06:03

Show the agent db can hold duplicate series by hash

53f8905

Signed-off-by: Kyle Eckhart <[email protected]>

Prevent duplicate SeriesRefs from being lost in db stripeSeries

920ee7f

Signed-off-by: Kyle Eckhart <[email protected]>

Drop default initialized value

8f68b4d

Signed-off-by: Kyle Eckhart <[email protected]>

More comments and only reset deleted if the new segment is larger

1e60d7f

Signed-off-by: Kyle Eckhart <[email protected]>

Manually manage db/rw to prevent windows test error

f4a1525

Signed-off-by: Kyle Eckhart <[email protected]>

Fix incorrect type from rebase

33afbb7

Signed-off-by: Kyle Eckhart <[email protected]>

kgeckhart force-pushed the kgeckhart/agent-db-fix-series-leak branch from 11f7253 to 33afbb7 Compare February 25, 2026 11:05

krajorama reviewed Feb 25, 2026

View reviewed changes

Comment thread tsdb/agent/db.go Outdated

Comment thread tsdb/agent/db_test.go Outdated

Comment thread tsdb/agent/series.go

kgeckhart added 2 commits February 25, 2026 16:52

Use Set in GetOrSet to enforce proper lock ordering

9c6b468

Signed-off-by: Kyle Eckhart <[email protected]>

Missing period and left over refactor

dca993e

Signed-off-by: Kyle Eckhart <[email protected]>

krajorama approved these changes Feb 26, 2026

View reviewed changes

krajorama merged commit 7cbe0e3 into prometheus:main Mar 10, 2026
33 checks passed

kgeckhart mentioned this pull request Mar 13, 2026

tsdb/agent: fix getOrCreate race #18292

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

tsdb/agent: Prevent duplicate SeriesRefs from being lost in stripeSeries#17538

tsdb/agent: Prevent duplicate SeriesRefs from being lost in stripeSeries#17538
krajorama merged 8 commits into
prometheus:mainfrom
kgeckhart:kgeckhart/agent-db-fix-series-leak

kgeckhart commented Nov 14, 2025

Uh oh!

krajorama commented Feb 10, 2026

Uh oh!

krajorama left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

kgeckhart commented Nov 14, 2025

Which issue(s) does the PR fix:

Does this PR introduce a user-facing change?

Uh oh!

krajorama commented Feb 10, 2026

Uh oh!

krajorama left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants