tsdb: Add `series_state.json` file to `wal/` directory to track state by RushabhMehta2005 · Pull Request #18303 · prometheus/prometheus

RushabhMehta2005 · 2026-03-15T14:41:33Z

This PR starts off the work in making scraping start in parallel with WAL replay, under the guidance of @codesome and @bboreham as my LFX mentors.
I have implemented file creation and file updation logic. The file is created if it does not already exist, it is updated by a background goroutine every 1 second. The last wal segment and last series ID are updated, and if prometheus is shutting down cleanly, the "clean_shutdown" field is also set to true.
Any suggestions about code quality or design are most welcome.

Which issue(s) does the PR fix:

NA

Does this PR introduce a user-facing change?

NONE

Signed-off-by: Rushabh Mehta <[email protected]>

RushabhMehta2005 · 2026-03-15T14:46:43Z

Force push was necessary as I forgot to sign the first commit, my bad.

Signed-off-by: Rushabh Mehta <[email protected]>

RushabhMehta2005 · 2026-03-16T16:19:10Z

Not sure why the tests are failing.

bboreham

See if you can understand why CI tests are failing - TestWALSegmentSizeOptions_AppendV2.

I made a few comments below, but they're stylistic, not really important compared to the test failures.

bboreham · 2026-03-17T18:48:14Z

+	path := filepath.Join(h.wal.Dir(), seriesStateFilename)
+	tmpPath := path + ".tmp"
+
+	if err := os.WriteFile(tmpPath, b, 0o666); err != nil {


666 is read and write permissions for everybody. Is this incorrect or too privileged? This is the default when we create a file.

bboreham · 2026-03-17T18:48:32Z

+	if err := os.Rename(tmpPath, path); err != nil {
+		h.logger.Warn("Failed to rename series state file", "err", err)


what would this failure mean?

My naive thinking makes me want to say that this will never actually happen. But if it does happen, we just lose 1 second of updated information.

bboreham · 2026-03-17T18:50:01Z

+	}
+
+	path := filepath.Join(h.wal.Dir(), seriesStateFilename)
+	tmpPath := path + ".tmp"


This could just be path. Also could be on the line above.

Not sure what you mean, do you want me to not create a temporary file?

bboreham · 2026-03-17T18:51:46Z

+		CleanShutdown:  cleanShutdown,
+	}
+
+	b, err := json.Marshal(state)


Consider marshalling straight to the file, rather than marshalling to memory then writing to file.

Alright, understood.

bboreham · 2026-03-18T11:44:15Z

I think, before merging this extra file and check loop, we'd want to put it behind a feature-flag so it doesn't affect most people.

Check previous PRs to see how feature flags are added.

Signed-off-by: Rushabh Mehta <[email protected]>

RushabhMehta2005 · 2026-03-19T11:40:49Z

See if you can understand why CI tests are failing - TestWALSegmentSizeOptions_AppendV2.

A channel I created for sending the signal to close the goroutine was being closed twice, leading to panic. Should be fixed now.

RushabhMehta2005 · 2026-03-19T12:05:53Z

I have responded to the review comments, put this PR behind "fast-startup" feature flag, and now most of the CI is passing. The failing CI test has to do with something not able to allocate enough memory, not sure why that would be the case because of this PR.

bboreham

lgtm, one style point

bboreham · 2026-03-20T08:59:36Z

 	h.stats.WALReplayStatus.Current = current
 }
+
+// Name of the file used to store the state.


Another style point: I would put all this in head_wal.go rather than head.go.

Understood, will do.

RushabhMehta2005 · 2026-03-20T09:34:35Z

After this final change, should I squash the commits to keep them clean?

Signed-off-by: Rushabh Mehta <[email protected]>

codesome

Nice work! Most are minor comments. Looking good in general.

Can you also add a simple unit test for this one? You can refer existing unit tests for some reference code. It can be as simple as

Start the Head block with this flag enabled.
Add 1 sample to the Head.
Wait for 1.1 second and check for the existence of file on disk and verify its contents.
Close the Head and check the contents of the file again.

codesome · 2026-03-21T17:47:39Z

After this final change, should I squash the commits to keep them clean?

I can squash merge the PR so it's not needed.

Signed-off-by: Rushabh Mehta <[email protected]>

RushabhMehta2005 · 2026-03-22T07:17:59Z

@codesome Have added the unit test needed, and fixed the nits. Now moving onto PR 2.

codesome

Great work!

RushabhMehta2005 · 2026-03-24T06:38:37Z

Let's go! Thank you for the merge.

##### [\`v3.11.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.11.0) - \[CHANGE] Hetzner SD: The `__meta_hetzner_datacenter` label is deprecated for the role `robot` but kept for backward compatibility, use the `__meta_hetzner_robot_datacenter` label instead. For the role `hcloud`, the label is deprecated and will stop working after the 1 July 2026. [#17850](prometheus/prometheus#17850) - \[CHANGE] Hetzner SD: The `__meta_hetzner_hcloud_datacenter_location` and `__meta_hetzner_hcloud_datacenter_location_network_zone` labels are deprecated, use the `__meta_hetzner_hcloud_location` and `__meta_hetzner_hcloud_location_network_zone` labels instead. [#17850](prometheus/prometheus#17850) - \[CHANGE] Promtool: Redirect debug output to stderr to avoid interfering with stdout-based tool output. [#18346](prometheus/prometheus#18346) - \[FEATURE] AWS SD: Add Elasticache Role. [#18099](prometheus/prometheus#18099) - \[FEATURE] AWS SD: Add RDS Role. [#18206](prometheus/prometheus#18206) - \[FEATURE] Azure SD: Add support for Azure Workload Identity authentication method. [#17207](prometheus/prometheus#17207) - \[FEATURE] Discovery: Introduce `prometheus_sd_last_update_timestamp_seconds` metric to track the last time a service discovery update was sent to consumers. [#18194](prometheus/prometheus#18194) - \[FEATURE] Kubernetes SD: Add support for node role selectors for pod roles. [#18006](prometheus/prometheus#18006) - \[FEATURE] Kubernetes SD: Introduce pod-based labels for deployment, cronjob, and job controller names: `__meta_kubernetes_pod_deployment_name`, `__meta_kubernetes_pod_cronjob_name` and `__meta_kubernetes_pod_job_name`, respectively. [#17774](prometheus/prometheus#17774) - \[FEATURE] PromQL: Add `</` and `>/` operators for trimming observations from native histograms. [#17904](prometheus/prometheus#17904) - \[FEATURE] PromQL: Add experimental `histogram_quantiles` variadic function for computing multiple quantiles at once. [#17285](prometheus/prometheus#17285) - \[FEATURE] TSDB: Add `storage.tsdb.retention.percentage` configuration to configure the maximum percent of disk usable for TSDB storage. [#18080](prometheus/prometheus#18080) - \[FEATURE] TSDB: Add an experimental `fast-startup` feature flag that writes a `series_state.json` file to the WAL directory to track active series state across restarts. [#18303](prometheus/prometheus#18303) - \[FEATURE] TSDB: Add an experimental `st-storage` feature flag. When enabled, Prometheus stores ingested start timestamps (ST, previously called Created Timestamp) from scrape or OTLP in the TSDB and Agent WAL, and exposes them via Remote Write 2. [#18062](prometheus/prometheus#18062) - \[FEATURE] TSDB: Add an experimental `xor2-encoding` feature flag for the new TSDB block float sample chunk encoding that is optimized for scraped data and allows encoding start timestamps. [#18062](prometheus/prometheus#18062) - \[ENHANCEMENT] HTTP client: Add AWS `external_id` support for sigv4. [#17916](prometheus/prometheus#17916) - \[ENHANCEMENT] Kubernetes SD: Deduplicate deprecation warning logs from the Kubernetes API to reduce noise. [#17829](prometheus/prometheus#17829) - \[ENHANCEMENT] TSDB: Remove old temporary checkpoints when creating a Checkpoint. [#17598](prometheus/prometheus#17598) - \[ENHANCEMENT] UI: Add autocomplete support for experimental `first_over_time` and `ts_of_first_over_time` PromQL functions. [#18318](prometheus/prometheus#18318) - \[ENHANCEMENT] Vultr SD: Upgrade govultr library from v2 to v3 for continued security patches and maintenance. [#18347](prometheus/prometheus#18347) - \[PERF] PromQL: Improve performance and reduce heap allocations in joins (VectorBinop)/And/Or/Unless. [#17159](prometheus/prometheus#17159) - \[PERF] PromQL: Partially address performance regression in native histogram aggregations due to using `KahanAdd`. [#18252](prometheus/prometheus#18252) - \[PERF] Remote write: Optimize WAL watching used for RW sending to reuse internal buffers. [#18250](prometheus/prometheus#18250) - \[PERF] TSDB: Optimize LabelValues intersection performance for matchers. [#18069](prometheus/prometheus#18069) - \[PERF] UI: Skip restacking on hover in stacked series charts. [#18230](prometheus/prometheus#18230) - \[BUGFIX] AWS SD: Fix EC2 SD ignoring the configured `endpoint` option, a regression from the AWS SDK v2 migration. [#18133](prometheus/prometheus#18133) - \[BUGFIX] AWS SD: Fix panic in EC2 SD when DescribeAvailabilityZones returns nil ZoneName or ZoneId. [#18133](prometheus/prometheus#18133) - \[BUGFIX] Agent: Fix memory leak caused by duplicate SeriesRefs being loaded as active series. [#17538](prometheus/prometheus#17538) - \[BUGFIX] Alerting: Fix alert state incorrectly resetting to pending when the FOR period is increased in the config file. [#18244](prometheus/prometheus#18244) - \[BUGFIX] Azure SD: Fix system-assigned managed identity not working when `client_id` is empty. [#18323](prometheus/prometheus#18323) - \[BUGFIX] Consul SD: Fix filter parameter not being applied to health service endpoint, causing Node and Node.Meta filters to be ignored. [#17349](prometheus/prometheus#17349) - \[BUGFIX] Kubernetes SD: Fix duplicate targets generated by `*DualStack` EndpointSlices policies. [#18192](prometheus/prometheus#18192) - \[BUGFIX] OTLP: Fix ErrTooOldSample being returned as HTTP 500 instead of 400 in PRW v2 histogram write paths, preventing infinite client retry loops. [#18084](prometheus/prometheus#18084) - \[BUGFIX] OTLP: Fix exemplars getting mixed between incorrect parts of a histogram. [#18056](prometheus/prometheus#18056) - \[BUGFIX] PromQL: Do not skip histogram buckets in queries where histogram trimming is used. [#18263](prometheus/prometheus#18263) - \[BUGFIX] Remote write: Fix `prometheus_remote_storage_sent_batch_duration_seconds` measuring before the request was sent. [#18214](prometheus/prometheus#18214) - \[BUGFIX] Rules: Fix alert state restoration when rule labels contain Go template expressions. [#18375](prometheus/prometheus#18375) - \[BUGFIX] Scrape: Fix panic when parsing bare label names without an equal sign in brace-only metric notation. [#18229](prometheus/prometheus#18229) - \[BUGFIX] TSDB: Fail early when `use-uncached-io` feature flag is set on unsupported environments. [#18219](prometheus/prometheus#18219) - \[BUGFIX] TSDB: Fall back to CLI flag values when retention is removed from config file. [#18200](prometheus/prometheus#18200) - \[BUGFIX] TSDB: Fix memory leaks in buffer pools by clearing reference fields before returning buffers to pools. [#17895](prometheus/prometheus#17895) - \[BUGFIX] TSDB: Fix missing mmap of histogram chunks during WAL replay. [#18306](prometheus/prometheus#18306) - \[BUGFIX] TSDB: Fix storage.tsdb.retention.time unit mismatch in file causing retention to be 1e6 times longer than configured. [#18200](prometheus/prometheus#18200) - \[BUGFIX] Tracing: Fix missing traceID in query log when tracing is enabled, previously only spanID was emitted. [#18189](prometheus/prometheus#18189) - \[BUGFIX] UI: Fix tooltip Y-offset drift when using multiple graph panels. [#18228](prometheus/prometheus#18228) - \[BUGFIX] UI: Update retention display in runtime info when config is reloaded. [#18200](prometheus/prometheus#18200)

RushabhMehta2005 requested review from a team, bwplotka, codesome, jesusvazquez and krajorama as code owners March 15, 2026 14:41

RushabhMehta2005 added 2 commits March 15, 2026 20:15

Add series_state.json file creation and updation logic.

d36e795

Signed-off-by: Rushabh Mehta <[email protected]>

Make comments follow the guidelines.

9bc8071

Signed-off-by: Rushabh Mehta <[email protected]>

RushabhMehta2005 force-pushed the parallel-wal-replay branch from c1e2f90 to 9bc8071 Compare March 15, 2026 14:46

Fix linter complaints

e69c5de

Signed-off-by: Rushabh Mehta <[email protected]>

bboreham reviewed Mar 17, 2026

View reviewed changes

RushabhMehta2005 added 3 commits March 19, 2026 13:15

Put PR behind feature flag fast-startup

1ec9f03

Signed-off-by: Rushabh Mehta <[email protected]>

Marshal updated information to file directly

7049dd7

Signed-off-by: Rushabh Mehta <[email protected]>

Fix linter failures

887c471

Signed-off-by: Rushabh Mehta <[email protected]>

bboreham approved these changes Mar 20, 2026

View reviewed changes

Move series state code from head.go to head_wal.go

d1d7cea

Signed-off-by: Rushabh Mehta <[email protected]>

codesome reviewed Mar 21, 2026

View reviewed changes

RushabhMehta2005 added 2 commits March 22, 2026 11:37

Fix nits

2a241ff

Signed-off-by: Rushabh Mehta <[email protected]>

Add unit test

0b5a013

Signed-off-by: Rushabh Mehta <[email protected]>

This was referenced Mar 22, 2026

tsdb: Fast startup via bounded WAL reverse scan RushabhMehta2005/prometheus#1

Closed

tsdb: Find the last series ID on startup from the last series id file and WAL scan #18333

Merged

codesome approved these changes Mar 24, 2026

View reviewed changes

codesome merged commit df61021 into prometheus:main Mar 24, 2026
33 checks passed

This was referenced Apr 3, 2026

tsdb: Refactor Head.Init(...) for fast startup #18438

Closed

tsdb: Refactor Head.Init(...) for fast startup #18442

Open

github-actions Bot mentioned this pull request Apr 20, 2026

Update prom/prometheus Docker tag to v3.11.0 claytono/infra#2051

Merged

1 task

anxkhn mentioned this pull request Jul 8, 2026

docs: document the fast-startup feature flag #19130

Closed

		if err := os.Rename(tmpPath, path); err != nil {
		h.logger.Warn("Failed to rename series state file", "err", err)

Uh oh!

Conversation

RushabhMehta2005 commented Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue(s) does the PR fix:

Does this PR introduce a user-facing change?

Uh oh!

RushabhMehta2005 commented Mar 15, 2026

Uh oh!

RushabhMehta2005 commented Mar 16, 2026

Uh oh!

bboreham left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bboreham commented Mar 18, 2026

Uh oh!

RushabhMehta2005 commented Mar 19, 2026

Uh oh!

RushabhMehta2005 commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bboreham left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RushabhMehta2005 commented Mar 20, 2026

Uh oh!

codesome left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codesome commented Mar 21, 2026

Uh oh!

RushabhMehta2005 commented Mar 22, 2026

Uh oh!

codesome left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

RushabhMehta2005 commented Mar 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

RushabhMehta2005 commented Mar 15, 2026 •

edited

Loading

RushabhMehta2005 commented Mar 19, 2026 •

edited

Loading