TSDB: use timestamps rather than WAL segment numbers to track how long deleted series should be retained in checkpoints#17029
Merged
Conversation
…g head GC Signed-off-by: Patryk Prus <[email protected]>
Signed-off-by: Patryk Prus <[email protected]>
…ations Signed-off-by: Patryk Prus <[email protected]>
pr00se
force-pushed
the
wal-checkpoint-dropped-samples
branch
from
August 8, 2025 18:12
6122727 to
0fea41e
Compare
Signed-off-by: Patryk Prus <[email protected]>
Signed-off-by: Patryk Prus <[email protected]>
Signed-off-by: Patryk Prus <[email protected]>
Member
|
/prombench main |
Contributor
|
⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️ Compared versions: After the successful deployment (check status here), the benchmarking results can be viewed at: Available Commands:
|
bboreham
reviewed
Aug 19, 2025
bboreham
approved these changes
Aug 19, 2025
bboreham
left a comment
bboreham
left a comment
Member
There was a problem hiding this comment.
LGTM. Although there are hundreds of lines of diffs, the main change, from holding a segment number to holding a timestamp, is not complicated.
Signed-off-by: Patryk Prus <[email protected]>
Member
|
Prombench seems identical. |
Member
|
/prombench cancel |
Contributor
|
Benchmark cancel is in progress. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue(s) does the PR fix:
Fixes #16859
As described in the above issue, the current implementation uses WAL segment numbers to indicate when a series record for a deleted or duplicate series should be dropped from a checkpoint. As the same code uses record timestamps to determine whether samples/histograms/etc. should be dropped, this can lead to series records getting discarded prematurely.
With this PR, we'll store the last timestamp that we should keep deleted/duplicate series records until instead. Gist of the solution:
walExpiriesfor all GCed series to themintused for the compaction (aka, the max timestamp that data for those deleted series might exist at)walExpiriesfor any duplicate series to the highest timestamp for any sample belonging to that serieswalExpiriesagainst themintbeing used for the checkpointThis will keep series records in WAL checkpoints until all the samples for those series have been removed.
Does this PR introduce a user-facing change?