tsdb: remove deleted series from WAL checkpoint without delay#12288
tsdb: remove deleted series from WAL checkpoint without delay#12288bboreham wants to merge 1 commit into
Conversation
Remove the `deleted` map that retained deleted series for 3 head
compaction cycles; it is not needed.
This should significantly improve the time to read the WAL, and reduce
memory usage for remote-write.
To illustrate, pictured below is a Prometheus TSDB that has been
collecting data since 10:00 UTC. WAL segments are named A, B, C, ...
```
10:00 12:00
Head ┌───────────────────┐
└───────────────────┘
WAL A- B- C- D- E-- F- G-
```
At approx 13:00, head compaction runs. A block is created from
10:00-12:00, and that data is dropped from the head. Any series which
stopped before 12:00 is garbage-collected from the head.
A WAL checkpoint is created from the first two thirds of the segments,
A-D. All samples before 12:00 are excluded. WAL segments A-D are removed
from disk.
```
10:00 12:00
Head ┌──────┐
└──────┘
Block ┌────────────┐
└────────────┘
WAL E-- F- G-
Checkpoint X
```
There is no need to remember any series deleted from the head in the WAL
checkpoint, since the condition to garbage-collect a series is that it
has no samples after 12:00.
Signed-off-by: Bryan Boreham <[email protected]>
jesusvazquez
left a comment
jesusvazquez
left a comment
There was a problem hiding this comment.
I found this useful to understand why this deleted map existed dfed85e
There is no need to remember any series deleted from the head in the WAL checkpoint, since the condition to garbage-collect a series is that it has no samples after 12:00.
After reading @bboreham 's explanation and going through the code I think it makes sense to no longer track these.
I'm also thinking that if a sudden restart / out of memory where to happen, we could only rely on whatever series refs are on the WAL since these deleted memory map would also dissapear so to me there is no point in making the live WAL checkpointing more complex than the worst case scenario.
I think this scenario did not play out as you suggest. On restart the series would be loaded from the WAL checkpoint, then get removed by Line 575 in 5442d7e Lines 1500 to 1502 in bae9a21 (and they would be delayed another 3 cycles from being fully removed from the WAL) |
|
@codesome pointed out there can be samples in the 1/3 of WAL not being cleaned up, but still the series has been removed from the head. In terms of the example, this means the samples came after 12:00 but also in the last 1/3 of the WAL. There are 12 WAL segments from A to L, so 8 of them A-H would be deleted by truncation. |
|
I'm not understanding why the initial commit dfed85e was needed:
what samples? we deleted them all for that series On top of that, to counter act the scenario Ganesh pointed out in Bryan's latest comment, maybe we should consider checkpointing/truncating the WAL based on the overall time range rather than just WAL segment? it's just the segment index we use to decide whether we delete the segment or not: https://github.com/prometheus/prometheus/blob/main/tsdb/head.go#L1201-L1208 |
I think you can only do that by scanning the entire WAL and rewriting to remove deleted series. Note timestamps may be supplied by targets, so it's possible that each WAL segment has an overlapping time range with the one before. |
Okay yes, this definitely complicates things. I was thinking we could just introduce another record type that has metadata
yes |
|
Closing this; I think #12297 is workable now and a better solution will take time. |
3 participants
Remove the
deletedmap that retained deleted series for 3 head compaction cycles; it is not needed.This should significantly improve the time to read the WAL, and reduce memory usage for remote-write.
Fixes #12286
To illustrate, pictured below is a Prometheus TSDB that has been collecting data since 10:00 UTC. WAL segments are named A, B, C, ...
At approx 13:00, head compaction runs. A block is created from 10:00-12:00, and that data is dropped from the head. Any series which stopped before 12:00 is garbage-collected from the head.
A WAL checkpoint is created from the first two thirds of the segments, A-D. All samples before 12:00 are excluded. WAL segments A-D are removed from disk.
There is no need to remember any series deleted from the head in the WAL checkpoint, since the condition to garbage-collect a series is that it has no samples after 12:00.