tsdb: drop deleted series from the WAL sooner#12297
Conversation
`head.deleted` holds the WAL segment in use at the time each series was removed from the head. At the end of `truncateWAL()` we will delete all segments up to `last`, so we can drop any series that were last seen in a segment at or before that point. Signed-off-by: Bryan Boreham <[email protected]>
|
Thanks for the detailed explanation in this and your other PR, I think I understand the original behaviour now. So this is my understanding; In So in this PR, the way we're deleting is the earliest/safest we could without any additional logic, but it's still not the earliest possible. Unless I am misunderstanding, as a safer alternative to your original PR, we could add code to track the last segment for which a sample was written to the WAL. At the time we truncate the WAL, if we're truncating up to segment id > series last segment id then we can delete the series completely (as in not write it's series record to the checkpoint). I've hacked that together here: d07b70a |
bboreham
changed the title
Signed-off-by: Bryan Boreham <[email protected]>
`head.deleted` holds the WAL segment in use at the time each series was removed from the head. At the end of `truncateWAL()` we will delete all segments up to `last`, so we can drop any series that were last seen in a segment at or before that point. Same change as in `tsdb/head.go`. Signed-off-by: Bryan Boreham <[email protected]>
|
Thanks to @rfratto for pointing out very similar code is in the I have applied the same change to use |
|
/prombench main |
|
I restarted the prombench Prometheus pods so they would reload the WAL. Head series peak is much reduced, as hoped - 9.56M instead of 17.68M: RSS peak is down to 28.6GB from 32.6GB: WAL checkpoint replay time came down from 60 seconds to 26 seconds. |
|
/prombench cancel Left it running a few more hours to see if the memory would equalise, but the one with smaller WAL checkpoint is consistently smaller. |
|
Benchmark cancel is in progress. |
|
It didn't help our WAL loading issues... still OOM's non stop |
This change causes deleted series to be deleted earlier based on the last segment involved in the checkpoint rather than the first segment involved in the checkpoint. This will then cause the next truncate to be more aggressive with dropped series, which should help free memory associated with deleted series in remote_write sooner. Mirrors prometheus/prometheus#12297 from @bboreham.
This change causes deleted series to be deleted earlier based on the last segment involved in the checkpoint rather than the first segment involved in the checkpoint. This will then cause the next truncate to be more aggressive with dropped series, which should help free memory associated with deleted series in remote_write sooner. Mirrors prometheus/prometheus#12297 from @bboreham.
5 participants
head.deletedholds the WAL segment in use at the time each series was removed from the head. At the end oftruncateWAL()we will delete all segments up tolast, so we can drop any series that were last seen in a segment at or before that point.Also, we can drop series from the
deletedmap up tolast, since all those files are now deleted.Improves #12286, but not a full fix.
Related to #12288.
Illustrated is a Prometheus TSDB that has been collecting data since 10:00 UTC.
WAL segments are named A, B, C, ...
Consider a series
foowhich received a few samples at 10:15, then stopped.The samples for series
fooare in WAL segment C.At approx 13:00, head compaction runs. A block is created from 10-12, and
that data is dropped from the head. Series
foois garbage-collected, butthe head notes in its 'deleted' map that it might be needed until WAL segment G has
been dropped.
A WAL checkpoint is created from the first two thirds of the segments, A-D.
WAL segments A-D are removed from disk.
At approx 15:00, head compaction runs again.
A WAL checkpoint is created covering segments E-H.
Now series
foois dropped from the checkpoint, since the 'deleted' map says we need it until G, but we will be deleting G.The 'deleted' map is cleaned of any series needed until segment 'H'.
Removing the series at 15:00 is an improvement on retaining it until 19:00 per the original issue #12286.
Let's also consider the case that was a problem for #12288: the samples are in segment J and we are truncating A-H:
The
deletedentry for samples would be J or later, so the series would be retained in the checkpoint, and in thedeletedmap.