Context
wlog.Checkpoint takes several parameters which govern the checkpointing process:
mint is the timestamp at which the WAL is being truncatedfirst is the id number of the first non-checkpoint WAL segmentlast is the id number of the last WAL segment that will be included in the checkpoint
keep is a function that takes a series ref and last and returns whether the series record is still necessary after WAL segment last is deleted
The checkpoint process then scans through the old checkpoint and WAL segments [first, last] and:
- for each series encountered, writes it into the new checkpoint if
keep returns true, and drops it otherwise
- for each other type of record encountered, writes the samples/histograms/etc. into the new checkpoint if their timestamp is
> mint, and drops it otherwise
Lastly it deletes WAL segments [first, last] from disk.
Issue
WAL segments are not time aligned, so mint will almost always fall in the middle of the timerange contained within some WAL segment. (In fact, WAL segments are not guaranteed to be non-overlapping, so it's possible for mint to fall in the middle of multiple WAL segment timeranges).
The problem occurs if a series is marked as not necessary after a WAL segment X (keep for the series returns false), but some samples for that series appear in segment X with timestamps > mint. The checkpoint will throw away the series reference, but copy over the samples > mint. The next time the WAL is replayed those samples will be missing a series reference and will be dropped.
Example
Details from an occurance in a production system:
- series
foo that has samples in the WAL through segment 3108, the last one at ts = 08:00:00.087Z
- the WAL segments contains records with the following time ranges:
...
segment 3106 has samples from 07:48:00Z to 07:56:09Z
segment 3107 has samples from 07:51:00Z to 07:59:25Z
segment 3108 has samples from 07:55:00Z to 08:00:04Z
...
- the checkpoint runs with:
mint = 08:00:00Zlast = 3110
The result is that the series record for foo is dropped but the sample at ts = 08:00:00.087Z is written into the checkpoint.
I've written a unit test that demonstrates this here. You can see the failure here.
In practice, #12286 actually prevents this from happening for series that are deleted from the head during routine garbage collection. However, since #16060 duplicate series encountered during WAL replay are similarly marked as "deleted" with their expiries set much closer to the actual WAL segment their final samples exist in, making this issue more likely to occur.
Context
wlog.Checkpointtakes several parameters which govern the checkpointing process:mintis the timestamp at which the WAL is being truncatedfirstis the id number of the first non-checkpoint WAL segmentlastis the id number of the last WAL segment that will be included in the checkpointkeepis a function that takes a series ref andlastand returns whether the series record is still necessary after WAL segmentlastis deletedThe checkpoint process then scans through the old checkpoint and WAL segments
[first, last]and:keepreturnstrue, and drops it otherwise> mint, and drops it otherwiseLastly it deletes WAL segments
[first, last]from disk.Issue
WAL segments are not time aligned, so
mintwill almost always fall in the middle of the timerange contained within some WAL segment. (In fact, WAL segments are not guaranteed to be non-overlapping, so it's possible formintto fall in the middle of multiple WAL segment timeranges).The problem occurs if a series is marked as not necessary after a WAL segment
X(keepfor the series returnsfalse), but some samples for that series appear in segmentXwith timestamps> mint. The checkpoint will throw away the series reference, but copy over the samples> mint. The next time the WAL is replayed those samples will be missing a series reference and will be dropped.Example
Details from an occurance in a production system:
foothat has samples in the WAL through segment3108, the last one atts = 08:00:00.087Zmint = 08:00:00Zlast = 3110The result is that the series record for
foois dropped but the sample atts = 08:00:00.087Zis written into the checkpoint.I've written a unit test that demonstrates this here. You can see the failure here.
In practice, #12286 actually prevents this from happening for series that are deleted from the head during routine garbage collection. However, since #16060 duplicate series encountered during WAL replay are similarly marked as "deleted" with their expiries set much closer to the actual WAL segment their final samples exist in, making this issue more likely to occur.