Proposal
Hey team! 👋 I'm here to ask whether you think it's a good idea to make Prometheus clean up leftover checkpoint.*.tmp directories during startup.
We're currently defering their cleanup in cases where Checkpoint() may exit early due to an error, but this block won't get called if Prometheus OOMs while reading a segment.
While looking at the way we handle block tmp dirs, it has already been suggested in a comment by @bwplotka.
|
// Block dir suffixes to make deletion and creation operations atomic. |
|
// We decided to do suffixes instead of creating meta.json as last (or delete as first) one, |
|
// because in error case you still can recover meta.json from the block content within local TSDB dir. |
|
// TODO(bwplotka): TSDB can end up with various .tmp files (e.g meta.json.tmp, WAL or segment tmp file. Think |
|
// about removing those too on start to save space. Currently only blocks tmp dirs are removed. |
Use case. Why is this important?
This would be useful in preventing users' disk usage from blowing up and also enable us to be a good citizen in a shared system by cleaning up useless leftover data.
Proposal
Hey team! 👋 I'm here to ask whether you think it's a good idea to make Prometheus clean up leftover checkpoint.*.tmp directories during startup.
We're currently defering their cleanup in cases where Checkpoint() may exit early due to an error, but this block won't get called if Prometheus OOMs while reading a segment.
While looking at the way we handle block tmp dirs, it has already been suggested in a comment by @bwplotka.
prometheus/tsdb/db.go
Lines 54 to 58 in 0df3489
Use case. Why is this important?
This would be useful in preventing users' disk usage from blowing up and also enable us to be a good citizen in a shared system by cleaning up useless leftover data.