Add --storage.tsdb.retention.percentage (#10287) by fatpat · Pull Request #12278 · prometheus/prometheus

fatpat · 2023-04-20T15:39:03Z

As discussed in #10287, this PR add the --storage.tsdb.retention.percentage.

It takes an uint between 0 (disabled) and 100.
if set, each time BeyondSizeRetention is called, storage.tsdb.rentention.size is temporary overwritten with the percentage of the current size of the partition on which the DB directory is stored.

roidelapluie · 2023-04-20T17:24:41Z

Can we make it work for windows too to keep feature parity?

fatpat · 2023-04-20T19:08:25Z

Can we make it work for windows too to keep feature parity?

yes it should be OK.

would it be possible to add a dependency to github.com/minio/minio/internal/disk as it seems complete and well maintained ?

or just continue to handle disk information directly from prometheus sources ?

fatpat · 2023-04-21T04:38:07Z

Can we make it work for windows too to keep feature parity?

yes it should be OK.

would it be possible to add a dependency to github.com/minio/minio/internal/disk as it seems complete and well maintained ?

or just continue to handle disk information directly from prometheus sources ?

forget it, I did not realized it was internal and cannot be used from prometheus.

roidelapluie · 2023-04-21T08:12:07Z

Did you copy code over from minio?

fatpat · 2023-04-21T09:22:41Z

Did you copy code over from minio?

only the strict necessary, but yes it's a piece of code that is inspired by minio code

roidelapluie · 2023-04-21T09:25:19Z

Minio is released under AGPL-3.0. We might get into troubled waters. Or you could check if the code was present in the Apache-2 releases of minio. We probably want attribution in the headers of the files as well.

fatpat · 2023-04-21T12:30:50Z

Minio is released under AGPL-3.0. We might get into troubled waters. Or you could check if the code was present in the Apache-2 releases of minio. We probably want attribution in the headers of the files as well.

No need I guess. Even if it looks like the same it's just a call to a windows syscall. cf doc from microsoft in the comment

fatpat · 2023-04-22T04:58:10Z

I still need to test it on windows

fatpat · 2023-04-22T05:03:51Z

I'm still not fully happy with the solution that I provide.

We clean up old blocks if the prom storage uses more that X% of the partition capacity. But If some other things are filling up the disk, it won't prevent the partition to get full and in this case prom is not working correctly.

I'm thinking about adding the possibility to remove blocks if free disk usage raises over a limit.

Let's say I want to remove old blocks if disk actual usage is over 90% no matter the size of the blocks. I just want to ensure prom will have spare space to write new data even if it means to delete old blocks, continuity of service first.

what do you think ?

roidelapluie · 2023-04-27T12:34:58Z

I think we should indeed take into consideration the size of the disk and not the size of the blocks, then remove enough blocks to get back below the threshold

roidelapluie · 2023-06-27T11:18:00Z

We have discussed this during the Prometheus bug scrub. We decided that we should only look at the ration between disk space used by Prometheus vs the disk size.

I will review this PR this week.

frittentheke · 2023-07-10T11:52:00Z

I like the simplicity of having a percentage value instead of some calculated number of GiB as a retention.
But if I may bluntly point at my issue about the lack of an "set it and forget it" approach (#11112) with the compaction and retention / expiry mechanism.

To me the most helpful feature would be to be able to just point Prometheus at a storage volume and tell it to use all of this this space and to do housekeeping / automagic expiry to never run out of space. Be there changes in rate or number of time series or whatever - Prometheus simply ensures the disk will not fill up 100%.

So take my comment back to this PR: Can I simply set 99% and be done with it?

roidelapluie · 2023-12-05T11:20:27Z

I still plan to review this and we agreed again during the bug scrub that we should only take care about the scenario where Prometheus has its own volume/partition.

There might be concerns with the size of the WAL and the size of the snapshots, but it is the same issue with the retention bytes. This PR should basically only set the retention bytes to the % of disk space.

bboreham · 2024-08-20T11:50:48Z

Hello from the bug-scrub! @roidelapluie do you think you will get back to this?

frittentheke · 2024-08-20T16:38:42Z

There might be concerns with the size of the WAL and the size of the snapshots, but it is the same issue with the retention bytes. This PR should basically only set the retention bytes to the % of disk space.

YES @roidelapluie, please see my comment / issue #11112 about this

bboreham · 2024-09-17T12:03:02Z

 	serverOnlyFlag(a, "storage.tsdb.retention.size", "Maximum number of bytes that can be stored for blocks. A unit is required, supported units: B, KB, MB, GB, TB, PB, EB. Ex: \"512MB\". Based on powers-of-2, so 1KB is 1024B.").
 		BytesVar(&cfg.tsdb.MaxBytes)

+	serverOnlyFlag(a, "storage.tsdb.retention.percentage", "Maximum percentage of the disk space that can be used to store blocks (prevails over storage.tsdb.retention.size).").


On first reading I interpreted "disk space" to mean "free space"; after looking more deeply I realize you implemented as a percentage of the total disk (partition) size.
Also "can be used to store blocks" might suggest that the head (WAL, chunks, etc.) is not included, whereas I believe it is.

Suggest to make this clearer; maybe add some commentary in the docs.

I can change the message, no problem. What would you suggest ?

bboreham · 2024-09-17T12:03:22Z

Hello again from the bug-scrub! We discussed in detail and this seems reasonable; there are a few merge conflicts to resolve.

fatpat · 2024-10-29T08:27:35Z

Hello again from the bug-scrub! We discussed in detail and this seems reasonable; there are a few merge conflicts to resolve.

done

bboreham · 2024-11-06T14:23:13Z

 	serverOnlyFlag(a, "storage.tsdb.retention.size", "Maximum number of bytes that can be stored for blocks. A unit is required, supported units: B, KB, MB, GB, TB, PB, EB. Ex: \"512MB\". Based on powers-of-2, so 1KB is 1024B.").
 		BytesVar(&cfg.tsdb.MaxBytes)

+	serverOnlyFlag(a, "storage.tsdb.retention.percentage", "Maximum percentage of the disk space that can be used to store blocks (prevails over storage.tsdb.retention.size).").


Suggested change

serverOnlyFlag(a, "storage.tsdb.retention.percentage", "Maximum percentage of the disk space that can be used to store blocks (prevails over storage.tsdb.retention.size).").

serverOnlyFlag(a, "storage.tsdb.retention.percentage", "Maximum percentage of disk used for TSDB storage (overrides storage.tsdb.retention.size).").

Then the more detailed commentary can go in docs/storage.md

frittentheke · 2024-11-06T15:28:04Z

With this PR seemingly moving I'd like to again (#12278 (comment)) mention the usability issue with setting e.g. "95%" and still running out of disk space due to #11112.

It would simply be AWESOME to have Prometheus be able to utilize all of the available space (e.g. running with this new flag set to 95% or even 99%) and not run out of space due to compaction.

So what are "safe" values for all those --storage.tsdb.retention.* settings that don't require complex calculations and constant adjustments (block size, compaction intervals, number of time series, churn, ...)

bboreham · 2025-08-26T11:23:07Z

Hello from the bug-scrub!

Seems we are nearly there; there is one review comment and one small merge conflict.

@fatpat if you're able to do this, please let us know, otherwise one of the maintainers can pick it up and clean up those things.

Signed-off-by: Jérôme LOYET <[email protected]>

fatpat · 2025-08-27T03:34:34Z

Hello from the bug-scrub!

Seems we are nearly there; there is one review comment and one small merge conflict.

@fatpat if you're able to do this, please let us know, otherwise one of the maintainers can pick it up and clean up those things.

Hi,

rebase's done.

Unfortunately I have no spare time to work on this soon. Feel free to take over, I'd love to see this PR merged. And if possible to take compaction into account would be a game changer for us too.

ldufr · 2026-02-17T07:34:42Z

After talking with @krajorama, I'll try to bring this to an end, but can't directly edit this PR (I'm not a maintainer), so following up in #18080.

frittentheke · 2026-02-17T07:54:55Z

Unfortunately I have no spare time to work on this soon. Feel free to take over, I'd love to see this PR merged. And if possible to take compaction into account would be a game changer for us too.

@ldufr will you be tackling the issue with compaction not taking into account when calculating storage percentage then?

ldufr · 2026-02-17T08:17:36Z

@frittentheke I wasn't going to tackle the issue together and I think it makes more sense here to do it separately.

roidelapluie · 2026-02-17T09:53:28Z

@frittentheke I wasn't going to tackle the issue together and I think it makes more sense here to do it separately.

Yes 2 different issues especially since that compation issue affects all the TSDB retention mechanisms (and is not as easy as it looks)

frittentheke · 2026-02-17T10:55:16Z

@frittentheke I wasn't going to tackle the issue together and I think it makes more sense here to do it separately.

Yes 2 different issues especially since that compation issue affects all the TSDB retention mechanisms (and is not as easy as it looks)

@roidelapluie after all there is my issue #11112 which is to track that one anyways.

roidelapluie · 2026-02-25T09:20:51Z

Thanks for the work put on this, #18080 has been merged, so I am closing this one.

fatpat requested a review from codesome as a code owner April 20, 2023 15:39

roidelapluie reviewed Apr 20, 2023

View reviewed changes

Comment thread util/runtime/statfs_uint32.go Outdated

fatpat force-pushed the retention_size_percentage branch from a7e9f3a to 8912e88 Compare April 20, 2023 19:04

fatpat force-pushed the retention_size_percentage branch 2 times, most recently from d794957 to 3b199aa Compare April 21, 2023 04:34

fatpat force-pushed the retention_size_percentage branch 2 times, most recently from fa11739 to a148c5a Compare April 21, 2023 07:13

fatpat force-pushed the retention_size_percentage branch 2 times, most recently from cadc74c to cf74ceb Compare April 21, 2023 09:20

fatpat force-pushed the retention_size_percentage branch 3 times, most recently from 5df8a81 to 3378275 Compare April 21, 2023 11:40

fatpat force-pushed the retention_size_percentage branch from 3378275 to d019d63 Compare April 21, 2023 19:25

roidelapluie self-assigned this Jun 27, 2023

nicolastakashi reviewed Jun 27, 2023

View reviewed changes

Comment thread cmd/prometheus/main.go

bboreham reviewed Sep 17, 2024

View reviewed changes

fatpat force-pushed the retention_size_percentage branch from d019d63 to 88d4be9 Compare October 29, 2024 08:24

fatpat requested a review from jesusvazquez as a code owner October 29, 2024 08:24

fatpat force-pushed the retention_size_percentage branch 2 times, most recently from 66b3ba1 to bf98acf Compare October 29, 2024 10:32

bboreham reviewed Nov 6, 2024

View reviewed changes

Add --storage.tsdb.retention.percentage (prometheus#10287)

a29537a

Signed-off-by: Jérôme LOYET <[email protected]>

fatpat force-pushed the retention_size_percentage branch from bf98acf to a29537a Compare August 27, 2025 03:18

bboreham added the help wanted label Jan 6, 2026

krajorama self-assigned this Jan 6, 2026

ldufr mentioned this pull request Feb 13, 2026

Add percentage based retention #18080

Merged

roidelapluie closed this Feb 25, 2026

	serverOnlyFlag(a, "storage.tsdb.retention.percentage", "Maximum percentage of the disk space that can be used to store blocks (prevails over storage.tsdb.retention.size).").
	serverOnlyFlag(a, "storage.tsdb.retention.percentage", "Maximum percentage of disk used for TSDB storage (overrides storage.tsdb.retention.size).").

Uh oh!

Conversation

fatpat commented Apr 20, 2023

Uh oh!

Uh oh!

roidelapluie commented Apr 20, 2023

Uh oh!

fatpat commented Apr 20, 2023

Uh oh!

fatpat commented Apr 21, 2023

Uh oh!

roidelapluie commented Apr 21, 2023

Uh oh!

fatpat commented Apr 21, 2023

Uh oh!

roidelapluie commented Apr 21, 2023

Uh oh!

fatpat commented Apr 21, 2023

Uh oh!

fatpat commented Apr 22, 2023

Uh oh!

fatpat commented Apr 22, 2023

Uh oh!

roidelapluie commented Apr 27, 2023

Uh oh!

roidelapluie commented Jun 27, 2023

Uh oh!

Uh oh!

frittentheke commented Jul 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

roidelapluie commented Dec 5, 2023

Uh oh!

bboreham commented Aug 20, 2024

Uh oh!

frittentheke commented Aug 20, 2024

Uh oh!

bboreham Sep 17, 2024

Choose a reason for hiding this comment

Uh oh!

fatpat Oct 29, 2024

Choose a reason for hiding this comment

Uh oh!

bboreham commented Sep 17, 2024

Uh oh!

fatpat commented Oct 29, 2024

Uh oh!

bboreham Nov 6, 2024

Choose a reason for hiding this comment

Uh oh!

frittentheke commented Nov 6, 2024

Uh oh!

bboreham commented Aug 26, 2025

Uh oh!

fatpat commented Aug 27, 2025

Uh oh!

ldufr commented Feb 17, 2026

Uh oh!

frittentheke commented Feb 17, 2026

Uh oh!

ldufr commented Feb 17, 2026

Uh oh!

roidelapluie commented Feb 17, 2026

Uh oh!

frittentheke commented Feb 17, 2026

Uh oh!

roidelapluie commented Feb 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

frittentheke commented Jul 10, 2023 •

edited

Loading