I'm creating a separate, hopefully more focused (and civil) issue in an attempt to start a discussion on the problems (as seen by me and a number of others) with and possible solutions for rate() and increase().

First, let me start by acknowledging that considering Prometheus' self-imposed constraints -- in particular having foo[5m] in the context of rate(foo[5m]) only produce the data points that are strictly contained in the 5 minute range -- the current rate() and increase() implementations are essentially the most comprehensive solution for the particular problem.

That being said, I believe the core problems/limitations in the implementation stem from the constrained definition of what is a time range and an only slightly more generous definition (i.e. including the last point before the range) would provide a number of significant improvements. Why/how does including the last point before the range make any sense? Well, you could look at it as "the (rate of) increase between the value of the timeseries X minutes ago and now". And just as the current value of a timeseries is the last collected/evaluated point, so the value of the timeseries X seconds ago is the last collected/evaluated point at that time (i.e. the one immediately preceding the start of the range).

Or looking at it another way, if we consider a scrape interval of 1m (and, for now, no missed scrapes or series beginning/ending), a 5m rate calculated "now" should cover the last 5m (plus jitter) starting from the last point collected, just as the value of the timeseries "now" is the last point collected.

Moving on to the possible benefits of expanding the range to include the one extra point (in the past, not in the future):

1. Evaluating a rate/increase over the last X minutes every X minutes without any loss of data. Currently the options are either (a) compute a rate over significantly more than X minutes (which results in rates being averaged out over unnecessarily long ranges); or (b) compute the rate over X minutes + scrape_interval, which will basically give you the actual rate over X minutes, but requires you to be consistently aware of both the evaluation and scrape intervals.

2. rate() and increase() evaluations at the resolution of the underlying data, later aggregatable over arbitrarily long time ranges. Going back to the 1m scrape interval example, one would be able to compute a rate over 1m every minute, then, in a console/dashboard use avg_over_time() to obtain rates over any number of minutes/hours/days (and the same with increase and sum_over_time). Currently the rule of thumb is to calculate rates over 2.5-3.5x the scraping interval every 2 scraping intervals, resulting in (a) unnecessarily low resolution and (b) every resulting rate covering anywhere between 1 and 3 actual counter increases (2-4 points) with each increase arbitrarily included in either 1 or 2 rates. To be fair, the current implementation could be used to evaluate a rate every 1m, but the specified range would have to be 2m, which is (to say the least) counterintuitive.

3. On-the-fly query_range rate/increase calculations with each increase only included in one rate/increase point and without data loss (which is what Grafana provides support for). Currently there are 2 issues that prevent this from working: (a) one needs to be aware of the scrape interval in order to bump the range by that much; and (b) neither Grafana nor Prometheus support the kind of time arithmetic necessary to query rate(foo[1h+1m]). (Not to mention the additional /(1h+1m) * 1h necessary to get a somewhat accurate increase).

4. Finally, and this is a judgment call, one could get integer increases from integer counters. Particularly in the case of rare events, it is somewhat jarring to see increases of 1.5 or 1.2 or 2.4 requests. This would work perfectly for ranges that are multiples (including 1x) of the scraping interval. If the range is not a multiple of the scraping interval (and I am actually curious why someone would do this to begin with), then a smoothly increasing counter will result in aliasing. The alternative is to do what the current implementation does, i.e. to compute the rate first (which would be ~constant for a smoothly increasing counter) and multiply it by the range. It would yield essentially the same result for the multiple of the scrape interval case (as the difference between the timestamps of the 2 ends would be ~= to the range) and smooth (but not integer) increases for the arbitrary range case.

To be clear, I am aware of the fact that timeseries don't come perfectly evenly spaced, that collections are missed at times and counters begin/end/restart, but I can't think of a particular case where my proposed improvement would behave differently from the current implementation. Please let me know if you think I'm missing something obvious.

Now let's move on to the downsides:

1. This would require a change to the definition of what a range is. True, but it can (and should) be done in such a way as to provide an extra point before the actual range, which would then be ignored in the vast majority of cases. E.g. the <operation>_over_time should definitely not make use of this extra point, as in my example above avg_over_time(foo:rate_1m[5m]) would already include the increase between foo offset 5m and the first point in the range. It could even be implemented as a separate field from the list of points in the range. I understand this is a controversial change, but in my view the benefits are worth it. And it is actually more consistent with the way instant values work.

2. Introducing yet another rate()/increase() function or replacing the existing ones and breaking backward compatibility is not ideal. Fully agreed, but again (personal opinion, confirmed by a handful of others) I believe the benefits are worth it. It could be pushed back to Prometheus 3.0 (if there is a plan for that) or hidden behind a flag or really whatever makes sense. I think it would improve the experience of lots of users, if made available in any way.

3. One other possible argument against this approach is "if the last point before the range is almost one scrape interval away from the range, why should it count as much as the others/at all?". The exact same argument could be made against the current implementation: "what if the first point in the range is almost one scrape interval away from the start, does it make sense to extrapolate that far?". Or, looking at it another way, if it's OK for the instant value of a timeseries to be one scrape interval old, then it should be similarly OK for a rate over whatever interval to be one scrape interval old.

Thanks, and I really hope this leads to a discussion of the benefits/downsides of the proposal or, more generally, on how rate()/interval() could be improved.