What did you do?

(Some context: I was asked about PromQL info: input to histogram_quantile needed to be fixed for monotonicity … warning for a particular query. While investigating, I didn't find anything wrong with the data. However, the culprit was rate() function.)

For some timeseries, rate() function breaks histogram bucket monotonicity. When used in histogram_quantile(), this then leads to a PromQL info message, which points to documentation. If you dig deeper, there's quite an alarming mention of invalid data:

The latter is evidence for an actual issue with the input data and is therefore flagged with an informational annotation reading input to histogram_quantile needed to be fixed for monotonicity. If you encounter this annotation, you should find and remove the source of the invalid data.

What did you expect to see?

It seems quite common to use histogram_quantile() together with rate(). Ideally, rate() would not break histogram bucket monotonicity – though probably it's not an easy change. Otherwise, maybe the PromQL info message or documentation needs some adjustment, mentioning the possibility that the data is fine and the message is a false positive caused by rate() function.

What did you see instead? Under which circumstances?

See this commit for a small test which displays how higher histogram bucket might end up with lower rate() value.

System information

No response

Prometheus version

No response

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response