Repository navigation
Signal to Prometheus that a query should match something #11339
Description
Proposal
Use case. Why is this important?
When I run a query like foo > 0 and it returns an empty response there are two possible reasons for it:
- query conditions isn't met (
foovalue is<= 0) - there are no time series matching
foo
But there's no way to know why the query returns an empty response without running more queries to verify if foo is present or not.
In any environment with a large number of recording/alerting rules this will lead to dead rules that no longer match any time series and therefore do not work. It's usually less of a problem for recording rules but can cause alerting rules to silently stop working over time as scraped services evolve and exported metrics change names, labels etc.
There is absent() function one can use to add some checks around missing metrics, but it requires one to write an extra alert per each metric, which again can stop working over time, so it suffers from the same underlaying issue.
To be clear: there's nothing wrong with this behaviour and it makes perfect sense. The problem is that it makes maintaining rules more challenging.
There are tools that help with this, like https://github.com/cloudflare/pint (see also https://blog.cloudflare.com/monitoring-our-monitoring/) but the whole problem comes down to Prometheus not really knowing if empty result is ok or not.
Idea: we shouldn't change the fundamentals of how PromQL works, but we could add a new modifier that signals Prometheus that given expression should return something. Let's call it required here for now.
An example of how this would work in practice would be:
foo required > 0
If foo doesn't match anything required modifier would tell Prometheus to fail this query and return us an error.
Why is this useful? It would allow users to write recording/alerting rules in a way that ensures that they either work or return errors. This would help to avoid a situation where alerting rule silently stop working because they no longer match anything due to metric/labels renames and similar changes.
Activity
Adds required keyword to fails when a metric value doesn't exist (pro…
alexgreenbank commentedon Oct 8, 2024
Hello from the bug scrub.
Apologies for the delay in reviewing this issue.
This touches on a more fundamental issue of how to deal with missing metrics. We shall look to add this to the dev summit agenda to discuss this wider topic.
juliusv commentedon Feb 27, 2025
I've also occasionally thought about this over the years. The potential for a selector to both intentionally and unintentionally produce empty results (and in the latter case, to cause alerting rules to fail silently) is one of the biggest footguns of PromQL. Being able to add some kind of expectation that a selector shouldn't return an empty result would be great. You could even go further and allow more assertions like "expect exactly 1 series", "expect at least 4 series", "expect a certain shape of labels on the result data", and so on, but at some point it's overkill.
In terms of syntax, we have the choice of completely new syntactical modifiers like
requiredor just introducing a function likeassert_nonempty(foo). A function could be added more easily and may also be clearer to read in some cases. And I think it could make sense to have this kind of assertion for arbitrary expressions, because some conditions you cannot check by looking at individual selectors. Take for examplefoo / bar. Maybe you always want that ratio to return something, but its result could be empty due to a label mismatch even if bothfooandbarindividually return a non-empty result. So being able to sayassert_nonempty(foo / bar)could be useful as well.beorn7 commentedon Mar 31, 2025
We discussed this at the dev summit:
CONSENSUS: We want to make it more convenient to handle unexpectedly missing metrics in querying; we need a design document that answers the noted questions.
The noted questions are:
Detailed meeting notes
krajorama commentedon Jun 13, 2025
An idea that came up during #16578 : there are some cases where the data is there but we detect some kind of an issue with it (non monotonic histogram buckets, problems with native histograms, mixing float/histograms, etc). This led me to think that a generic solution would be to allow alerting/failing on the presence of annotations. Which would mean that the the new new operator/function in the above discussion would just pass what it gets and if it doesn't get anything it would generate an annotation.
In the alert/recording rule configuration you could chose what to do with annotations.
beorn7 commentedon Jun 14, 2025
As annotations are kind-of "soft fails", maybe we could at the same time create a similar option to choose what to do if the rule evaluation fails.
Currently, this happens essentially via meta-monitoring, using the
prometheus_rule_evaluation_failures_totalmetric, alerting the operator of the Prometheus server rather than the owner of the rule. You could do some routing via rule group, but that would be a separately maintained thing.Maybe some rule option where you could provide labels to be attached to "synthetic" alerts created if the rule evaluation fails and/or creates annotations.