-
Notifications
You must be signed in to change notification settings - Fork 852
Add label-filter optimization for multi-metric queries #3253
Description
Is your feature request related to a problem? Please describe.
Today, multi-metric queries like node_used_bytes{$LABELS1} / node_total_bytes{$LABELS2} appear to fetch all the series matching the numerator and all the series fetching the denominator, even if the intersection of $LABELS1 and $LABELS2 (and therefore, result of the expression) only produces one (or a few) series. This is especially costly if either labelset is the empty set.
I've found that users tend to be quite lazy and think they can get away with only putting their label selector in one part of the query. After all, their query gives the right answer! But the query evaluation takes 10-100x longer than needed as way more data than is necessary is fetched.
Describe the solution you'd like
When the optimizer can determine in advance that one (or an intersection of more than one) labelset in the query is "controlling" on the output, it should use that labelset for lookups on all queries.
It's possible to start simple here and add sophistication over time:
metric1{labels} (arith|comp)_binary_op metric2->labelsis controllingmetric1{labels1} (arith|comp)_binary_op metric2{labels2}-> a syntheticlabelsthat combines all non-repeated keys fromlabels1andlabels2is controlling.- One step further: a key in both
labels1andlabels2is inlabelsas the positive-lookahead of the values fromlabels1andlabels2. - This can be extended for any number of arithmetic or comparison binary operators
- This can be extended for
andas well.
Describe alternatives you've considered
User training is hard and is not working consistently
There may be broader optimization work possible that could subsume this suggestion
Additional context
@siebenmann describes this problem on his blog in thoughtful detail: https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusLabelNonOptimization