We're using the prometheus-emitter for metrics on druid 26.0.0. This is all working OK, metrics are flowing and all is well with the world (almost).
Query metrics are showing up as expected, but the prometheus endpoint never appears to timeout the metrics.
For example, we'll run a query, the metric for that query will show up (druid_query_wait_time as an example) and be scraped by prometheus. A minute/hour/day later, that same query metric is seemingly still being exposed by the prometheus emitter. Adding in some of the dimensions for query metrics (id being one), the cardinality is increased and it's causing the number of metrics to grow rapidly and causing stability issues.
It's all pretty standard prometheus-emitter deployments using the exporter strategy, so I'm wondering whether I'm missing something (maybe a configuration option for the emitter to reset the metrics it's exposing after a pre-defined period).
The http emitter has a druid.emitter.http.flushMillis setting which (I appreciate is pushing rather than scraping) is the behaviour I think we're missing
We're using the prometheus-emitter for metrics on druid 26.0.0. This is all working OK, metrics are flowing and all is well with the world (almost).
Query metrics are showing up as expected, but the prometheus endpoint never appears to timeout the metrics.
For example, we'll run a query, the metric for that query will show up (druid_query_wait_time as an example) and be scraped by prometheus. A minute/hour/day later, that same query metric is seemingly still being exposed by the prometheus emitter. Adding in some of the dimensions for query metrics (id being one), the cardinality is increased and it's causing the number of metrics to grow rapidly and causing stability issues.
It's all pretty standard prometheus-emitter deployments using the exporter strategy, so I'm wondering whether I'm missing something (maybe a configuration option for the emitter to reset the metrics it's exposing after a pre-defined period).
The http emitter has a druid.emitter.http.flushMillis setting which (I appreciate is pushing rather than scraping) is the behaviour I think we're missing