I think as a PoC, this is interesting. It would be interested to feed this to a test audience to see how this flies.

However, I have concerns if this should ever be generally accessible, even behind a feature flag. In general, introducing a whole new configuration section on the PromQL level rubs me the wrong way.

Feature flags are already putting conditions to how a given PromQL query is executed, but at least those are fairly simple binary decisions. This config is quite complex and deep, and it will essentially change how PromQL works in a fairly invasive fashion. For exploring a feature, that's fine. But I don't want to live in a world where people look at a PromQL expression, and the next question they ask is if they could see the PromQL engine config used to evaluate the expression.

I'm ~fine with configuring scrapes so that info metrics can be described there. That's just changing the input data on ingestion. (But of course, for that to work, we need to implement storing this information in the TSDB, and that's a much bigger barrier.)

Nice! I like the discussion as well.

2c:

But I don't want to live in a world where people look at a PromQL expression, and the next question they ask is if they could see the PromQL engine config used to evaluate the expression.

• I agree in principle, but our alternative is to have that "config" shipped via input data. But wouldn't this have similar result of "give me PromQL expression and also your scrape target /metrics text,so I know what are the target labels configured there?"
• Sounds like the long term plan is to have "configuration" being not needed as the input data would give you that (e.g. improve protocols). However, it's fine to start with hardcoded thing and perhaps some hidden config.
• Mechanisms looks fine, but do you mean you want to change your design to attach your labels by default with info function? That might be easy to use, but likely introduce overhead and potential surprises. It feels an explicit function might be better?

But wouldn't this have similar result of "give me PromQL expression and also your scrape target /metrics text,so I know what are the target labels configured there?"

I think it's a reasonable thing to say "the outcome of your PromQL expression depends mostly on what's in your TSDB". That's why I think if there is ingestion config that changes the date ultimately stored in the TSDB, that's nothing new and expected. Adding an additional config is a different thing. (Feature flags are already like that, but they are simple and inherently meant for experimental things.)

Mechanisms looks fine, but do you mean you want to change your design to attach your labels by default with info function? That might be easy to use, but likely introduce overhead and potential surprises. It feels an explicit function might be better?

Yeah, I am also in the "explicit is better than implicit" camp. I imagine the desire for "auto-adding" labels from info metrics comes from one problem with the "map all the labels on ingestion" approach: With the latter, you cannot change your mind later. If you realize you want to map more labels, you cannot easily apply that retroactively to historic data. But you can always change the query-time config and use it to query historic data.

I share Beorn's and Bartek's concern here, I don't think a query language should give different results depending on configuration. I believe the main use-case for the info() function is to unblock folks coming from OpenTelemetry background and it's already set in stone that their info metric is target_info. It's fine to have this hardcoded.

I understand that this would unblock with other info metrics, mostly coming from kube-state-metrics, but I'd love to see this being solved with an approach that doesn't require query language configuration. Bonus points if we don't even need the info() metric at all, e.g. storing info as metadata and make metadata part of query results 😬

I imagine the desire for "auto-adding" labels from info metrics comes from one problem with the "map all the labels on ingestion" approach: With the latter, you cannot change your mind later. If you realize you want to map more labels, you cannot easily apply that retroactively to historic data.

@beorn7 The second problem that arises is that when changing which OTel resource attributes to promote to labels, you also have a temporary doubling in series (i.e. old label set + new label set).

I understand that this would unblock with other info metrics, mostly coming from kube-state-metric.

@ArthurSens actually, the point of this approach is rather the different UX of not having to call info in queries. It's been a user feedback, that they don't want to have to change their queries. Potentially, this PR could also be changed to only support target_info with instance/job as the identifying labels. The added flexibility through configuration isn't really the point.

the point of this approach is rather the different UX of not having to call info in queries. It's been a user feedback, that they don't want to have to change their queries.

I think that's the core of what we are discussing here. Some users want to "automatically" change the result of their queries without actually changing the queries. This rubs some of us the wrong way, because now we make the outcome of a query depend on some external setting.

I think it is fine to "preprocess" the query in a wrapper, separate from core Prometheus. I assume the kind of users requiring "change the result of my query without changing the query" will work through some kind of UI that could do that for them.

To solve this within PromQL, maybe it could be piggybacked into the vaguely planned with feature, which currently is meant to just add additional label matchers to all selectors in their scope. But now we are talking about combining two not-yet-implemented features (info function and with), so maybe let's first see how the info function flies.

To solve this within PromQL, maybe it could be piggybacked into the vaguely planned with feature

Thanks @beorn7! I had never heard of with.

It is very vaguely planned. Hidden in a dev summit note from ages ago…

https://docs.google.com/document/d/11LC3wJcVk00l8w5P3oLQ-m3Y37iom6INAMEu2ZAGIIE/edit?pli=1#bookmark=id.m8eq3rv3yopu

Closing PR, as it's not what the team would like to proceed with :)