This should improve in the future once OTel SDKs start generating service.instance.id: open-telemetry/semantic-conventions#312

Even after that PR, service.instance.id is only recommended, right? Is there a plan to make service.instance.id required for OTel SDK clients? Also, it doesn't change the situation for non-SDK clients AFAICT.

Thats correct. SDKs will start generating it by default, but there are ways to get around it, and the collector won't always send it. It will get better, but not be solved entirely

We need to resolve this in the entity WG somehow (coordinating with prometheus).

Ideally we have a way to give you a grouping (job) and unique identifier (instance) for any piece of telemetry you receive from OTEL.

At a minimum, I think we should be able to provide a unique identifier (instance), where "job/instance" matching should always work.

As a path forward, we'll take this as a constraint on Entity work to make sure we can fulfill this condition.

@jsuereth the info MVP, as it stands, does assume that identifying labels are job and instance, but in the future it should be able to detect which are an info metric's identifying labels. The MVP avoids trying to detect the identifying labels, since it'll require persisting knowledge of them (which is a bigger problem to solve).

As I see it currently at least, the main thing with the Entity model in this respect should be that OTLP write requests always specify identifying Resource attributes in a sensible way, so that Prometheus can map them to corresponding labels. WDYT?

At a minimum, I think we should be able to provide a unique identifier (instance), where "job/instance" matching should always work.

I think at the minimum providing a unique instance (or equivalent) identifying resource attribute should work well from Prometheus' PoV at least.

"By default" the old metric is marked as stale as soon as the scrape happens that contains the new metric. The problem is that this doesn't work if ingesting via OTLP, which is obviously common in an OTel use case. Only in that case, you run into the problem of five minutes of duplication. (Which is the reason behind a minimal proposal to "fix" the problem (discussed elsewhere): Somehow make staleness handling work with OTLP ingestion.)

You may be interested in the current OTel Entities proposal, which is designed to carry non-identifying resource information: https://github.com/open-telemetry/oteps/blob/6d6febfaf05f130c703e2dd0fa91dfacada82a7d/text/entities/0256-entities-data-model.md. My hope would be that we can potentially map these to OM info metrics in the future.

edit: I forgot that I already left the same comment on the previous proposal docs.

@dashpole yes, this point in fact refers to the proposed Entity model, since it will allow for non-identifying resource/entity attributes to change.

And metrics that are "info metrics in spirit only" (kube_pod_labels) aren't even covered at all in OM because it would be invalid in OM to mark a metric as info metric that doesn't end on _info. (Quote from OM spec: "The Sample MetricName for the value of a MetricPoint for a MetricFamily of type Info MUST have the suffix '_info'.") I believe to unleash the full potential of info metrics, we need to relax this requirement in OM.

Just saying this here for the record. We don't need to discuss it in this proposal (or maybe just as a footnote).

I also worry about this being limited to OM, as most of the ecosystem seems to still be using prometheus text/protobuf formats today.

Did we consider to expanding this to also include gauge metrics with an _info suffix? Many OTel prometheus exporters still export target_info as a gauge due to lack of OM support, or because the Prometheus client doesn't support Info metrics.

Even OM needs an amendment for its text version.

So the best outcome here is to "fix" OpenMetrics (including marking identifying labels in the text format and allowing info metrics that do not end on _info, but also a number of other things that have hampered adoption so far) so that it finally OM gets adopted everywhere.

The second best outcome is to also amend classic Prometheus exposition formats. (It's a bit weird because I'd rather see the effort of changing the exposition format to lead directly to OM everywhere rather than more versions of the many formats that we already have.)

In the meantime, I would go for scrape config options to mark metrics of whatever source and format as info metrics, including the option to mark certain labels as identifying labels. (The heuristic "target labels → identifying labels, other labels → date labels" should be fine in most cases.) A rule to say "all metrics ending on _info are info metrics" would be easy to configure. And then special cases could be configured to say "kube_pod_labels coming from kube-state-metrics should be considered an info metric with namespace and pod as the identifying labels. Etc.

Maybe we could mention here that info metrics with additional identifying labels in the exposition format are rare. Usually, just the target labels added upon ingestion are the identifying labels, so this would be a good first guess. (Works for build_info, but doesn't work for kube_pod_labels.)

I'm adding the following sentence, does it work for you @beorn7?

Most info metrics' identifying labels will be job and instance, but there are some exceptions (e.g. kube_pod_labels).

Well, the damage is much bigger -- we essentially use all labels as "identifying" causing massive churn and cardinality bomb. With this feature we essentially increase use of info metric, so that's perhaps relevant to fix.

Perhaps you could mention that there is room for a future iteration addressing these issues in various forms, one could think of info metrics with persistence or some other kind of metadata store.

@jesusvazquez should that perhaps be mentioned in the Goals section? Before revising the proposal after @codesome's feedback, solving these problems in Prometheus was part of the goals. My understanding of the "Pitfalls of the current solution" section is that it should be referring to status quo in Prometheus, before the proposed solution.

To be honest I'm a bit unsure of what to put in the Pitfalls section currently, until @codesome reviews so the final form of the proposal is clearer.

Is the goal to also select certain metrics based on certain target_info non-identifying label?

e.g. would I be able to do something like info(http_request{...}, {"go_version" = "1.22.3"}, so filtering http_requests by label which only exists on target e.g. version of the Go that target was compiled with?

I added more specification about the new TSDB API and its use of label matchers. If I understand you correctly, the answer is yes:

If data label matchers are provided, time series are only included in the result if matching data labels from info metrics were found.

Nice! Then I wonder if simpler solution would be worth to discuss as an alternative e.g. special prefixed labels: https://cloud.google.com/stackdriver/docs/managed-prometheus/promql#metadata-labels 🤔

I will have a read, thanks.

After reading the link about metadata labels, as I understand Bartek, the idea is to perhaps use metadata like syntax instead of introducing an info function. I think the underlying TSDB API would be the same. Alternative syntax to info is definitely on the table, also @jesusvazquez has thought about this.

If I understand the Stackdriver documentation correctly, metadata labels are, from the outside API/PromQL perspective equivalent to adding all OTel resource attributes (or version number from build_info or pod labels from kube_pod_labels) to all metrics as label values (just with a prefix so that you have some hope (not a strategy!) that no label name collisions will happen, and that users can recognize which labels are coming from metadata).

Obviously, a backend can be implemented in a way to make that efficient, so that the efficiency concerns can be addressed. However, there are still concerns about UX, like every metric will now have a ginormous amount of labels, which needs additional tooling to get under control (like displaying metadata labels differently). It as a long-held Prometheus best practice to not add the same information (like the version number of the exposing binary) to every metric exposed, but bundle them in an info metric (like build_info), and the motivation for that was not just a concern about storage efficiency in the backend, but also UX.

My understanding is that this whole proposal is an attempt to solve the problem in the "classic" Prometheus way by making info metrics more powerful and more usable. The proposal to add all of the "metadata" to every metric in form of metadata labels is still on the table, but it's a different proposal.

Thanks for sharing those insights @beorn7 :)

Makes sense, thanks!

I think this statements sounds a bit contradictory to the above:

This means that knowledge of info metrics' identifying labels are lost, ...

I suspect you have a different understanding of the "Pitfalls of the current solution" section than I do. That section has the following purpose (from the template):

What specific problems are we hitting with the current solution? Why it’s not enough?

Similar question as the data labels - if the identifying labels change in between let's say t1 (with t0 being a older time and t2 being a future time), do we use the old identifying labels for queries touching t0 to t1, and then new identifying labels for queries after t1? Let's document this.

Maybe a change of identifying labels should really be seen as a completely independent new series, without any staleness marker insertion. As said, this should be rare (while the fallout of a change in the data labels in the absence of staleness markers, e.g. what you get when ingesting OTLP, triggered the whole effort). And it could actually mean the new info series is actually meant as a completely independent new series.

do we use the old identifying labels for queries touching t0 to t1, and then new identifying labels for queries after t1?

@codesome yes, that is my current thinking at least. My thinking is to track through the info metric's samples, what is the identifying label set at any given point in time. Agree to document it properly, I'll give my proposal a re-read, to see whether I need to clarify further.

Maybe a change of identifying labels should really be seen as a completely independent new series, without any staleness marker insertion.

@beorn7 there shouldn't be any need for a staleness marker insertion, when the identifying label set changes, so long as the entire series' label set remains the same. My thinking is to just record a new sample for the info metric, that records the new set of identifying labels. Does that make sense to you as well?

It probably works (but again, I would almost see this as an implementation detail).

Staleness marker will be inserted anyway if you scrape Prometheus-style, and the problems you have because of staleness markers not happening with OTLP ingestion will also happen anyway. It's not just for info metrics, and I think we don't need a bespoke staleness-marker ingestion solution here.

Do we want to allow writing it as just target_info, and with additional matchers like target_info{k8s_cluster_name=~".+"}? Or is there a reason why it needs to be {__name__="target_info"}?

Maybe the current implementation just recognizes the selecter of it starts with curly braces. Generally, I guess any valid vector matcher, including just the name, should be valid.

The current implementation only recognizes a selector starting with curly braces indeed. But it's also a very rough and hacky implementation, as I'm concentrating on getting the underlying design (TSDB API) right primarily.

Maybe we can, at the end of the day, use a normal vector matcher after all, just that the PromQL engine has to intervene before it is fully evaluated.

Yesterday, I discussed with @juliusv his valid concern about using a vector matcher here in a non-standard way, and we found that you could stretch things just a tiny bit to see this vector matcher as a normal vector matcher. It will (potentially) select a lot, but all the info metrics we want will be selected, too, so one might argue the info function simply trims this selection down to the relevant info metrics. Of course, for efficiency reasons, we want to do this trimming proactively and not first load all the metrics matching the vector matcher into memory, but that's an optimization that could also be applied to any other PromQL expression without breaking any semantics. The info function also looks back at the originally specified labels to find out what labels to copy. But that doesn't break the semantics, either.

The info function also looks back at the originally specified labels to find out what labels to copy.

Yeah, the absent() function already does similar magic.

Now on a language typing level, we'll still have the problem that we only have four types: strings, scalars, and instant/range vectors. We currently don't have any way to specify that a function expects not just one of those four types as an input, but a specific AST node type (specifically an instant vector selector, not just any instant-vector-typed result). I guess for a start we could ignore this in parsing and have the function check at run-time whether the parameter is really an instant vector selector and error out otherwise.

absent() is indeed a good role model. I think we could just accept any instant vector-typed result. If the expression doesn't evaluate to one or more info metric, we can issue a warning and not do anything. But generally, we are going down into details very prematurely here.

Can you help me understand this example? I know the current rule for aggregating rates is Rate then sum, never sum then rate.

Now, will this turn into rate then info then sum, or could you also info then rate then sum?

Not sure if there is any specific reason to do rate() before info(), but sum() after info().

Disclaimer: I just copied this example from Beorn's original document. I'll answer based on my knowledge of info though.

rate(http_server_request_duration_seconds_count[2m]) becomes a sub-query to the info function, which adds the k8s_cluster_name label from target_info to the http_server_request_duration_seconds_count time series resulting from the sub-query. The sum expression aggregates by the labels k8s_cluster_name and http_status_code. If you don't apply sum after info, the k8s_cluster_name label won't be available.

As for whether you could do info before rate; I haven't tested, but I think it would work. However, it looks less logical/comprehensible to me, as you really want the rate of http_server_request_duration_seconds_count[2m].

@aknuds1 already explained nicely why you have to do the sum last.

In current PromQL, the order before the sum has to be range selector → rate → info.

The range selector is a construct that only works directly with a metric, i.e. info(http_server_request_duration_seconds_count)[2m] doesn't even parse. (You could do a sub-query, i.e. info(http_server_request_duration_seconds_count)[2m:], but that's quite a different thing.)

Neither can you apply info to a range selector: info(http_server_request_duration_seconds_count[2m]) doesn't even parse again, because the grammar knows that info applies to an instant vector, not a range vector. (And the concept of overloaded function, i.e. functions with different type signature but the same name, doesn't exist in PromQL. Within the current framework, you could only implement a separate info_range function or something, but I hope we don't need that.)

This might have been discussed before, but I wonder if we are missing the simplest solution of query rewriting as a considered alternative. Given the example query on line 119, can we rewrite it into a join and let the engine execute it with the existing semantics of PromQL. Information needed for the join can either be implicitly deducted or specified by the user as arguments.

The info function can't be simply reduced to syntactic sugar for joins though, for the following reasons at least:

1. The info function automatically finds suitable info metrics to join with (the MVP only supports target_info, but we aim to lift that limitation in the future)
2. The info function automatically knows which are info metrics' identifying labels, i.e. which to join on (the MVP assumes instance and job, but we aim to lift that limitation in the future)
3. The info function solves the problem of temporary join query conflicts between info metrics due to non-identifying labels changing, until the old version goes stale

Information needed for the join can either be implicitly deducted or specified by the user as arguments.

Assuming we were to use query rewriting, how would you deduce which info metric(s) to join against (when info is no longer limited to just the target_info metric)? If the user, as in join queries, has to supply which info metric to join with and which labels to join on plus which labels to include, have we really gained much versus status quo?

I can only see this working well for a static list of well known info metrics like.. target_info or by specifying which info metrics to use expicitely. If i have multiple info metrics that match by accident and dont specify matchers I probably get confusing labels added. Thats not possible if i specify the info metric explicitely or constrain it to a well known list like target_info. Prometheus is too flexible to make such convenience safe I think.

I can only see this working well for a static list of well known info metrics like.. target_info or by specifying which info metrics to use expicitely.

@MichaHoffmann if that is your concern with the info function (and @fpetkovski's?), I think that should be an explicit criticism of the proposed design, rather than suggesting info as proposed here can be simply reduced to syntactic sugar. If you think the design should be explicit about picking info metrics rather than automatic (although that can be filtered down via the __name__ label matcher), I think there should be an alternative proposal. I'm not sure though if it's most feasible to list competing proposals in the same document, or whether they should be separate.

Oh, Sorry I didnt want to propose something differently, I was just concerning out loud. What I want to say is that the name label matcher for info metric probably should be requirement for this to be generally least surprising for the user.

Thanks @MichaHoffmann, I can note your wish about the name label matcher being a requirement, so a consensus can be reached.

I should note that my original idea was indeed that there is generally a relatively low number of info metrics so that a default matching behavior of "all the info metrics known to the TSDB" is feasible, which is in fact one of the reasons that makes the "joining" easier.

Note that with a "general metadata store", you would probably also end up with "give me all the metadata you know for this metric", which is (more or less) equivalent to "give me all the info metrics that have an identifying labels match with this metric".

I was thinking having an option to specify info metrics to join is not a bad idea. We can still by default join with all info metrics, but have an option to specify a subset as well. Maybe that is something we can add later in a backward compatible fashion. Any thoughts on this @aknuds1 @beorn7?

Yeah, me thought was that the info function should focus on the cases where it "magically" works, and if things get complicated (with label collisions, too many info metrics that match, …), then you can just revert to the traditional join syntax, which is mostly so complex because it specifies everything explicitly. Or in other words: We are diluting the added value of the info function if we add some of the knobs that the traditional join syntax has (and which makes it so unwieldy).

But we can still have this alternative here for public consideration (and to show that this thought was considered).

I guess it would be reasonable if this only works for info metrics that were present in the same scrape maybe?

I don't think we should have that restriction. kube_pod_labels is an example where you might apply things cross-scrape. OTLP ingestion is an example where there isn't even a notion of "scrape".

For metrics ingested via a scrape, the target labels will be identifying labels by default anyway, so the whole proposed semantics of the info function will implicitly limit info metric matching to the same scrape in most cases (but that's a behavior that needs to be configurable in a different way, e.g. if scraping KSM, one might want to not have the KSM target labels as identifying labels to enable kube_pod_labels matching cross-target.)

I was thinking having an option to specify info metrics to join is not a bad idea.

@codesome @beorn7 the proposal suggests the ability to specify certain info metrics by matching on __name__:

• A data label matcher like __name__="target_info" can be used to restrict the info metrics used.

Did you consider this?

Maybe there could also be a configuration parameter to control this behaviour?