We have generic metrics (such as http.server.request.duration or messaging.publish.duration).
We also have precedents extending such metrics.
E.g. .NET semconv extends http.server.request.duration and adds .NET-specific attribute on it.
In case of messaging, in #687 we're adding system-specific attributes
On Kafka, we'd like to report partition id. As a result, if we proceed with #687 we'll have a common set of metrics, but each individual messaging system would be able to add its own attributes.
We have other similar cases:
- connection-level metrics (we have overlapping ones for DB and .NET http connections).
- DB generic call duration and custom attributes that would be added by extensions
Problem: We don't have a uniform solution on how to approach extension.
Option 1. Create different metric for each db/messaging/rpc system with their own set of attributes
E.g.
- messaging.kafka.send.duration
- messaging.rabbitmq.publish.duration
- ....
Pros:
- can follow system-specific terminology
- can add system-specific metrics post-stability
Cons:
- no common dashboards/alerts
- hard to make metrics consistent. Possible mitigation could be that extension mechanism that allows to update metric name, but inherits instrument, unit, and attributes.
Option 2: Have a common metric, add custom attributes to it in the extensions
E.g.
we have one common messaging.publish.duration with some common attributes.
- kafka metric would also have
messaging.system="kafka", messaging.kafka.partition_id="foo" attributes
- rabbitmq would have
messaging.system="rabbitmq", messaging.rabbitmq.routing_key="bar" attributes
Pros:
- can create generic dashboards/alerts which usually need to be adjusted for each system
- according to @trask "it's the least bad option"
Cons:
- if I have multiple messaging systems, it could be overwhelming to have all of them in one metric
- terminology needs to be unified
- as we're stabilizing certain extensions, we'd have to find all attributes for their metrics prior to stabilization.
Regardless of the option we pick, we'd need to add some tooling to support extensibility. Related #592
We have generic metrics (such as
http.server.request.durationormessaging.publish.duration).We also have precedents extending such metrics.
E.g. .NET semconv extends
http.server.request.durationand adds .NET-specific attribute on it.In case of messaging, in #687 we're adding system-specific attributes
On Kafka, we'd like to report partition id. As a result, if we proceed with #687 we'll have a common set of metrics, but each individual messaging system would be able to add its own attributes.
We have other similar cases:
Problem: We don't have a uniform solution on how to approach extension.
Option 1. Create different metric for each db/messaging/rpc system with their own set of attributes
E.g.
Pros:
Cons:
Option 2: Have a common metric, add custom attributes to it in the extensions
E.g.
we have one common
messaging.publish.durationwith some common attributes.messaging.system="kafka", messaging.kafka.partition_id="foo"attributesmessaging.system="rabbitmq", messaging.rabbitmq.routing_key="bar"attributesPros:
Cons:
Regardless of the option we pick, we'd need to add some tooling to support extensibility. Related #592