Skip to content

Define metric extension mechanism #760

@lmolkova

Description

@lmolkova

We have generic metrics (such as http.server.request.duration or messaging.publish.duration).
We also have precedents extending such metrics.
E.g. .NET semconv extends http.server.request.duration and adds .NET-specific attribute on it.

In case of messaging, in #687 we're adding system-specific attributes

On Kafka, we'd like to report partition id. As a result, if we proceed with #687 we'll have a common set of metrics, but each individual messaging system would be able to add its own attributes.

We have other similar cases:

  • connection-level metrics (we have overlapping ones for DB and .NET http connections).
  • DB generic call duration and custom attributes that would be added by extensions

Problem: We don't have a uniform solution on how to approach extension.

Option 1. Create different metric for each db/messaging/rpc system with their own set of attributes

E.g.

  • messaging.kafka.send.duration
  • messaging.rabbitmq.publish.duration
  • ....

Pros:

  • can follow system-specific terminology
  • can add system-specific metrics post-stability

Cons:

  • no common dashboards/alerts
  • hard to make metrics consistent. Possible mitigation could be that extension mechanism that allows to update metric name, but inherits instrument, unit, and attributes.

Option 2: Have a common metric, add custom attributes to it in the extensions

E.g.
we have one common messaging.publish.duration with some common attributes.

  • kafka metric would also have messaging.system="kafka", messaging.kafka.partition_id="foo" attributes
  • rabbitmq would have messaging.system="rabbitmq", messaging.rabbitmq.routing_key="bar" attributes

Pros:

  • can create generic dashboards/alerts which usually need to be adjusted for each system
  • according to @trask "it's the least bad option"

Cons:

  • if I have multiple messaging systems, it could be overwhelming to have all of them in one metric
  • terminology needs to be unified
  • as we're stabilizing certain extensions, we'd have to find all attributes for their metrics prior to stabilization.

Regardless of the option we pick, we'd need to add some tooling to support extensibility. Related #592

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions