Skip to content

Inconsistent metric deduplication in Otel2PrometheusConverter #6277

@nluk

Description

@nluk

Describe the bug
Prometheus MetricSnapshots constructor throws because of duplicated metric names. This is a part of the Otel2PrometheusConverter::convert call.

An Exception occurred while scraping metrics: java.lang.IllegalArgumentException: logback_events: duplicate metric name
	at io.prometheus.metrics.model.snapshots.MetricSnapshots.<init>(MetricSnapshots.java:43)
	at io.opentelemetry.exporter.prometheus.Otel2PrometheusConverter.convert(Otel2PrometheusConverter.java:112)
	at io.opentelemetry.exporter.prometheus.PrometheusMetricReader.collect(PrometheusMetricReader.java:56)
	at io.prometheus.metrics.model.registry.MultiCollector.collect(MultiCollector.java:26)
	at io.prometheus.metrics.model.registry.PrometheusRegistry.scrape(PrometheusRegistry.java:72)
	at io.prometheus.metrics.exporter.common.PrometheusScrapeHandler.scrape(PrometheusScrapeHandler.java:112)
	at io.prometheus.metrics.exporter.common.PrometheusScrapeHandler.handleRequest(PrometheusScrapeHandler.java:53)
	at io.prometheus.metrics.exporter.httpserver.MetricsHandler.handle(MetricsHandler.java:43)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:95)
	at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:98)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:851)
	at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:95)
	at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:818)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

Steps to reproduce
Create two MetricData objects with different names, that resolve to the same prometheusName and run convert.

Testcase to reproduce the error: main...nluk:opentelemetry-java:bug/prometheus-converter-duplicate-metric-name

What did you expect to see?

Hard to say really. Metrics of the same type may be merged, but the fact that there are two competing metrics a.b and a_b might indicate a misconfiguration.

What did you see instead?

The Otel2PrometheusConverter::putOrMerge method uses snapshot.getMetadata().getName() as a unique map key. So metric.name and metric_name are two separate entries.

Prometheus compares sorted snapshot.getMetadata().getPrometheusName() link to find collisions. This means that metric.name and metric_name are both represented as metric_name, and a collision is detected.

What version and what artifacts are you using?
Artifacts: aws-opentelemetry-agent 1.32.1, so opentelemetry 1.32.1 link that targets otel-sdk 1.34.1

Environment
Compiler: Temurin-17.0.5+8
OS: Ubuntu 22.04

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions