Remote Write V2 and converted classic histograms to NHCB #17075
Description
What did you do?
I am trying to send Prometheus metrics to an OTel collector, using remote write. As the prometheusremotewritereceiver only supports remote write v2, I need to convert the classic histograms to NHCB in order for them to be processed by it.
To achieve this I set send_native_histograms: true in the remote_write configuration and convert_classic_histograms_to_nhcb: true globally
What did you expect to see?
I expected to see metrics being outputted from my OTel collector and no error messages on Prometheus
What did you see instead? Under which circumstances?
I saw an error message stating that some metrics are being mark as unspecified, and the metrics correspond to histograms.
I created a receiver where I could play around and from what I understand, the message Timeseries has the metric type Histogram on metadata and if this timeseries message has samples set then it is a classic histogram, otherwise it is a native one (or an NHCB), but what I believe is happening is that the metric type is being set to Unspecified meaning it will be dropped.
Shouldn't NHCBs have the type set to Histogram?
System information
Linux 5.10.207 x86_64 (container image output)
Prometheus version
prometheus, version 3.5.0 (branch: HEAD, revision: 8be3a9560fbdd18a94dedec4b747c35178177202)
build user: root@4451b64cb451
build date: 20250714-16:15:23
go version: go1.24.5
platform: linux/amd64
tags: netgo,builtinassets
Prometheus configuration file
global:
scrape_interval: 15s
evaluation_interval: 15s
convert_classic_histograms_to_nhcb: true
remote_write:
- url: http://otel-collector:4318/api/v1/write
protobuf_message: io.prometheus.write.v2.Request
send_native_histograms: true
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']Alertmanager version
Alertmanager configuration file
Logs
time=2025-08-22T18:57:51.186Z level=ERROR source=queue_manager.go:1668 msg="non-recoverable error" component=remote remote_name=de350a url=http://otel-collector:4318/api/v1/write failedSampleCount=241 failedHistogramCount=10 failedExemplarCount=0 err="server returned HTTP status 400 Bad Request: unsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"go_gc_heap_allocs_by_size_bytes\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"go_gc_heap_frees_by_size_bytes\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"go_gc_pauses_seconds\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"go_sched_latencies_seconds\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"go_sched_pauses_stopping_other_seconds\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"go_sched_pauses_total_gc_seconds\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"go_sched_pauses_total_other_seconds\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"prometheus_tsdb_compaction_chunk_range_seconds\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"prometheus_tsdb_compaction_chunk_samples\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"prometheus_tsdb_compaction_chunk_size_bytes\"\n"