Description

Hi! In the recent Prometheus release v3.7.0-rc.0, a new feature was added:

• [FEATURE] Remote-write: Add type and unit labels to outgoing time series in remote-write 2.0 when the type-and-unit-labels feature flag is enabled. (#17033)

I tried enabling this feature, but the type and unit labels don’t appear in Prometheus after being sent via remote-write v2.
I’d appreciate any help to figure out what I might be doing wrong.

Setup

Both Prometheus and Prometheus Agent are running on the same host.
The agent scrapes metrics from node-exporter and sends them to Prometheus via remote-write v2.

flowchart LR
    subgraph test-monitoring-host
        pa("prometheus-agent</br>[version 3.7.0-rc.0]")
        ne("node-exporter</br>[version 1.9.1]")
        prometheus("Prometheus</br>[version 3.7.0-rc.0]")        
    end

    pa <-->|scrape| ne
    pa -->|Remote Write| prometheus

Prometheus Agent configuration

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: 'node-exporter'
    scrape_interval: 15s
    file_sd_configs:
    - files:
      - '/opt/prometheus-agent/test/node-exporter.yaml'
      refresh_interval: 1m
    relabel_configs:
      - source_labels: [__address__]
        regex: '(.*):(.*)'
        target_label: instance
        replacement: '${1}'
      - source_labels: [__address__]
        regex: '(.*):(.*)'
        target_label: port
        replacement: '${2}'
    tls_config:
      insecure_skip_verify: true

remote_write:
  - url: "http://127.0.0.1:9090/api/v1/write"
    name: "Remote write to prometheus"
    protobuf_message: io.prometheus.write.v2.Request
    metadata_config:
      send: true
    write_relabel_configs:
      - source_labels: ['__name__']
        regex: 'prometheus_.*|go_.*'
        action: drop

Prometheus agent startup flags:

--config.file=/opt/prometheus-agent/prometheus-agent.yml
--web.listen-address=0.0.0.0:9091
--agent
--storage.agent.path=/mnt/tsdb/prometheus-agent
--storage.agent.retention.max-time=1d
--enable-feature=metadata-wal-records,type-and-unit-labels
--log.level=debug

Prometheus configuration

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['127.0.0.1:9090']

Prometheus startup flags:

--config.file=/opt/prometheus/prometheus.yml
--storage.tsdb.path=/mnt/tsdb/prometheus
--web.listen-address=0.0.0.0:9090
--storage.tsdb.retention.time=1d
--web.enable-remote-write-receiver
--web.remote-write-receiver.accepted-protobuf-messages=io.prometheus.write.v2.Request
--enable-feature=metadata-wal-records,type-and-unit-labels
--log.level=debug

Steps to Reproduce

• Start Prometheus and Prometheus Agent (both v3.7.0-rc.0) with the configs and feature flags above
• Let the agent scrape metrics from node-exporter
• Check metrics received by Prometheus (i'm using Promlens)

Expected Behavior

Metrics received by Prometheus should include type and unit labels, as described in #17033

Problem

Type and unit labels don’t appear in Prometheus after being sent via remote-write v2

Environment

[root@exp-01 tmp]# cat /etc/os-release
NAME="AlmaLinux"
VERSION="9.6 (Sage Margay)"

Would you please confirm whether this feature is already fully implemented in the v2 remote-write receiver, or if additional flags/config are required on the receiving side?
Let me know if you’d like me to provide full debug logs or a test environment to reproduce the issue.