Hi! Thanks a lot to the Prometheus team for adding the new type-and-unit-labels feature!

Related PR

While testing it, i noticed some differences in behavior depending on the Prometheus mode.

This behavior was tested after applying the changes from #17329.
While the improvement works correctly in standard Prometheus mode, the following results were observed when using the agent mode (details below).

Observed behavior

Case 1 - Prometheus scrapes targets directly

Everything works as expected - the __type__ label appears in the resulting metrics.

Prometheus config

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['test-node-exporter:9100']
    relabel_configs:
      - source_labels: [__address__]
        regex: '(.*):(.*)'
        target_label: instance
        replacement: '${1}'
      - source_labels: [__address__]
        regex: '(.*):(.*)'
        target_label: port
        replacement: '${2}'

Prometheus startup flags

--enable-feature=type-and-unit-labels,metadata-wal-records
--web.enable-remote-write-receiver
--web.remote-write-receiver.accepted-protobuf-messages=io.prometheus.write.v2.Request

Result

[root@exp-01 grizzlybite]# curl -s 'http://192.168.100.7:9090/api/v1/query?query=node_memory_MemAvailable_bytes'  | jq '.data.result[0].metric'

{
  "__name__": "node_memory_MemAvailable_bytes",
  "__type__": "gauge",
  "instance": "test-node-exporter",
  "job": "node-exporter",
  "port": "9100"
}

Case 2 - Prometheus Agent sends data via Remote Write v2

When Prometheus runs in --agent mode and sends metrics through remote_write using protobuf v2,
the resulting metrics do not contain the __type__ label - even with the same feature flags enabled.

Prometheus (receiver)

--enable-feature=type-and-unit-labels,metadata-wal-records
--web.enable-remote-write-receiver
--web.remote-write-receiver.accepted-protobuf-messages=io.prometheus.write.v2.Request

Prometheus Agent

remote_write:
  - url: "http://127.0.0.1:9090/api/v1/write"
    protobuf_message: io.prometheus.write.v2.Request
    queue_config:
      max_shards: 40
      capacity: 10000

Agent startup flags

--agent
--enable-feature=type-and-unit-labels,metadata-wal-records

Result

[root@exp-01 grizzlybite]# curl -s 'http://192.168.100.7:9090/api/v1/query?query=node_memory_MemAvailable_bytes'  | jq '.data.result[0].metric'
{
  "__name__": "node_memory_MemAvailable_bytes",
  "instance": "test-node-exporter",
  "job": "node-exporter",
  "port": "9100"
}

Summary

It looks like the __type__ label is currently not propagated when metrics are sent via Prometheus Agent -> Prometheus using Remote Write v2, even though both sides have
--enable-feature=type-and-unit-labels enabled.

If this is the expected behavior (e.g., by design for the agent), please clarify.
Otherwise, I’d be happy to help test any future fixes or patches!

Environment

• Prometheus 3.7.1
• node-exporter 1.9.1
• Tested on AlmaLinux 9.6 (Sage Margay)