Store metadata for OpenMetrics _created metrics #10164
Description
What did you do?
Deployed an example application emitting Prometheus metrics in a Kubernetes cluster alongside a Prometheus deployment.
What did you expect to see?
All metrics with properly formatted HELP and TYPE strings have their metadata available in the Prometheus UI and API.
What did you see instead? Under which circumstances?
When querying for metrics in the graph UI, I am able to see the metric time series plotted. However, the metric metadata in the autocomplete tooltip is missing.
I went to the metadata API at /api/v1/metadata and saw the metadata was missing there as well.
I verified the target metrics endpoint looks to be exporting properly-formatted metrics.
Environment
- System information:
Linux 5.4.144+ x86_64
- Prometheus version:
prometheus, version 2.32.1 (branch: HEAD, revision: 41f1a8125e664985dd30674e5bdf6b683eff5d32)
build user: root@54b6dbd48b97
build date: 20211217-22:08:06
go version: go1.17.5
platform: linux/amd64
- Alertmanager version:
N/A
- Prometheus configuration file:
global:
scrape_interval: 30s
scrape_timeout: 10s
evaluation_interval: 1m
scrape_configs:
- job_name: prometheus
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
follow_redirects: true
static_configs:
- targets:
- localhost:9090
- job_name: prom-example
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
follow_redirects: true
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
separator: ;
regex: o11y-demo
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_namespace]
separator: ;
regex: (.*)
target_label: namespace
replacement: $1
action: replace
- source_labels: [__meta_kubernetes_pod_name, __meta_kubernetes_pod_container_port_name]
separator: ;
regex: (.+);(.+)
target_label: instance
replacement: $1:$2
action: replace
- source_labels: [__meta_kubernetes_pod_container_port_name]
separator: ;
regex: metrics
replacement: $1
action: keep
kubernetes_sd_configs:
- role: pod
kubeconfig_file: ""
follow_redirects: true
-
Alertmanager configuration file:
N/A -
Logs:
N/A
Please let me know if there's any more detail I can provide and thank you!