Skip to content

Commit b6f5a4f

Browse files
lexfreilexfrei
authored
feat(monitoring): stamp application identity on scraped series (#5216)
## What this PR does Adds a default scrape class `application-identity` to both VMAgents: the cluster-wide one in `monitoring-agents` and the per-tenant one in `monitoring`. Every series scraped from a pod that carries the lineage labels `apps.cozystack.io/application.{group,kind,name}` gets them as `apps_cozystack_io_application_{group,kind,name}`. Why: the per-application metrics scope in the design proposal [Tenant resource-consumption metrics](cozystack/community#94) (Design 4a) needs to tell which application an exporter series belongs to. Today that is only possible indirectly, through pod names. How the labels are protected against forgery by an exporter: - At the target stage, the class always sets all three labels: to the pod's real lineage value, or to a placeholder `-` when the pod has none (`-` is not a valid Kubernetes label value). With `honorLabels: false`, any copy the exporter publishes therefore collides with the target label and is renamed to `exported_*`. - At the metric stage, the class drops every `exported_*` copy. - The placeholder is removed by the VMAgent's global remote-write relabeling (`inlineRelabelConfig`), not at the metric stage: vmagent adds `up` and `scrape_*` with the target labels after metric relabeling, so only the remote-write stage sees those series. Targets without lineage therefore keep exactly the label set they have today on every series, including `up` and `scrape_*`. - With `honorLabels: true` the exporter's value wins and no relabel rule can tell it apart. ingress-nginx scrapes with `honorLabels: true`, and its series describe Ingress objects in any namespace, so the controller's identity would misattribute traffic; its scrape now drops the identity labels explicitly, from the target (so `up` and `scrape_*` stay unchanged) and from the scraped samples (so an exporter's copy cannot survive). Switching it to `honorLabels: false` was not done, because that renames `namespace` and breaks the existing ingress dashboards. #### How it was tested - helm-unittest for `monitoring-agents`, `monitoring` and `ingress`. These tests check the rendered config only; the relabeling behaviour was verified offline by running vmagent v1.136.0, built from source, against the rendered rules: `up` and `scrape_*` carry no identity for targets without lineage, and the placeholder appears on no series. - The rendered `scrapeClasses` validate against the vendored VMAgent CRD schema. - The rendered rules were exercised on a locally built vmagent v1.136.0 (the version the bundled operator deploys) writing to a local VictoriaMetrics: with and without lineage, with and without a forging exporter, `honorLabels` true and false, and the ingress-nginx endpoint rules. Checked on exporter samples and on `up`, `scrape_samples_scraped` and `scrape_series_added`; the previous rules reproduce the `-` placeholder on the generated series of targets without lineage, the new ones leave no placeholder on any series. - Verified on a test cluster with the earlier revision that removed the placeholder at the metric stage (the move to remote-write relabeling is not yet verified on a cluster): series carry the application labels; metrics read for a Redis instance through the per-application proxy root contained only that instance's pods; vmagent logs show no errors. #### Known limitations - Scrapes with `honorLabels: true` other than ingress-nginx are not protected against an exporter that publishes the identity labels itself. Today these are the vendored qdrant and harbor ServiceMonitors; their exporters are platform images with fixed label names, and their series describe the application's own pods. - A tenant's raw `vmagent.inlineScrapeConfig` bypasses scrape classes entirely, and so do `additionalScrapeConfigs`. VMProbe takes only auth settings from a scrape class (operator v0.68.4); no VMProbe exists in the platform today. - A scrape object's own relabel rules run after the class and can overwrite the labels; tenants cannot create VictoriaMetrics operator objects, so this is platform-controlled. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Downstream: no trigger matched. Candidate: none affected, because the diff changes templates and tests in `packages/system/monitoring-agents`, `packages/system/monitoring` and `packages/extra/ingress` without adding, renaming or removing a package and without touching any `values.yaml` or schema. The telemetry metrics the telemetry server reads (`cozy_cluster_info`, `cozy_application_count`) are produced by the operator, not by these scrapes. The proposal text in `cozystack/community` (#94, Design 4a) should be corrected to name qdrant and harbor as further `honorLabels: true` scrapes; that is not a trigger-map item. ### Release note ```release-note feat(monitoring): VMAgent now adds `apps_cozystack_io_application_{group,kind,name}` labels to every series scraped from a pod that carries application lineage labels. Because the label set of those series changes, each of them becomes a new series after the upgrade: graphs and dashboards show a break at the rollout, queries over a range that spans it see two series per pod, and alert `for:` timers on those series restart. Series of targets without lineage labels, including node-level series (cAdvisor, kubelet) and kube-state-metrics, and all ingress-nginx series keep their current label set; this includes their `up` and `scrape_*` series. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Improvements** * Monitoring metrics now use application identity labels derived from Kubernetes pod metadata for covered scrape targets. Exporter-supplied duplicates and placeholder values are removed, including from `up` and `scrape_*` metrics. * Ingress metrics no longer retain application identity labels at either target or metric relabeling stages. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 906eeb2 + 699b5ce commit b6f5a4f

6 files changed

Lines changed: 295 additions & 0 deletions

File tree

‎packages/extra/ingress/templates/nginx-scrape.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,13 @@ spec:
99
matchNames:
1010
- {{ .Release.Namespace }}
1111
podMetricsEndpoints:
12+
# Most series describe the Ingress objects served, in any namespace, so
13+
# the application identity the vmagent scrape class stamps from the
14+
# controller pod would misattribute them; and with honorLabels an
15+
# exporter-supplied identity would win over the class. The controller's
16+
# own series (nginx process, reload status, go_*, up, scrape_*) lose the
17+
# Ingress application's identity too; that is the accepted cost. The
18+
# target-stage drop covers up and scrape_*, which skip metric relabeling.
1219
- port: metrics
1320
honorLabels: true
1421
interval: 30s
@@ -19,6 +26,11 @@ spec:
1926
targetLabel: node
2027
- targetLabel: tier
2128
replacement: cluster
29+
- action: labeldrop
30+
regex: apps_cozystack_io_application_(group|kind|name)
31+
metricRelabelConfigs:
32+
- action: labeldrop
33+
regex: apps_cozystack_io_application_(group|kind|name)
2234
selector:
2335
matchLabels:
2436
app.kubernetes.io/component: controller
@@ -46,6 +58,11 @@ spec:
4658
targetLabel: controller
4759
- targetLabel: tier
4860
replacement: cluster
61+
- action: labeldrop
62+
regex: apps_cozystack_io_application_(group|kind|name)
63+
metricRelabelConfigs:
64+
- action: labeldrop
65+
regex: apps_cozystack_io_application_(group|kind|name)
4966
selector:
5067
matchLabels:
5168
app.kubernetes.io/component: controller

‎packages/extra/ingress/tests/application_identity_test.yaml‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
suite: ingress-nginx series do not carry an application identity
2+
templates:
3+
- templates/nginx-scrape.yaml
4+
release:
5+
name: ingress
6+
namespace: tenant-root
7+
8+
# The vmagent scrape class stamps the scraped pod's application identity on
9+
# every series. Most ingress-nginx series describe the Ingress objects it
10+
# serves, in whatever namespace they live (honorLabels keeps their
11+
# namespace), so the controller's identity would misattribute them, and with
12+
# honorLabels true an exporter-supplied identity would win over the class.
13+
# The controller's own health series lose that identity too, by design. Both
14+
# endpoints drop the identity instead: from the target, because vmagent
15+
# adds up and scrape_* with the target labels after metric relabeling, and
16+
# from the scraped samples, because an exporter's copy is not renamed.
17+
tests:
18+
- it: drops the identity on every endpoint
19+
asserts:
20+
- lengthEqual:
21+
path: spec.podMetricsEndpoints
22+
count: 1
23+
- equal:
24+
path: spec.podMetricsEndpoints[0].metricRelabelConfigs
25+
value:
26+
- action: labeldrop
27+
regex: apps_cozystack_io_application_(group|kind|name)
28+
29+
# Endpoint rules run after the scrape class rules, so the drop takes
30+
# effect wherever it sits in this list; it is pinned last so that no
31+
# later rule can set the identity again.
32+
- it: drops the identity from the basic endpoint's target as its last rule
33+
documentIndex: 0
34+
asserts:
35+
- equal:
36+
path: spec.podMetricsEndpoints[0].relabelConfigs
37+
value:
38+
- replacement: nginx-ingress-controller
39+
targetLabel: job
40+
- sourceLabels: [__meta_kubernetes_pod_node_name]
41+
targetLabel: node
42+
- targetLabel: tier
43+
replacement: cluster
44+
- action: labeldrop
45+
regex: apps_cozystack_io_application_(group|kind|name)
46+
47+
- it: drops the identity from the detailed endpoint's target as its last rule
48+
documentIndex: 1
49+
asserts:
50+
- equal:
51+
path: spec.podMetricsEndpoints[0].relabelConfigs
52+
value:
53+
- replacement: nginx-ingress-controller
54+
targetLabel: job
55+
- sourceLabels: [__meta_kubernetes_pod_node_name]
56+
targetLabel: node
57+
- sourceLabels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
58+
targetLabel: controller
59+
- targetLabel: tier
60+
replacement: cluster
61+
- action: labeldrop
62+
regex: apps_cozystack_io_application_(group|kind|name)

‎packages/system/monitoring-agents/templates/vmagent.yaml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,39 @@ spec:
2020

2121
scrapeInterval: 30s
2222
selectAllByDefault: true
23+
# The class sets all three identity labels on every target it covers, "-"
24+
# when the pod has no lineage ("-" is not a valid Kubernetes label value,
25+
# so it never collides with a real one). Under honorLabels false every
26+
# exporter-supplied copy then collides, is renamed to exported_* and
27+
# dropped. Under honorLabels true the exporter's copy wins, so such a
28+
# scrape must drop these labels itself. A scrape object's own relabel
29+
# rules run after these: they may drop the labels but must never set them.
30+
# Not covered: VMProbe (operator v0.68.4 takes only auth from a class) and
31+
# raw scrape configs. The placeholder is removed by the remote-write
32+
# relabeling below.
33+
scrapeClasses:
34+
- name: application-identity
35+
default: true
36+
relabelConfigs:
37+
{{- range list "group" "kind" "name" }}
38+
- targetLabel: apps_cozystack_io_application_{{ . }}
39+
replacement: "-"
40+
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_{{ . }}]
41+
regex: (.+)
42+
targetLabel: apps_cozystack_io_application_{{ . }}
43+
{{- end }}
44+
metricRelabelConfigs:
45+
- action: labeldrop
46+
regex: (exported_)+apps_cozystack_io_application_(group|kind|name)
47+
# Not in metricRelabelConfigs: vmagent adds up and scrape_* after metric
48+
# relabeling, so the placeholder would stay on those series.
49+
inlineRelabelConfig:
50+
{{- range list "group" "kind" "name" }}
51+
- sourceLabels: [apps_cozystack_io_application_{{ . }}]
52+
regex: "-"
53+
targetLabel: apps_cozystack_io_application_{{ . }}
54+
replacement: ""
55+
{{- end }}
2356
additionalScrapeConfigs:
2457
name: additional-scrape-configs
2558
key: prometheus-additional.yaml

‎packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
suite: cluster-wide vmagent stamps application identity on exporter series
2+
templates:
3+
- templates/vmagent.yaml
4+
release:
5+
name: monitoring-agents
6+
namespace: cozy-monitoring
7+
8+
# The tenant-resource-metrics proxy scopes a query to one application by
9+
# apps_cozystack_io_application_{group,kind,name}. The labels are only a
10+
# boundary if the scraped pod cannot choose them: the class sets all three
11+
# on every target it covers (a placeholder when the pod has no lineage), so with
12+
# honorLabels false an exporter's own copy is renamed to exported_* and
13+
# dropped. The same rules live in the tenant vmagent of the monitoring
14+
# package and are pinned there by the same assertions.
15+
tests:
16+
- it: every scrape gets the application identity class unless it names another
17+
asserts:
18+
- equal:
19+
path: spec.scrapeClasses[0].name
20+
value: application-identity
21+
- equal:
22+
path: spec.scrapeClasses[0].default
23+
value: true
24+
- lengthEqual:
25+
path: spec.scrapeClasses
26+
count: 1
27+
28+
- it: the identity comes from the scraped pod's lineage labels, with a placeholder when the pod has none
29+
asserts:
30+
- equal:
31+
path: spec.scrapeClasses[0].relabelConfigs
32+
value:
33+
- targetLabel: apps_cozystack_io_application_group
34+
replacement: "-"
35+
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_group]
36+
regex: (.+)
37+
targetLabel: apps_cozystack_io_application_group
38+
- targetLabel: apps_cozystack_io_application_kind
39+
replacement: "-"
40+
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_kind]
41+
regex: (.+)
42+
targetLabel: apps_cozystack_io_application_kind
43+
- targetLabel: apps_cozystack_io_application_name
44+
replacement: "-"
45+
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_name]
46+
regex: (.+)
47+
targetLabel: apps_cozystack_io_application_name
48+
49+
- it: an exporter-supplied identity is dropped
50+
asserts:
51+
- equal:
52+
path: spec.scrapeClasses[0].metricRelabelConfigs
53+
value:
54+
- action: labeldrop
55+
regex: (exported_)+apps_cozystack_io_application_(group|kind|name)
56+
57+
# vmagent adds up and scrape_* after metric relabeling (VictoriaMetrics
58+
# lib/promscrape addAutoMetrics calls addRows with needRelabel false), so a
59+
# placeholder removed there would stay on those series of every target
60+
# without lineage. The remote-write relabeling sees every series.
61+
- it: the placeholder is removed from every series, including up and scrape_*
62+
asserts:
63+
- equal:
64+
path: spec.inlineRelabelConfig
65+
value:
66+
- sourceLabels: [apps_cozystack_io_application_group]
67+
regex: "-"
68+
targetLabel: apps_cozystack_io_application_group
69+
replacement: ""
70+
- sourceLabels: [apps_cozystack_io_application_kind]
71+
regex: "-"
72+
targetLabel: apps_cozystack_io_application_kind
73+
replacement: ""
74+
- sourceLabels: [apps_cozystack_io_application_name]
75+
regex: "-"
76+
targetLabel: apps_cozystack_io_application_name
77+
replacement: ""

‎packages/system/monitoring/templates/vm/vmagent.yaml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,39 @@ spec:
2424

2525
scrapeInterval: 30s
2626
selectAllByDefault: false
27+
# The class sets all three identity labels on every target it covers, "-"
28+
# when the pod has no lineage ("-" is not a valid Kubernetes label value,
29+
# so it never collides with a real one). Under honorLabels false every
30+
# exporter-supplied copy then collides, is renamed to exported_* and
31+
# dropped. Under honorLabels true the exporter's copy wins, so such a
32+
# scrape must drop these labels itself. A scrape object's own relabel
33+
# rules run after these: they may drop the labels but must never set them.
34+
# Not covered: VMProbe (operator v0.68.4 takes only auth from a class) and
35+
# raw scrape configs. The placeholder is removed by the remote-write
36+
# relabeling below.
37+
scrapeClasses:
38+
- name: application-identity
39+
default: true
40+
relabelConfigs:
41+
{{- range list "group" "kind" "name" }}
42+
- targetLabel: apps_cozystack_io_application_{{ . }}
43+
replacement: "-"
44+
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_{{ . }}]
45+
regex: (.+)
46+
targetLabel: apps_cozystack_io_application_{{ . }}
47+
{{- end }}
48+
metricRelabelConfigs:
49+
- action: labeldrop
50+
regex: (exported_)+apps_cozystack_io_application_(group|kind|name)
51+
# Not in metricRelabelConfigs: vmagent adds up and scrape_* after metric
52+
# relabeling, so the placeholder would stay on those series.
53+
inlineRelabelConfig:
54+
{{- range list "group" "kind" "name" }}
55+
- sourceLabels: [apps_cozystack_io_application_{{ . }}]
56+
regex: "-"
57+
targetLabel: apps_cozystack_io_application_{{ . }}
58+
replacement: ""
59+
{{- end }}
2760
{{- with .Values.vmagent.inlineScrapeConfig }}
2861
inlineScrapeConfig: |
2962
{{- . | nindent 4 }}

‎packages/system/monitoring/tests/vmagent_application_identity_test.yaml‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
suite: tenant vmagent stamps application identity on exporter series
2+
templates:
3+
- templates/vm/vmagent.yaml
4+
release:
5+
name: monitoring
6+
namespace: tenant-foo
7+
8+
# Same rules as the cluster-wide vmagent in monitoring-agents, pinned there
9+
# by the same assertions; see that suite for why the placeholder and the
10+
# drop exist.
11+
tests:
12+
- it: every scrape gets the application identity class unless it names another
13+
asserts:
14+
- equal:
15+
path: spec.scrapeClasses[0].name
16+
value: application-identity
17+
- equal:
18+
path: spec.scrapeClasses[0].default
19+
value: true
20+
- lengthEqual:
21+
path: spec.scrapeClasses
22+
count: 1
23+
24+
- it: the identity comes from the scraped pod's lineage labels, with a placeholder when the pod has none
25+
asserts:
26+
- equal:
27+
path: spec.scrapeClasses[0].relabelConfigs
28+
value:
29+
- targetLabel: apps_cozystack_io_application_group
30+
replacement: "-"
31+
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_group]
32+
regex: (.+)
33+
targetLabel: apps_cozystack_io_application_group
34+
- targetLabel: apps_cozystack_io_application_kind
35+
replacement: "-"
36+
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_kind]
37+
regex: (.+)
38+
targetLabel: apps_cozystack_io_application_kind
39+
- targetLabel: apps_cozystack_io_application_name
40+
replacement: "-"
41+
- sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_name]
42+
regex: (.+)
43+
targetLabel: apps_cozystack_io_application_name
44+
45+
- it: an exporter-supplied identity is dropped
46+
asserts:
47+
- equal:
48+
path: spec.scrapeClasses[0].metricRelabelConfigs
49+
value:
50+
- action: labeldrop
51+
regex: (exported_)+apps_cozystack_io_application_(group|kind|name)
52+
53+
# vmagent adds up and scrape_* after metric relabeling (VictoriaMetrics
54+
# lib/promscrape addAutoMetrics calls addRows with needRelabel false), so a
55+
# placeholder removed there would stay on those series of every target
56+
# without lineage. The remote-write relabeling sees every series.
57+
- it: the placeholder is removed from every series, including up and scrape_*
58+
asserts:
59+
- equal:
60+
path: spec.inlineRelabelConfig
61+
value:
62+
- sourceLabels: [apps_cozystack_io_application_group]
63+
regex: "-"
64+
targetLabel: apps_cozystack_io_application_group
65+
replacement: ""
66+
- sourceLabels: [apps_cozystack_io_application_kind]
67+
regex: "-"
68+
targetLabel: apps_cozystack_io_application_kind
69+
replacement: ""
70+
- sourceLabels: [apps_cozystack_io_application_name]
71+
regex: "-"
72+
targetLabel: apps_cozystack_io_application_name
73+
replacement: ""

0 commit comments

Comments
 (0)