Skip to content

Commit 144fd7b

Browse files
fix(kubernetes, monitoring): send tenant cluster metrics to the tenant's vminsert
The monitoring-agents chart reads vmagent.remoteWrite.urls as a list and takes the tenant label from global.target, while the kubernetes app still passed a single remoteWrite.url and vmagent.externalLabels.tenant. Both keys were dropped in silence, so the vmagent of every tenant cluster fell back to the chart defaults: it wrote to vminsert services in cozy-monitoring that do not exist inside the tenant cluster, queued the failed writes on the node disk, and labelled every series tenant="cozy-monitoring". No tenant cluster metric reached the tenant's Monitoring. The app now passes urls. The chart prefers externalLabels.tenant over global.target, so the label stays on the tenant that owns the cluster rather than on the tenant whose Monitoring receives the data, which can be a parent. The label is quoted, so a tenant name that reads as a YAML boolean or number still renders as the string externalLabels requires. The root install sets neither key and renders the same label values. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent 0b3f5dd commit 144fd7b

4 files changed

Lines changed: 110 additions & 2 deletions

File tree

‎packages/apps/kubernetes/templates/helmreleases/monitoring-agents.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,8 @@ spec:
4545
cluster: {{ .Release.Name }}
4646
tenant: {{ .Release.Namespace }}
4747
remoteWrite:
48-
url: http://vminsert-shortterm.{{ $targetTenant }}.svc.{{ $clusterDomain }}:8480/insert/0/prometheus
48+
urls:
49+
- http://vminsert-shortterm.{{ $targetTenant }}.svc.{{ $clusterDomain }}:8480/insert/0/prometheus
4950
fluent-bit:
5051
readinessProbe:
5152
httpGet:
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
suite: monitoring-agents HelmRelease values match the monitoring-agents chart
2+
3+
# The values below are consumed by packages/system/monitoring-agents, which
4+
# reads vmagent.remoteWrite.urls as a list and the tenant label from
5+
# vmagent.externalLabels.tenant. A key the chart does not read is dropped in
6+
# silence, and the chart then falls back to cozy-monitoring defaults that do
7+
# not resolve inside a tenant cluster. Its counterpart is
8+
# packages/system/monitoring-agents/tests/vmagent_remote_write_test.yaml.
9+
templates:
10+
- templates/helmreleases/monitoring-agents.yaml
11+
12+
values:
13+
- values/common.yaml
14+
15+
release:
16+
name: kubernetes-clusterprod
17+
namespace: tenant-a
18+
19+
set:
20+
addons.monitoringAgents.enabled: true
21+
_namespace.monitoring: tenant-root
22+
_cluster:
23+
cluster-domain: cozy.local
24+
25+
tests:
26+
- it: sends metrics to the shortterm vminsert of the monitoring tenant as a urls list
27+
asserts:
28+
- equal:
29+
path: spec.values.vmagent.remoteWrite
30+
value:
31+
urls:
32+
- http://vminsert-shortterm.tenant-root.svc.cozy.local:8480/insert/0/prometheus
33+
34+
- it: labels series with the tenant that owns the cluster, not the monitoring tenant
35+
asserts:
36+
- equal:
37+
path: spec.values.vmagent.externalLabels
38+
value:
39+
cluster: kubernetes-clusterprod
40+
tenant: tenant-a
41+
- notExists:
42+
path: spec.values.global

‎packages/system/monitoring-agents/templates/vmagent.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ spec:
1010
shardCount: 1
1111
externalLabels:
1212
cluster: {{ .Values.vmagent.externalLabels.cluster }}
13-
tenant: {{ .Values.global.target }}
13+
tenant: {{ .Values.vmagent.externalLabels.tenant | default .Values.global.target | quote }}
1414
extraArgs:
1515
{{- toYaml (deepCopy .Values.vmagent.extraArgs | mergeOverwrite (fromYaml (include "monitoring-agents.vmagent.defaultExtraArgs" .))) | nindent 4 }}
1616
remoteWrite:
Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
suite: vmagent remote write targets and tenant label
2+
3+
# The root bundle sets only global.target, and the chart defaults derive both
4+
# the remote write URLs and the tenant label from it. A tenant Kubernetes
5+
# cluster gets this chart through packages/apps/kubernetes instead, which
6+
# passes vmagent.remoteWrite.urls and vmagent.externalLabels.tenant and leaves
7+
# global.target at its default. If either value is dropped, vmagent writes to
8+
# cozy-monitoring services that do not exist in the tenant cluster, or labels
9+
# every series tenant="cozy-monitoring".
10+
templates:
11+
- templates/vmagent.yaml
12+
release:
13+
name: cozy-monitoring-agents
14+
namespace: cozy-monitoring
15+
tests:
16+
- it: root install writes to both stores of global.target and labels series with it
17+
set:
18+
global.target: tenant-root
19+
asserts:
20+
- equal:
21+
path: spec.remoteWrite
22+
value:
23+
- url: http://vminsert-shortterm.tenant-root.svc:8480/insert/0/prometheus
24+
- url: http://vminsert-longterm.tenant-root.svc:8480/insert/0/prometheus
25+
- equal:
26+
path: spec.externalLabels.tenant
27+
value: tenant-root
28+
29+
- it: tenant cluster writes only to the URLs its parent app passes
30+
set:
31+
vmagent:
32+
externalLabels:
33+
cluster: kubernetes-clusterprod
34+
tenant: tenant-a
35+
remoteWrite:
36+
urls:
37+
- http://vminsert-shortterm.tenant-a.svc.cozy.local:8480/insert/0/prometheus
38+
asserts:
39+
- equal:
40+
path: spec.remoteWrite
41+
value:
42+
- url: http://vminsert-shortterm.tenant-a.svc.cozy.local:8480/insert/0/prometheus
43+
44+
- it: tenant cluster labels series with the owning tenant, not global.target
45+
set:
46+
vmagent:
47+
externalLabels:
48+
cluster: kubernetes-clusterprod
49+
tenant: tenant-a
50+
asserts:
51+
- equal:
52+
path: spec.externalLabels
53+
value:
54+
cluster: kubernetes-clusterprod
55+
tenant: tenant-a
56+
57+
- it: tenant label stays a string when the value reads as a YAML scalar
58+
set:
59+
vmagent:
60+
externalLabels:
61+
tenant: "true"
62+
asserts:
63+
- equal:
64+
path: spec.externalLabels.tenant
65+
value: "true"

0 commit comments

Comments
 (0)