Skip to content

Commit 55d6c52

Browse files
fix(kubernetes, monitoring): send tenant cluster metrics to the tenant's vminsert
The monitoring-agents chart reads vmagent.remoteWrite.urls as a list and takes the tenant label from global.target, while the kubernetes app still passed a single remoteWrite.url and vmagent.externalLabels.tenant. Both keys were dropped in silence, so the vmagent of every tenant cluster fell back to the chart defaults: it wrote to vminsert services in cozy-monitoring that do not exist inside the tenant cluster, queued the failed writes on the node disk, and labelled every series tenant="cozy-monitoring". No tenant cluster metric reached the tenant's Monitoring. The tenant label has to name the tenant that owns the cluster, which is not always the tenant whose Monitoring receives the data: a tenant without its own Monitoring inherits a parent's. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent 0b3f5dd commit 55d6c52

4 files changed

Lines changed: 110 additions & 2 deletions

File tree

‎packages/apps/kubernetes/templates/helmreleases/monitoring-agents.yaml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,8 @@ spec:
4545
cluster: {{ .Release.Name }}
4646
tenant: {{ .Release.Namespace }}
4747
remoteWrite:
48-
url: http://vminsert-shortterm.{{ $targetTenant }}.svc.{{ $clusterDomain }}:8480/insert/0/prometheus
48+
urls:
49+
- http://vminsert-shortterm.{{ $targetTenant }}.svc.{{ $clusterDomain }}:8480/insert/0/prometheus
4950
fluent-bit:
5051
readinessProbe:
5152
httpGet:
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
suite: monitoring-agents HelmRelease values match the monitoring-agents chart
2+
3+
# The values below are consumed by packages/system/monitoring-agents, which
4+
# reads vmagent.remoteWrite.urls as a list and the tenant label from
5+
# vmagent.externalLabels.tenant. A key the chart does not read is dropped in
6+
# silence, and the chart then falls back to cozy-monitoring defaults that do
7+
# not resolve inside a tenant cluster. Its counterpart is
8+
# packages/system/monitoring-agents/tests/vmagent_remote_write_test.yaml.
9+
templates:
10+
- templates/helmreleases/monitoring-agents.yaml
11+
12+
values:
13+
- values/common.yaml
14+
15+
release:
16+
name: kubernetes-clusterprod
17+
namespace: tenant-a
18+
19+
set:
20+
addons.monitoringAgents.enabled: true
21+
_namespace.monitoring: tenant-root
22+
_cluster:
23+
cluster-domain: cozy.local
24+
25+
tests:
26+
- it: sends metrics to the shortterm vminsert of the monitoring tenant as a urls list
27+
asserts:
28+
- equal:
29+
path: spec.values.vmagent.remoteWrite
30+
value:
31+
urls:
32+
- http://vminsert-shortterm.tenant-root.svc.cozy.local:8480/insert/0/prometheus
33+
34+
- it: labels series with the tenant that owns the cluster, not the monitoring tenant
35+
asserts:
36+
- equal:
37+
path: spec.values.vmagent.externalLabels
38+
value:
39+
cluster: kubernetes-clusterprod
40+
tenant: tenant-a
41+
- notExists:
42+
path: spec.values.global

‎packages/system/monitoring-agents/templates/vmagent.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ spec:
1010
shardCount: 1
1111
externalLabels:
1212
cluster: {{ .Values.vmagent.externalLabels.cluster }}
13-
tenant: {{ .Values.global.target }}
13+
tenant: {{ .Values.vmagent.externalLabels.tenant | default .Values.global.target | quote }}
1414
extraArgs:
1515
{{- toYaml (deepCopy .Values.vmagent.extraArgs | mergeOverwrite (fromYaml (include "monitoring-agents.vmagent.defaultExtraArgs" .))) | nindent 4 }}
1616
remoteWrite:
Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
suite: vmagent remote write targets and tenant label
2+
3+
# The root bundle sets only global.target, and the chart defaults derive both
4+
# the remote write URLs and the tenant label from it. A tenant Kubernetes
5+
# cluster gets this chart through packages/apps/kubernetes instead, which
6+
# passes vmagent.remoteWrite.urls and vmagent.externalLabels.tenant and leaves
7+
# global.target at its default. If either value is dropped, vmagent writes to
8+
# cozy-monitoring services that do not exist in the tenant cluster, or labels
9+
# every series tenant="cozy-monitoring".
10+
templates:
11+
- templates/vmagent.yaml
12+
release:
13+
name: cozy-monitoring-agents
14+
namespace: cozy-monitoring
15+
tests:
16+
- it: root install writes to both stores of global.target and labels series with it
17+
set:
18+
global.target: tenant-root
19+
asserts:
20+
- equal:
21+
path: spec.remoteWrite
22+
value:
23+
- url: http://vminsert-shortterm.tenant-root.svc:8480/insert/0/prometheus
24+
- url: http://vminsert-longterm.tenant-root.svc:8480/insert/0/prometheus
25+
- equal:
26+
path: spec.externalLabels.tenant
27+
value: tenant-root
28+
29+
- it: tenant cluster writes only to the URLs its parent app passes
30+
set:
31+
vmagent:
32+
externalLabels:
33+
cluster: kubernetes-clusterprod
34+
tenant: tenant-a
35+
remoteWrite:
36+
urls:
37+
- http://vminsert-shortterm.tenant-a.svc.cozy.local:8480/insert/0/prometheus
38+
asserts:
39+
- equal:
40+
path: spec.remoteWrite
41+
value:
42+
- url: http://vminsert-shortterm.tenant-a.svc.cozy.local:8480/insert/0/prometheus
43+
44+
- it: tenant cluster labels series with the owning tenant, not global.target
45+
set:
46+
vmagent:
47+
externalLabels:
48+
cluster: kubernetes-clusterprod
49+
tenant: tenant-a
50+
asserts:
51+
- equal:
52+
path: spec.externalLabels
53+
value:
54+
cluster: kubernetes-clusterprod
55+
tenant: tenant-a
56+
57+
- it: tenant label stays a string when the value reads as a YAML scalar
58+
set:
59+
vmagent:
60+
externalLabels:
61+
tenant: "true"
62+
asserts:
63+
- equal:
64+
path: spec.externalLabels.tenant
65+
value: "true"

0 commit comments

Comments
 (0)