Skip to content

Commit 1287f29

Browse files
myasnikovdaniilmyasnikovdaniil
committed
fix(monitoring): scope tenant VMAlert to its own rule namespaces
A tenant's VMAlert was rendered with selectAllByDefault and no rule selector, which by the operator's rules means "select every VMRule in the cluster". It then evaluated those rules against the tenant's own vmselect, which only holds that tenant's metrics. Infrastructure alerts are written as absent(up{job="kubelet"} == 1) and friends, so they fired in every tenant precisely because the series is absent there -- the noise reported in #2775: CDI operator is down, plus Kubelet / KubeAPI / KubeControllerManager / KubeScheduler disappeared from target discovery. Those rules live outside any tenant namespace (cozy-monitoring holds 38 of them, cozy-kubevirt-cdi and cozy-linstor one each), so no change to where they are rendered can fix this; the selection is what is too wide. Scope a tenant VMAlert by the namespace.cozystack.io/monitoring label, the same way vmagent.yaml already scopes scrapes. The label's value names the tenant whose monitoring stack covers that namespace, so the selector matches the tenant's own namespace plus any child tenant inheriting this stack -- exactly the namespaces whose metrics reach this vmselect. ruleSelector stays unset, so all VMRules in those namespaces continue to match, tenant-authored ones included. Root monitoring keeps selectAllByDefault: the infrastructure namespaces carry no such label, and a selector would silence the platform's own alerts. Root behaviour is therefore unchanged. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 67b4e23 commit 1287f29

2 files changed

Lines changed: 74 additions & 0 deletions

File tree

‎packages/system/monitoring/templates/vm/vmalert.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,34 @@ spec:
2222
remoteWrite:
2323
url: http://vminsert-{{ .name }}.{{ $.Release.Namespace }}.svc:8480/insert/0/prometheus/api/v1/write
2424
resources: {}
25+
{{- if eq $.Release.Namespace "tenant-root" }}
26+
{{- /*
27+
Root monitoring is the platform-wide view: it must keep seeing the
28+
infrastructure VMRules, which live outside any tenant namespace
29+
(cozy-monitoring, cozy-kubevirt-cdi, cozy-linstor, ...). Those namespaces
30+
carry no namespace.cozystack.io/monitoring label, so any selector would
31+
cut them off.
32+
*/}}
2533
selectAllByDefault: true
34+
{{- else }}
35+
{{- /*
36+
A tenant VMAlert evaluates rules against its own vmselect, which only holds
37+
that tenant's metrics. With selectAllByDefault and no selector it picks up
38+
every VMRule in the cluster, so infrastructure alerts written as
39+
absent(up{job="kubelet"} == 1) fire in the tenant precisely because the
40+
series is absent there (issue #2775).
41+
42+
Scope rules the same way vmagent.yaml scopes scrapes: by the
43+
namespace.cozystack.io/monitoring label, whose value names the tenant whose
44+
monitoring stack covers that namespace (apps/tenant/templates/namespace.yaml).
45+
That selects this tenant's own namespace plus any child tenant inheriting
46+
this monitoring stack -- exactly the namespaces whose metrics reach this
47+
vmselect. ruleSelector stays unset, so all VMRules in those namespaces match.
48+
*/}}
49+
selectAllByDefault: false
50+
ruleNamespaceSelector:
51+
matchLabels:
52+
namespace.cozystack.io/monitoring: {{ $.Release.Namespace }}
53+
{{- end }}
2654
{{- break }}
2755
{{- end }}

‎packages/system/monitoring/tests/vmalert_rule_scope_test.yaml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
suite: VMAlert selects VMRules only from namespaces its own metrics come from
2+
3+
# A tenant VMAlert queries the tenant's own vmselect. Left with
4+
# selectAllByDefault and no selector it also picks up infrastructure VMRules
5+
# from cozy-monitoring, cozy-kubevirt-cdi and friends, and those are written as
6+
# absent(up{job="..."} == 1) -- so they fire in the tenant precisely because the
7+
# series is absent there (issue #2775). Scoping by the
8+
# namespace.cozystack.io/monitoring label mirrors how vmagent.yaml scopes
9+
# scrapes. Root keeps selectAllByDefault, because the infrastructure namespaces
10+
# carry no such label and a selector would silence the platform's own alerts.
11+
12+
set:
13+
_namespace:
14+
host: example.org
15+
ingress: tenant-root
16+
_cluster:
17+
expose-ingress: ingress
18+
host: example.org
19+
20+
tests:
21+
- it: keeps the cluster-wide selection for root monitoring
22+
release:
23+
name: monitoring-system
24+
namespace: tenant-root
25+
templates:
26+
- templates/vm/vmalert.yaml
27+
asserts:
28+
- equal:
29+
path: spec.selectAllByDefault
30+
value: true
31+
- notExists:
32+
path: spec.ruleNamespaceSelector
33+
34+
- it: scopes a tenant to the namespaces its monitoring stack covers
35+
release:
36+
name: monitoring-system
37+
namespace: tenant-foo
38+
templates:
39+
- templates/vm/vmalert.yaml
40+
asserts:
41+
- equal:
42+
path: spec.selectAllByDefault
43+
value: false
44+
- equal:
45+
path: spec.ruleNamespaceSelector.matchLabels["namespace.cozystack.io/monitoring"]
46+
value: tenant-foo

0 commit comments

Comments
 (0)