Skip to content

Commit 9d4221b

Browse files
committed
fix(monitoring): send alerts to every Alertmanager replica
vmalert had a single notifier, the headless vmalertmanager-alertmanager Service. It resolves to all three pods, but each alert reached one replica at a time and moved to another when vmalert reconnected. On clusters running 1.6.4 all active alerts sat on one replica while the other two had none, and on one of them the alert count of each replica changed about 170 times a day. The Alertmanager cluster deduplicates notifications only when every replica receives every alert. The replica an alert left let it expire and sent a resolved notification, and the replica it reached sent a new firing one. vmalert now lists each Alertmanager pod as its own notifier, using the <name>-<i>.<service>.<namespace>.svc address the operator itself gives StatefulSet pods. The replica count comes from one helper that also sets spec.replicaCount of the VMAlertmanager, so the two cannot drift. Assisted-by: LLM Signed-off-by: Yan Bondarenko <[email protected]>
1 parent 3a46360 commit 9d4221b

4 files changed

Lines changed: 68 additions & 2 deletions

File tree

‎packages/system/monitoring/templates/_helpers.tpl‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -327,3 +327,12 @@
327327
{{- define "monitoring.tracingCentral" -}}
328328
{{- if and (ne .Release.Namespace "tenant-root") (eq (toString (dig "tracingCentral" false (.Values._namespace | default dict))) "true") -}}true{{- end -}}
329329
{{- end -}}
330+
331+
{{- /*
332+
Alertmanager replica count. vmalert lists each replica as its own
333+
notifier from the same number, because the Alertmanager cluster only
334+
deduplicates notifications when every replica receives every alert.
335+
*/}}
336+
{{- define "monitoring.alertmanager.replicas" -}}
337+
3
338+
{{- end -}}

‎packages/system/monitoring/templates/alerta/alerta.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -419,7 +419,7 @@ spec:
419419
apps.cozystack.io/application.group: apps.cozystack.io
420420
apps.cozystack.io/application.kind: Monitoring
421421
apps.cozystack.io/application.name: {{ $.Release.Name }}
422-
replicaCount: 3
422+
replicaCount: {{ include "monitoring.alertmanager.replicas" $ }}
423423
configSecret: alertmanager
424424
{{- if and $emailEnabled $email.authUsername }}
425425
secrets:

‎packages/system/monitoring/templates/vm/vmalert.yaml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,9 @@ spec:
1616
extraArgs:
1717
remoteWrite.disablePathAppend: "true"
1818
notifiers:
19-
- url: http://vmalertmanager-alertmanager.{{ $.Release.Namespace }}.svc:9093
19+
{{- range $i := until (int (include "monitoring.alertmanager.replicas" $)) }}
20+
- url: http://vmalertmanager-alertmanager-{{ $i }}.vmalertmanager-alertmanager.{{ $.Release.Namespace }}.svc:9093
21+
{{- end }}
2022
remoteRead:
2123
url: http://vmselect-{{ .name }}.{{ $.Release.Namespace }}.svc:8481/select/0/prometheus
2224
remoteWrite:
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
suite: vmalert notifies every Alertmanager replica
2+
templates:
3+
- templates/vm/vmalert.yaml
4+
- templates/alerta/alerta.yaml
5+
6+
release:
7+
name: monitoring
8+
namespace: tenant-root
9+
10+
set:
11+
_cluster:
12+
root-host: example.org
13+
_namespace:
14+
host: tenant-root.example.org
15+
ingress: tenant-root
16+
metricsStorages:
17+
- name: shortterm
18+
retentionPeriod: "3d"
19+
deduplicationInterval: "15s"
20+
21+
# The Alertmanager cluster deduplicates notifications only when every replica
22+
# receives every alert. A notifier pointing at the headless Service reaches
23+
# one pod per connection, so an alert lives on whichever replica that
24+
# connection landed on and moves when vmalert reconnects. Each replica it
25+
# leaves sends a resolved notification and each one it reaches sends a new
26+
# firing one.
27+
tests:
28+
- it: lists each Alertmanager pod as a notifier instead of the shared Service
29+
template: templates/vm/vmalert.yaml
30+
asserts:
31+
- equal:
32+
path: spec.notifiers
33+
value:
34+
- url: http://vmalertmanager-alertmanager-0.vmalertmanager-alertmanager.tenant-root.svc:9093
35+
- url: http://vmalertmanager-alertmanager-1.vmalertmanager-alertmanager.tenant-root.svc:9093
36+
- url: http://vmalertmanager-alertmanager-2.vmalertmanager-alertmanager.tenant-root.svc:9093
37+
38+
- it: runs as many Alertmanager replicas as vmalert has notifiers
39+
template: templates/alerta/alerta.yaml
40+
documentSelector:
41+
path: kind
42+
value: VMAlertmanager
43+
asserts:
44+
- equal:
45+
path: spec.replicaCount
46+
value: 3
47+
48+
- it: addresses the Alertmanager pods in the release namespace
49+
template: templates/vm/vmalert.yaml
50+
release:
51+
namespace: tenant-foo
52+
asserts:
53+
- equal:
54+
path: spec.notifiers[2].url
55+
value: http://vmalertmanager-alertmanager-2.vmalertmanager-alertmanager.tenant-foo.svc:9093

0 commit comments

Comments
 (0)