Skip to content

Commit 8502629

Browse files
committed
fix(piraeus-operator): keep the LINSTOR error-report alerts quiet when the controller pod moves
linstor_error_reports_count comes from the LINSTOR controller, so its series carry the controller pod's instance and controller_node labels. When the pod moves to another node, or an external label of the scrape changes, the existing report counts come back as new series. VictoriaMetrics' increase() counts the first samples of a series from zero when they are small, and report counts are small, so linstorSatelliteErrorRate fires for every node with an old report and keeps firing for 15 minutes. The up guard matches on instance and lets the new series through. The controller also leaves a satellite out of the metric while it is not connected, so a satellite that comes back after more than five minutes has its old count counted again in the same way. Keyed on hostname, a moved controller or a changed label continues the same series, and holding the last value for an hour bridges a disconnect or a slow controller restart. The up guard matches on() because the result no longer has an instance label. The controller rule takes the same form. Its hostname is the pod name and a new pod starts with no reports, but a label change hit it the same way. Assisted-by: LLM Signed-off-by: Yan Bondarenko <[email protected]>
1 parent aed7fa8 commit 8502629

3 files changed

Lines changed: 233 additions & 3 deletions

File tree

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
#!/usr/bin/env bats
2+
# Replays the LINSTOR error-report alerts from
3+
# packages/system/piraeus-operator/alerts/piraeus-datastore.yaml through
4+
# vmalert-tool, pinned to the VictoriaMetrics release that the
5+
# victoria-metrics-operator deploys by default. Its increase() counts the first
6+
# samples of a series as new when they are small, and the report counts are
7+
# small, so the cases pin that a series which starts over after a controller
8+
# move, a label change or a disconnect does not report old error reports again.
9+
# Only pulling the image touches the network.
10+
11+
load test_helper
12+
13+
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
14+
VMALERT_TOOL_IMAGE="victoriametrics/vmalert-tool:v1.136.0@sha256:417346d7b6fdb66d8e3984887c2ad114535ec7d983eeef53381b56c776f333f1"
15+
16+
@test "LINSTOR error-report alerts pass their vmalert-tool cases" {
17+
docker info >/dev/null 2>&1 || {
18+
echo "docker is required to run the cases in $VMALERT_TOOL_IMAGE" >&2
19+
return 1
20+
}
21+
work=$(mktemp -d)
22+
yq '.spec' "$REPO_ROOT/packages/system/piraeus-operator/alerts/piraeus-datastore.yaml" > "$work/rules.yaml"
23+
cp "$REPO_ROOT/hack/testdata/linstor-error-report-alerts/unittest.yaml" "$work/unittest.yaml"
24+
rc=0
25+
docker run --rm --network none -v "$work:/w:ro" "$VMALERT_TOOL_IMAGE" \
26+
unittest --files /w/unittest.yaml > "$work/out" 2>&1 || rc=$?
27+
cat "$work/out"
28+
passed=$(grep -cx 'SUCCESS' "$work/out" || true)
29+
rm -rf "$work"
30+
[ "$rc" -eq 0 ]
31+
[ "$passed" -eq 1 ]
32+
}
Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
rule_files:
2+
- rules.yaml
3+
4+
evaluation_interval: 15s
5+
6+
tests:
7+
- name: controller pod moves to another node
8+
interval: 30s
9+
input_series:
10+
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370", controller_node="node-a"}'
11+
values: '1x38 0 stale'
12+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", controller_node="node-a", hostname="node-a", module="SATELLITE"}'
13+
values: '1x38 stale'
14+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", controller_node="node-a", hostname="node-b", module="SATELLITE"}'
15+
values: '2x38 stale'
16+
- series: 'up{job="linstor-controller", instance="10.0.0.2:3370", controller_node="node-b"}'
17+
values: '_x41 1x79'
18+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.2:3370", controller_node="node-b", hostname="node-a", module="SATELLITE"}'
19+
values: '_x41 1x58 2x20'
20+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.2:3370", controller_node="node-b", hostname="node-b", module="SATELLITE"}'
21+
values: '_x41 2x79'
22+
alert_rule_test:
23+
- eval_time: 21m
24+
groupname: linstor.rules
25+
alertname: linstorSatelliteErrorRate
26+
- eval_time: 28m
27+
groupname: linstor.rules
28+
alertname: linstorSatelliteErrorRate
29+
- eval_time: 35m
30+
groupname: linstor.rules
31+
alertname: linstorSatelliteErrorRate
32+
- eval_time: 45m
33+
groupname: linstor.rules
34+
alertname: linstorSatelliteErrorRate
35+
- eval_time: 51m
36+
groupname: linstor.rules
37+
alertname: linstorSatelliteErrorRate
38+
exp_alerts:
39+
- exp_labels:
40+
hostname: node-a
41+
severity: warning
42+
exp_annotations:
43+
description: |
44+
LINSTOR Satellite "node-a" reports 1 errors in the last 15 minutes.
45+
Use "linstor error-reports list --nodes node-a" to inspect the reports.
46+
47+
- name: an external label of the scrape changes
48+
interval: 30s
49+
input_series:
50+
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370", cluster="cozystack"}'
51+
values: '1x39 stale'
52+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cozystack", hostname="node-a", module="SATELLITE"}'
53+
values: '1x39 stale'
54+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cozystack", hostname="linstor-controller-abc", module="CONTROLLER"}'
55+
values: '1x39 stale'
56+
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370", cluster="cluster-a"}'
57+
values: '_x41 1x79'
58+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cluster-a", hostname="node-a", module="SATELLITE"}'
59+
values: '_x41 1x79'
60+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cluster-a", hostname="linstor-controller-abc", module="CONTROLLER"}'
61+
values: '_x41 1x79'
62+
alert_rule_test:
63+
- eval_time: 21m
64+
groupname: linstor.rules
65+
alertname: linstorSatelliteErrorRate
66+
- eval_time: 30m
67+
groupname: linstor.rules
68+
alertname: linstorSatelliteErrorRate
69+
- eval_time: 21m
70+
groupname: linstor.rules
71+
alertname: linstorControllerErrorRate
72+
- eval_time: 30m
73+
groupname: linstor.rules
74+
alertname: linstorControllerErrorRate
75+
76+
- name: a satellite drops out of the metric while it is disconnected
77+
interval: 30s
78+
input_series:
79+
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370"}'
80+
values: '1x220'
81+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-a", module="SATELLITE"}'
82+
values: '1x38 stale _x80 1x100'
83+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-b", module="SATELLITE"}'
84+
values: '2x220'
85+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-c", module="SATELLITE"}'
86+
values: '1x38 stale _x80 2x100'
87+
alert_rule_test:
88+
- eval_time: 61m
89+
groupname: linstor.rules
90+
alertname: linstorSatelliteErrorRate
91+
exp_alerts:
92+
- exp_labels:
93+
hostname: node-c
94+
severity: warning
95+
exp_annotations:
96+
description: |
97+
LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes.
98+
Use "linstor error-reports list --nodes node-c" to inspect the reports.
99+
- eval_time: 70m
100+
groupname: linstor.rules
101+
alertname: linstorSatelliteErrorRate
102+
exp_alerts:
103+
- exp_labels:
104+
hostname: node-c
105+
severity: warning
106+
exp_annotations:
107+
description: |
108+
LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes.
109+
Use "linstor error-reports list --nodes node-c" to inspect the reports.
110+
- eval_time: 76m
111+
groupname: linstor.rules
112+
alertname: linstorSatelliteErrorRate
113+
114+
- name: a new report fires for the 15 minute window
115+
interval: 30s
116+
input_series:
117+
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370"}'
118+
values: '1x120'
119+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-a", module="SATELLITE"}'
120+
values: '1x39 2x80'
121+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-b", module="SATELLITE"}'
122+
values: '2x120'
123+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-c", module="SATELLITE"}'
124+
values: '0x39 1x80'
125+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="linstor-controller-abc", module="CONTROLLER"}'
126+
values: '0x39 1x80'
127+
alert_rule_test:
128+
- eval_time: 21m
129+
groupname: linstor.rules
130+
alertname: linstorSatelliteErrorRate
131+
exp_alerts:
132+
- exp_labels:
133+
hostname: node-a
134+
severity: warning
135+
exp_annotations:
136+
description: |
137+
LINSTOR Satellite "node-a" reports 1 errors in the last 15 minutes.
138+
Use "linstor error-reports list --nodes node-a" to inspect the reports.
139+
- exp_labels:
140+
hostname: node-c
141+
severity: warning
142+
exp_annotations:
143+
description: |
144+
LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes.
145+
Use "linstor error-reports list --nodes node-c" to inspect the reports.
146+
- eval_time: 34m
147+
groupname: linstor.rules
148+
alertname: linstorSatelliteErrorRate
149+
exp_alerts:
150+
- exp_labels:
151+
hostname: node-a
152+
severity: warning
153+
exp_annotations:
154+
description: |
155+
LINSTOR Satellite "node-a" reports 1 errors in the last 15 minutes.
156+
Use "linstor error-reports list --nodes node-a" to inspect the reports.
157+
- exp_labels:
158+
hostname: node-c
159+
severity: warning
160+
exp_annotations:
161+
description: |
162+
LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes.
163+
Use "linstor error-reports list --nodes node-c" to inspect the reports.
164+
- eval_time: 36m
165+
groupname: linstor.rules
166+
alertname: linstorSatelliteErrorRate
167+
- eval_time: 21m
168+
groupname: linstor.rules
169+
alertname: linstorControllerErrorRate
170+
exp_alerts:
171+
- exp_labels:
172+
hostname: linstor-controller-abc
173+
severity: warning
174+
exp_annotations:
175+
description: |
176+
LINSTOR Controller reports 1 errors in the last 15 minutes.
177+
Use "linstor error-reports list --since 15minutes" to see them.
178+
- eval_time: 36m
179+
groupname: linstor.rules
180+
alertname: linstorControllerErrorRate
181+
182+
- name: a failed controller scrape still holds the alert back
183+
interval: 30s
184+
input_series:
185+
- series: 'up{job="linstor-controller", instance="10.0.0.1:3370"}'
186+
values: '1x19 0x140 1x60'
187+
- series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-a", module="SATELLITE"}'
188+
values: '1x19 stale _x140 1x60'
189+
alert_rule_test:
190+
- eval_time: 85m
191+
groupname: linstor.rules
192+
alertname: linstorSatelliteErrorRate
193+
- eval_time: 93m
194+
groupname: linstor.rules
195+
alertname: linstorSatelliteErrorRate
196+
- eval_time: 100m
197+
groupname: linstor.rules
198+
alertname: linstorSatelliteErrorRate

‎packages/system/piraeus-operator/alerts/piraeus-datastore.yaml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@ spec:
2929
LINSTOR Satellite "{{ $labels.hostname }}" reports {{ $value }} errors in the last 15 minutes.
3030
Use "linstor error-reports list --nodes {{ $labels.hostname }}" to inspect the reports.
3131
expr: |
32-
increase(linstor_error_reports_count{module="SATELLITE"}[15m]) > 0
33-
and on(instance, job)
32+
increase((max by (hostname) (last_over_time(linstor_error_reports_count{module="SATELLITE"}[1h])))[15m:1m]) > 0
33+
and on()
3434
min_over_time(up{job="linstor-controller"}[15m]) == 1
3535
labels:
3636
severity: warning
@@ -39,7 +39,7 @@ spec:
3939
description: |
4040
LINSTOR Controller reports {{ $value }} errors in the last 15 minutes.
4141
Use "linstor error-reports list --since 15minutes" to see them.
42-
expr: increase(linstor_error_reports_count{module="CONTROLLER"}[15m]) > 0
42+
expr: increase((max by (hostname) (last_over_time(linstor_error_reports_count{module="CONTROLLER"}[1h])))[15m:1m]) > 0
4343
labels:
4444
severity: warning
4545
- alert: linstorSatelliteNotOnline

0 commit comments

Comments
 (0)