Skip to content

Commit c983b5c

Browse files
authored
fix(linstor): use severity warning instead of warn in piraeus-datastore alerts (#3799)
## What this PR does Seven of the piraeus-datastore alert rules label themselves `severity: warn`. Alerta, which the monitoring package ships as the alert sink, validates that value and `warn` is not on its list. Every webhook delivery for these alerts dies with a 500. ``` Severity (warn) is not one of security, critical, major, minor, warning, indeterminate, informational, normal, ok, cleared, debug, trace, unknown ``` Alertmanager retries seven times and drops the notification, and the alert never shows up anywhere. The affected rules are the LINSTOR/DRBD ones, such as storage pool at capacity, lost quorum, unintentional diskless and stuck resync. These are exactly the alerts you least want silently dropped. Found this on a v1.4.6 cluster where `linstorStoragePoolAtCapacity` was genuinely firing. The cluster had 153 dropped notifications in 40 minutes and nothing in Alerta. Renaming the label to `warning`, which the other three rules in the same file already use, stopped the failures immediately and the alerts came through. The same labels sit on `main` and in `v1.6.1`, so a backport to `release-1.6` would be appreciated. The test commit adds a bats guard so the next one gets caught in CI. It walks every `PrometheusRule` and `VMRule` under `packages/` outside vendored `charts/`, helm templates included, and fails on a severity outside Alerta's default alarm model, which is exactly where `warn`, `info`, `error` and `fatal` fall. Helm actions are stripped before the file is read, so a rule behind a `{{- if }}` or an unquoted `{{ .Release.Namespace }}` is checked rather than skipped, and a file that still does not parse fails the contract instead of thinning it out. On the current tree that is 194 alerting rules in 41 scanned files. The guard found one more on `main`. `DatabaseAutoscalerAtMax` in `packages/system/keda/templates/alerts.yaml` carries `severity: info`, which Alerta refuses the same way. It is renamed to `informational`, the value the model has for that level and the one seven other rules in the tree already use. Moving that alert off `info` also stops `InfoInhibitor` in `packages/system/monitoring-agents/alerts/general.rules.yaml` from firing. That rule matches any `severity="info"` alert and carries `severity: major`, so on `main` it usually reached Alerta whenever `DatabaseAutoscalerAtMax` fired, even though the keda alert itself was refused. After this change no rule carries `info`, so operators no longer see that `InfoInhibitor` notification. Fixes #2411. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff. It touches severity labels inside `packages/system/piraeus-operator/alerts/` and `packages/system/keda/templates/`, plus a bats file under `hack/`. No package is added or renamed, and there is no values schema, CRD, tooling the satellites gate on or node contract change. Nothing downstream restates these labels. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(linstor): LINSTOR/DRBD alerts carried severity `warn`, which Alerta rejects, so they were never delivered. They now use the standard `warning` severity, and the keda at-max autoscaler alert moves from `info` to `informational` for the same reason. ```
2 parents 2c2f2c1 + bd2cc0a commit c983b5c

3 files changed

Lines changed: 237 additions & 8 deletions

File tree

‎hack/alert-severity-contract.bats‎

Lines changed: 229 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,229 @@
1+
#!/usr/bin/env bats
2+
3+
# Alerta's default alarm model, the set the webhook accepts. `none` is left out
4+
# on purpose: the Alertmanager config routes severity="none" to a blackhole
5+
# receiver, so such an alert is never delivered, which is the outcome this
6+
# contract exists to catch. Rule files under packages/**/charts/** are out of
7+
# scope: they are re-vendored by `make update`, so a bad severity there is
8+
# fixed upstream or in patches/, not by editing the label.
9+
ALERT_SEVERITY_ALLOWED=" security critical major minor warning indeterminate informational normal ok cleared debug trace unknown "
10+
11+
helm_actions_stripped() {
12+
awk '
13+
function strip_comments(s, pre, rest) {
14+
while (1) {
15+
if (incomment) {
16+
if (match(s, /\*\/[ \t]*-?[}][}]/)) { s = substr(s, RSTART + RLENGTH); incomment = 0 } else { return "" }
17+
}
18+
if (match(s, /[{][{]-?[ \t]*\/\*/)) {
19+
pre = substr(s, 1, RSTART - 1); rest = substr(s, RSTART + RLENGTH)
20+
if (match(rest, /\*\/[ \t]*-?[}][}]/)) { s = pre substr(rest, RSTART + RLENGTH) } else { incomment = 1; return pre }
21+
} else { return s }
22+
}
23+
}
24+
{
25+
line = strip_comments($0)
26+
if (incomment && line ~ /^[ \t]*$/) { next }
27+
if (line ~ /^[ \t]*[{][{].*[}][}][ \t]*$/) { next }
28+
gsub(/[{][{][^}]*[}][}]/, "TEMPLATED", line)
29+
print line
30+
}
31+
' "$1"
32+
}
33+
34+
alert_severity_rows() {
35+
severity_root=${ALERT_SEVERITY_ROOT:-"$(dirname "${BATS_TEST_FILENAME:-$0}")/.."}
36+
severity_files=$(mktemp) || return 1
37+
severity_rows=$(mktemp) || {
38+
rm -f "$severity_files"
39+
return 1
40+
}
41+
severity_stripped=$(mktemp) || {
42+
rm -f "$severity_files" "$severity_rows"
43+
return 1
44+
}
45+
46+
if ! (
47+
cd "$severity_root" &&
48+
git grep -Il -E '^[[:space:]]*kind:[[:space:]]*(PrometheusRule|VMRule)[[:space:]]*$' -- \
49+
'packages/**/*.yaml' \
50+
'packages/**/*.yml' \
51+
':(exclude)packages/**/charts/**'
52+
) > "$severity_files"; then
53+
rm -f "$severity_files" "$severity_rows" "$severity_stripped"
54+
return 1
55+
fi
56+
57+
if ! : > "$severity_rows"; then
58+
rm -f "$severity_files" "$severity_rows" "$severity_stripped"
59+
return 1
60+
fi
61+
while IFS= read -r file; do
62+
if ! (cd "$severity_root" && helm_actions_stripped "$file" > "$severity_stripped" && FILE="$file" yq 'select(.kind == "PrometheusRule" or .kind == "VMRule") | .spec.groups[]? | .rules[]? | select(has("alert")) | [strenv(FILE), .alert, ((.labels.severity // "<missing>") | tostring)] | @tsv' "$severity_stripped") >> "$severity_rows"; then
63+
echo "$file: failed to parse alert rules" >&2
64+
rm -f "$severity_files" "$severity_rows" "$severity_stripped"
65+
return 1
66+
fi
67+
done < "$severity_files"
68+
69+
if ! sed '/^[[:space:]]*$/d' "$severity_rows"; then
70+
rm -f "$severity_files" "$severity_rows" "$severity_stripped"
71+
return 1
72+
fi
73+
rm -f "$severity_files" "$severity_rows" "$severity_stripped"
74+
}
75+
76+
alert_severity_contract() {
77+
rows=$(mktemp) || return 1
78+
if ! alert_severity_rows > "$rows"; then
79+
rm -f "$rows"
80+
return 1
81+
fi
82+
bad=0
83+
tab=$(printf '\t')
84+
while IFS="$tab" read -r file alert severity; do
85+
accepted=0
86+
for allowed in $ALERT_SEVERITY_ALLOWED; do
87+
if [ "$allowed" = "$severity" ]; then
88+
accepted=1
89+
break
90+
fi
91+
done
92+
if [ "$accepted" -eq 0 ]; then
93+
echo "$file: $alert -> '$severity'" >&2
94+
bad=$((bad + 1))
95+
fi
96+
done < "$rows"
97+
rm -f "$rows"
98+
[ "$bad" -eq 0 ] || return 1
99+
}
100+
101+
@test "alert rules are discovered" {
102+
rows=$(mktemp) || return 1
103+
if ! alert_severity_rows > "$rows"; then
104+
rm -f "$rows"
105+
return 1
106+
fi
107+
count=$(wc -l < "$rows" | tr -d ' ')
108+
rm -f "$rows"
109+
[ "$count" -gt 0 ]
110+
}
111+
112+
@test "every alert rule carries a severity the alert sink accepts" {
113+
alert_severity_contract
114+
}
115+
116+
@test "a parser failure cannot leave the contract green" {
117+
fixture=$(mktemp -d)
118+
mkdir -p "$fixture/packages/system/example/alerts"
119+
printf '%s\n' \
120+
'apiVersion: operator.victoriametrics.com/v1beta1' \
121+
'kind: VMRule' \
122+
'spec:' \
123+
' groups:' \
124+
' - name: valid' \
125+
' rules:' \
126+
' - alert: ValidAlert' \
127+
' labels:' \
128+
' severity: warning' \
129+
> "$fixture/packages/system/example/alerts/00-valid.yaml"
130+
printf '%s\n' \
131+
'apiVersion: operator.victoriametrics.com/v1beta1' \
132+
'kind: VMRule' \
133+
'spec: [' \
134+
> "$fixture/packages/system/example/alerts/99-malformed.yaml"
135+
git -C "$fixture" init -q
136+
git -C "$fixture" add packages/system/example/alerts/00-valid.yaml packages/system/example/alerts/99-malformed.yaml
137+
138+
parser_succeeded=0
139+
ALERT_SEVERITY_ROOT="$fixture" alert_severity_rows >/dev/null 2>&1 || parser_succeeded=$?
140+
rm -rf "$fixture"
141+
[ "$parser_succeeded" -ne 0 ]
142+
}
143+
144+
@test "template rule sources cannot bypass the severity contract" {
145+
fixture=$(mktemp -d)
146+
mkdir -p "$fixture/packages/extra/example/templates"
147+
printf '%s\n' \
148+
'apiVersion: monitoring.coreos.com/v1' \
149+
'kind: PrometheusRule' \
150+
'spec:' \
151+
' groups:' \
152+
' - name: template' \
153+
' rules:' \
154+
' - alert: InvalidTemplateSeverity' \
155+
' labels:' \
156+
' severity: warn' \
157+
> "$fixture/packages/extra/example/templates/prometheus-rules.yaml"
158+
git -C "$fixture" init -q
159+
git -C "$fixture" add packages/extra/example/templates/prometheus-rules.yaml
160+
161+
contract_succeeded=0
162+
ALERT_SEVERITY_ROOT="$fixture" alert_severity_contract >/dev/null 2>&1 || contract_succeeded=$?
163+
rm -rf "$fixture"
164+
[ "$contract_succeeded" -ne 0 ]
165+
}
166+
167+
@test "helm template actions do not hide a rule from the contract" {
168+
fixture=$(mktemp -d)
169+
mkdir -p "$fixture/packages/system/example/templates"
170+
printf '%s\n' \
171+
'{{- if .Values.alerts.enabled }}' \
172+
'{{- /*' \
173+
'Rules gated on a value, behind a comment that spans lines.' \
174+
'*/}}' \
175+
'apiVersion: monitoring.coreos.com/v1' \
176+
'kind: PrometheusRule' \
177+
'metadata:' \
178+
' name: example' \
179+
' namespace: {{ .Release.Namespace }}' \
180+
'spec:' \
181+
' groups:' \
182+
' - name: example' \
183+
' rules:' \
184+
' {{- /* one-line comment */}}' \
185+
' - alert: TemplatedRuleWithBadSeverity' \
186+
' expr: up == 0' \
187+
' labels:' \
188+
' severity: warn' \
189+
' annotations:' \
190+
' description: {{ "{{ $labels.instance }}" }} is down' \
191+
'{{- end }}' \
192+
> "$fixture/packages/system/example/templates/alerts.yaml"
193+
git -C "$fixture" init -q
194+
git -C "$fixture" add packages/system/example/templates/alerts.yaml
195+
196+
rows=$(mktemp)
197+
parser_succeeded=0
198+
ALERT_SEVERITY_ROOT="$fixture" alert_severity_rows > "$rows" 2>/dev/null || parser_succeeded=$?
199+
found=$(grep -c 'TemplatedRuleWithBadSeverity' "$rows" || true)
200+
contract_succeeded=0
201+
ALERT_SEVERITY_ROOT="$fixture" alert_severity_contract >/dev/null 2>&1 || contract_succeeded=$?
202+
rm -rf "$fixture" "$rows"
203+
[ "$parser_succeeded" -eq 0 ]
204+
[ "$found" -eq 1 ]
205+
[ "$contract_succeeded" -ne 0 ]
206+
}
207+
208+
@test "two accepted words in one severity are rejected" {
209+
fixture=$(mktemp -d)
210+
mkdir -p "$fixture/packages/system/example/alerts"
211+
printf '%s\n' \
212+
'apiVersion: operator.victoriametrics.com/v1beta1' \
213+
'kind: VMRule' \
214+
'spec:' \
215+
' groups:' \
216+
' - name: joined' \
217+
' rules:' \
218+
' - alert: JoinedSeverity' \
219+
' labels:' \
220+
' severity: critical major' \
221+
> "$fixture/packages/system/example/alerts/00-joined.yaml"
222+
git -C "$fixture" init -q
223+
git -C "$fixture" add packages/system/example/alerts/00-joined.yaml
224+
225+
contract_succeeded=0
226+
ALERT_SEVERITY_ROOT="$fixture" alert_severity_contract >/dev/null 2>&1 || contract_succeeded=$?
227+
rm -rf "$fixture"
228+
[ "$contract_succeeded" -ne 0 ]
229+
}

‎packages/system/keda/templates/alerts.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -146,7 +146,7 @@ spec:
146146
kube_horizontalpodautoscaler_spec_max_replicas{job="kube-state-metrics",horizontalpodautoscaler=~"keda-hpa-.*"}
147147
for: 30m
148148
labels:
149-
severity: info
149+
severity: informational
150150
annotations:
151151
summary: Database autoscaler has been at maxReplicas for 30m.
152152
description: HPA {{ "{{ $labels.horizontalpodautoscaler }}" }} in namespace {{ "{{ $labels.namespace }}" }} has wanted its maximum replica count (autoscaling.maxReplicas) for 30m under sustained load — raise maxReplicas if more read capacity is needed.

‎packages/system/piraeus-operator/alerts/piraeus-datastore.yaml‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ spec:
6363
Storage pool "{{ $labels.storage_pool }}" on node "{{ $labels.node }}" ({{ $labels.driver }}={{ $labels.backing_pool }}) has less than 20% free space available.
6464
expr: ( linstor_storage_pool_capacity_free_bytes / linstor_storage_pool_capacity_total_bytes ) < 0.20
6565
labels:
66-
severity: warn
66+
severity: warning
6767
- name: drbd.rules
6868
rules:
6969
- alert: drbdReactorOffline
@@ -79,46 +79,46 @@ spec:
7979
DRBD Resource "{{ $labels.name }}" on "{{ $labels.node }}" is not connected to "{{ $labels.conn_name }}": {{ $labels.drbd_connection_state }}.
8080
expr: drbd_connection_state{drbd_connection_state!="Connected"} > 0
8181
labels:
82-
severity: warn
82+
severity: warning
8383
- alert: drbdDeviceNotUpToDate
8484
annotations:
8585
description: |
8686
DRBD device "{{ $labels.name }}" on "{{ $labels.node }}" has unexpected device state "{{ $labels.drbd_device_state }}".
8787
expr: drbd_device_state{drbd_device_state!~"UpToDate|Diskless"} > 0
8888
labels:
89-
severity: warn
89+
severity: warning
9090
- alert: drbdDeviceUnintentionalDiskless
9191
annotations:
9292
description: |
9393
DRBD device "{{ $labels.name }}" on "{{ $labels.node }}" is unintenionally diskless.
9494
This usually indicates IO errors reported on the backing device. Check the kernel log.
9595
expr: drbd_device_unintentionaldiskless > 0
9696
labels:
97-
severity: warn
97+
severity: warning
9898
- alert: drbdDeviceWithoutQuorum
9999
annotations:
100100
description: |
101101
DRBD device "{{ $labels.name }}" on "{{ $labels.node }}" has no quorum.
102102
This usually indicates connectivity issues.
103103
expr: drbd_device_quorum == 0
104104
labels:
105-
severity: warn
105+
severity: warning
106106
- alert: drbdResourceSuspended
107107
annotations:
108108
description: |
109109
DRBD resource "{{ $labels.name }}" on "{{ $labels.node }}" has been suspended for 1m.
110110
for: 1m
111111
expr: drbd_resource_suspended > 0
112112
labels:
113-
severity: warn
113+
severity: warning
114114
- alert: drbdResourceResyncWithoutProgress
115115
annotations:
116116
description: |
117117
DRBD resource "{{ $labels.name }}" on "{{ $labels.node }}" has been in Inconsistent without resync progress for 5 minutes.
118118
This may indicate there is no connection to UpToDate data, or a stuck resync.
119119
expr: drbd_device_state{drbd_device_state="Inconsistent"} and delta(drbd_peerdevice_outofsync_bytes[5m]) >= 0
120120
labels:
121-
severity: warn
121+
severity: warning
122122
- alert: drbdResourceWithNoUpToDateReplicas
123123
annotations:
124124
description: |

0 commit comments

Comments
 (0)