Skip to content

Commit 185c5ec

Browse files
authored
fix(monitoring): route InfoInhibitor to the blackhole receiver (#4948)
## What this PR does `InfoInhibitor` in `packages/system/monitoring-agents/alerts/general.rules.yaml` is the kube-prometheus rule that exists only to feed an Alertmanager inhibit rule, and its own description says it should go to a null receiver. The `update` target that first imported the rules rewrote its `severity: none` to `major`, and the platform Alertmanager config has no inhibit rule, so the alert goes to Alerta one level below critical. Alerta's Telegram and Slack plugins forward it with the default `disabledSeverity: []`, and once email delivery is configured, a `notify_email="true"` label inherited from the info alert gets it emailed as well. It fires whenever an alert with `severity="info"` is active and no warning or critical alert fires in the same namespace and cluster. #3799 moved the last first-party rule off `info`, but rules the platform does not write still use it. CDI v1.66.1 generates `CDIStorageProfilesIncomplete`, KubeVirt v1.9.0 generates `KubeVirtVMGuestMemoryAvailableLow` and `KubeVirtDeprecatedAPIRequested`, all at `info`, and operators add their own. We hit it on a v1.6.4 cluster with one custom `severity: info` rule that is always firing, and Alerta showed a permanent MAJOR `InfoInhibitor`. The info alert itself cannot show up there, because Alerta's default alarm model has no `info`. This puts the label back to `none`, the value kube-prometheus ships. The platform Alertmanager config already routes `severity="none"` to the `blackhole` receiver ahead of every route that delivers, and the Monitoring parameters already document that such alerts never reach email. `hack/alert-severity-contract.bats` refuses `none` on purpose, because an alert carrying it is never delivered, so InfoInhibitor is listed as the one alert for which that is the point. The exemption covers `none` and that name only. The contract now also checks that InfoInhibitor carries `none`, that `none` still fails on any other alert, and that the exemption does not let InfoInhibitor through with a severity Alerta refuses. A new helm-unittest suite in `packages/system/monitoring` pins the route order, so that only the heartbeat route can come ahead of the blackhole, with and without the email routes. There is no inhibit rule here, and the expression still matches `info` only. Wiring the inhibition up would hide every informational alert from Alerta unless a warning fires in the same namespace, which changes behaviour on every cluster and is a separate decision. Dropping InfoInhibitor altogether would be equivalent for delivery, and I can rework the PR that way if you prefer it. v1.6.5 ships the same rule and the same routes, so a backport to `release-1.6` would be appreciated. The commit cherry-picks onto it without conflicts. ### Validation - `bats hack/alert-severity-contract.bats` passes 12 of 12, and so does the same file under `hack/cozytest.sh`. Without the label change the new InfoInhibitor test fails on `[ "$delivered" -eq 0 ]`. With the exemption removed the tree-wide contract fails on `packages/system/monitoring-agents/alerts/general.rules.yaml: InfoInhibitor -> 'none'`. With the exemption widened to any alert, or to any severity, the matching fixture test fails. - `helm unittest` in `packages/system/monitoring` passes 25 suites and 306 tests. Deleting the `severity="none"` route fails both new cases, and an email route placed ahead of it fails the email case. - `helm unittest` in `packages/system/monitoring-agents` passes 6 suites and 19 tests, and the rendered `alerts-general.rules` VMRule carries `severity: none` for InfoInhibitor. - Rendered the Alertmanager config and ran `amtool config routes test` from Alertmanager v0.31.0, the default of VM operator v0.68.4, with InfoInhibitor's labels. `severity=major` routes to `alerta`, and to `alerta,email` once email is configured and `notify_email=true` is inherited. `severity=none` routes to `blackhole` in every case. `amtool check-config` reports 0 inhibit rules. - On `release-1.6` with this commit cherry-picked, the contract and both helm-unittest suites pass. - Not exercised on a live cluster. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Walked the trigger map against the diff. It changes one alert label under `packages/system/monitoring-agents/alerts/`, extends the test in `hack/alert-severity-contract.bats` and adds a test suite under `packages/system/monitoring/tests/`. No package is added or renamed, and no values, schema, CRD, platform values, variant, make target, tool the satellites call, node prerequisite or metric changes. The website already says that `severity="none"` alerts are blackholed and never reach email, and no page mentions InfoInhibitor, so nothing there goes stale. ### Release note ```release-note fix(monitoring): InfoInhibitor no longer reaches Alerta as a major alert. It carries severity none again, which the platform Alertmanager routes to the blackhole, as upstream kube-prometheus intends. Before, any active alert with severity info, from a custom rule or from one CDI or KubeVirt generates, raised a major InfoInhibitor in Alerta and in its Telegram and Slack notifications. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * `InfoInhibitor` alerts are now treated as non-notifying, preventing them from generating alert notifications. Other alerts retain their existing severity handling, and heartbeat alerts continue to follow their configured route. * The non-notifying route takes precedence over email delivery rules that include `InfoInhibitor`. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents dd6ff66 + e847596 commit 185c5ec

3 files changed

Lines changed: 141 additions & 1 deletion

File tree

‎hack/alert-severity-contract.bats‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,10 @@ load test_helper
99
# scope: they are re-vendored by `make update`, so a bad severity there is
1010
# fixed upstream or in patches/, not by editing the label.
1111
ALERT_SEVERITY_ALLOWED=" security critical major minor warning indeterminate informational normal ok cleared debug trace unknown "
12+
# The alerts that exist only to be dropped, the one place `none` is right.
13+
# InfoInhibitor is the source side of an Alertmanager inhibit rule, which the
14+
# platform config does not have, so delivering it at any severity is noise.
15+
ALERT_SEVERITY_BLACKHOLED=" InfoInhibitor "
1216

1317
helm_actions_stripped() {
1418
awk '
@@ -104,6 +108,14 @@ alert_severity_contract() {
104108
break
105109
fi
106110
done
111+
if [ "$severity" = "none" ]; then
112+
for blackholed in $ALERT_SEVERITY_BLACKHOLED; do
113+
if [ "$blackholed" = "$alert" ]; then
114+
accepted=1
115+
break
116+
fi
117+
done
118+
fi
107119
if [ "$accepted" -eq 0 ]; then
108120
echo "$file: $alert -> '$severity'" >&2
109121
bad=$((bad + 1))
@@ -128,6 +140,83 @@ alert_severity_contract() {
128140
alert_severity_contract
129141
}
130142

143+
@test "InfoInhibitor is labelled for the blackhole route" {
144+
rows=$(mktemp) || return 1
145+
if ! alert_severity_rows > "$rows"; then
146+
rm -f "$rows"
147+
return 1
148+
fi
149+
tab=$(printf '\t')
150+
found=$(grep -c "${tab}InfoInhibitor${tab}" "$rows" || true)
151+
delivered=$(grep "${tab}InfoInhibitor${tab}" "$rows" | grep -vc "${tab}none\$" || true)
152+
rm -f "$rows"
153+
[ "$found" -gt 0 ]
154+
[ "$delivered" -eq 0 ]
155+
}
156+
157+
@test "severity none passes only on an alert that exists to be dropped" {
158+
fixture=$(mktemp -d)
159+
mkdir -p "$fixture/packages/system/example/alerts"
160+
printf '%s\n' \
161+
'apiVersion: operator.victoriametrics.com/v1beta1' \
162+
'kind: VMRule' \
163+
'spec:' \
164+
' groups:' \
165+
' - name: general' \
166+
' rules:' \
167+
' - alert: InfoInhibitor' \
168+
' labels:' \
169+
' severity: none' \
170+
> "$fixture/packages/system/example/alerts/00-inhibitor.yaml"
171+
git -C "$fixture" init -q
172+
git -C "$fixture" add packages/system/example/alerts/00-inhibitor.yaml
173+
174+
inhibitor_succeeded=0
175+
ALERT_SEVERITY_ROOT="$fixture" alert_severity_contract >/dev/null 2>&1 || inhibitor_succeeded=$?
176+
177+
printf '%s\n' \
178+
'apiVersion: operator.victoriametrics.com/v1beta1' \
179+
'kind: VMRule' \
180+
'spec:' \
181+
' groups:' \
182+
' - name: silent' \
183+
' rules:' \
184+
' - alert: SilentAlert' \
185+
' labels:' \
186+
' severity: none' \
187+
> "$fixture/packages/system/example/alerts/01-silent.yaml"
188+
git -C "$fixture" add packages/system/example/alerts/01-silent.yaml
189+
190+
silent_succeeded=0
191+
ALERT_SEVERITY_ROOT="$fixture" alert_severity_contract >/dev/null 2>&1 || silent_succeeded=$?
192+
rm -rf "$fixture"
193+
[ "$inhibitor_succeeded" -eq 0 ]
194+
[ "$silent_succeeded" -ne 0 ]
195+
}
196+
197+
@test "an alert that exists to be dropped still fails on a refused severity" {
198+
fixture=$(mktemp -d)
199+
mkdir -p "$fixture/packages/system/example/alerts"
200+
printf '%s\n' \
201+
'apiVersion: operator.victoriametrics.com/v1beta1' \
202+
'kind: VMRule' \
203+
'spec:' \
204+
' groups:' \
205+
' - name: general' \
206+
' rules:' \
207+
' - alert: InfoInhibitor' \
208+
' labels:' \
209+
' severity: info' \
210+
> "$fixture/packages/system/example/alerts/00-inhibitor.yaml"
211+
git -C "$fixture" init -q
212+
git -C "$fixture" add packages/system/example/alerts/00-inhibitor.yaml
213+
214+
contract_succeeded=0
215+
ALERT_SEVERITY_ROOT="$fixture" alert_severity_contract >/dev/null 2>&1 || contract_succeeded=$?
216+
rm -rf "$fixture"
217+
[ "$contract_succeeded" -ne 0 ]
218+
}
219+
131220
@test "a parser failure cannot leave the contract green" {
132221
fixture=$(mktemp -d)
133222
mkdir -p "$fixture/packages/system/example/alerts"

‎packages/system/monitoring-agents/alerts/general.rules.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,6 @@ spec:
5353
!= "InfoInhibitor", severity =~ "warning|critical", alertstate="firing"} ==
5454
1
5555
labels:
56-
severity: major
56+
severity: none
5757
exported_instance: global
5858
service: general.rules
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
suite: Alertmanager blackhole route
2+
templates:
3+
- templates/alerta/alerta.yaml
4+
5+
release:
6+
name: monitoring
7+
namespace: tenant-root
8+
9+
set:
10+
_cluster:
11+
root-host: example.org
12+
_namespace:
13+
host: tenant-root.example.org
14+
ingress: tenant-root
15+
16+
# Rules labelled severity="none", InfoInhibitor among them, exist to be
17+
# dropped, and Alerta answers that severity with a 500. Only the heartbeat
18+
# route may come ahead of the blackhole, so no delivering route claims them.
19+
tests:
20+
- it: drops severity none ahead of every delivering route
21+
asserts:
22+
- documentSelector:
23+
path: metadata.labels.app
24+
value: alertmanager
25+
matchRegex:
26+
path: stringData["alertmanager.yaml"]
27+
pattern: '\n routes:\n - receiver: alerta_no_resolve\n repeat_interval: 1m\n matchers:\n - .event="Heartbeat".\n - receiver: blackhole\n matchers:\n - .severity="none".\n'
28+
29+
- it: keeps severity none out of email even when its alert name is listed
30+
set:
31+
alerta:
32+
alerts:
33+
email:
34+
35+
36+
smarthost: "smtp.example.org:465"
37+
alertnames:
38+
- InfoInhibitor
39+
asserts:
40+
- documentSelector:
41+
path: metadata.labels.app
42+
value: alertmanager
43+
matchRegex:
44+
path: stringData["alertmanager.yaml"]
45+
pattern: '\n routes:\n - receiver: alerta_no_resolve\n repeat_interval: 1m\n matchers:\n - .event="Heartbeat".\n - receiver: blackhole\n matchers:\n - .severity="none".\n'
46+
- documentSelector:
47+
path: metadata.labels.app
48+
value: alertmanager
49+
matchRegex:
50+
path: stringData["alertmanager.yaml"]
51+
pattern: '(?s)\n - receiver: blackhole\n matchers:\n - .severity="none".\n.*\n - receiver: email\n group_wait: 1m\n group_interval: 5m\n repeat_interval: .12h.\n matchers:\n - .alertname=~"\^\(InfoInhibitor\)\$".\n'

0 commit comments

Comments
 (0)