Repository navigation
Commit 185c5ec
authored
fix(monitoring): route InfoInhibitor to the blackhole receiver (#4948)
## What this PR does
`InfoInhibitor` in
`packages/system/monitoring-agents/alerts/general.rules.yaml` is the
kube-prometheus rule that exists only to feed an Alertmanager inhibit
rule, and its own description says it should go to a null receiver. The
`update` target that first imported the rules rewrote its `severity:
none` to `major`, and the platform Alertmanager config has no inhibit
rule, so the alert goes to Alerta one level below critical. Alerta's
Telegram and Slack plugins forward it with the default
`disabledSeverity: []`, and once email delivery is configured, a
`notify_email="true"` label inherited from the info alert gets it
emailed as well.
It fires whenever an alert with `severity="info"` is active and no
warning or critical alert fires in the same namespace and cluster. #3799
moved the last first-party rule off `info`, but rules the platform does
not write still use it. CDI v1.66.1 generates
`CDIStorageProfilesIncomplete`, KubeVirt v1.9.0 generates
`KubeVirtVMGuestMemoryAvailableLow` and
`KubeVirtDeprecatedAPIRequested`, all at `info`, and operators add their
own. We hit it on a v1.6.4 cluster with one custom `severity: info` rule
that is always firing, and Alerta showed a permanent MAJOR
`InfoInhibitor`. The info alert itself cannot show up there, because
Alerta's default alarm model has no `info`.
This puts the label back to `none`, the value kube-prometheus ships. The
platform Alertmanager config already routes `severity="none"` to the
`blackhole` receiver ahead of every route that delivers, and the
Monitoring parameters already document that such alerts never reach
email. `hack/alert-severity-contract.bats` refuses `none` on purpose,
because an alert carrying it is never delivered, so InfoInhibitor is
listed as the one alert for which that is the point. The exemption
covers `none` and that name only.
The contract now also checks that InfoInhibitor carries `none`, that
`none` still fails on any other alert, and that the exemption does not
let InfoInhibitor through with a severity Alerta refuses. A new
helm-unittest suite in `packages/system/monitoring` pins the route
order, so that only the heartbeat route can come ahead of the blackhole,
with and without the email routes.
There is no inhibit rule here, and the expression still matches `info`
only. Wiring the inhibition up would hide every informational alert from
Alerta unless a warning fires in the same namespace, which changes
behaviour on every cluster and is a separate decision. Dropping
InfoInhibitor altogether would be equivalent for delivery, and I can
rework the PR that way if you prefer it.
v1.6.5 ships the same rule and the same routes, so a backport to
`release-1.6` would be appreciated. The commit cherry-picks onto it
without conflicts.
### Validation
- `bats hack/alert-severity-contract.bats` passes 12 of 12, and so does
the same file under `hack/cozytest.sh`. Without the label change the new
InfoInhibitor test fails on `[ "$delivered" -eq 0 ]`. With the exemption
removed the tree-wide contract fails on
`packages/system/monitoring-agents/alerts/general.rules.yaml:
InfoInhibitor -> 'none'`. With the exemption widened to any alert, or to
any severity, the matching fixture test fails.
- `helm unittest` in `packages/system/monitoring` passes 25 suites and
306 tests. Deleting the `severity="none"` route fails both new cases,
and an email route placed ahead of it fails the email case.
- `helm unittest` in `packages/system/monitoring-agents` passes 6 suites
and 19 tests, and the rendered `alerts-general.rules` VMRule carries
`severity: none` for InfoInhibitor.
- Rendered the Alertmanager config and ran `amtool config routes test`
from Alertmanager v0.31.0, the default of VM operator v0.68.4, with
InfoInhibitor's labels. `severity=major` routes to `alerta`, and to
`alerta,email` once email is configured and `notify_email=true` is
inherited. `severity=none` routes to `blackhole` in every case. `amtool
check-config` reports 0 inhibit rules.
- On `release-1.6` with this commit cherry-picked, the contract and both
helm-unittest suites pass.
- Not exercised on a live cluster.
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
Walked the trigger map against the diff. It changes one alert label
under `packages/system/monitoring-agents/alerts/`, extends the test in
`hack/alert-severity-contract.bats` and adds a test suite under
`packages/system/monitoring/tests/`. No package is added or renamed, and
no values, schema, CRD, platform values, variant, make target, tool the
satellites call, node prerequisite or metric changes. The website
already says that `severity="none"` alerts are blackholed and never
reach email, and no page mentions InfoInhibitor, so nothing there goes
stale.
### Release note
```release-note
fix(monitoring): InfoInhibitor no longer reaches Alerta as a major alert. It carries severity none again, which the platform Alertmanager routes to the blackhole, as upstream kube-prometheus intends. Before, any active alert with severity info, from a custom rule or from one CDI or KubeVirt generates, raised a major InfoInhibitor in Alerta and in its Telegram and Slack notifications.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Bug Fixes**
* `InfoInhibitor` alerts are now treated as non-notifying, preventing
them from generating alert notifications. Other alerts retain their
existing severity handling, and heartbeat alerts continue to follow
their configured route.
* The non-notifying route takes precedence over email delivery rules
that include `InfoInhibitor`.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->3 files changed
Lines changed: 141 additions & 1 deletion
File tree
- hack
- packages/system
- monitoring-agents/alerts
- monitoring/tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
12 | 16 | | |
13 | 17 | | |
14 | 18 | | |
| |||
104 | 108 | | |
105 | 109 | | |
106 | 110 | | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
107 | 119 | | |
108 | 120 | | |
109 | 121 | | |
| |||
128 | 140 | | |
129 | 141 | | |
130 | 142 | | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
131 | 220 | | |
132 | 221 | | |
133 | 222 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
56 | | - | |
| 56 | + | |
57 | 57 | | |
58 | 58 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
0 commit comments