Repository navigation
Commit 08c535c
authored
fix(monitoring): give kube-apiserver its own scrape size limit (#4427)
## What this PR does
kube-apiserver's `/metrics` grows with the number of resources and CRDs,
and on a full install it outgrows the 32MB `promscrape.maxScrapeSize`
that monitoring-agents sets globally. vmagent then refuses every scrape
of the API server, `up` drops to 0, and `KubeAPIDown` and `TargetDown`
fire against a healthy control plane while every rule over API server
metrics goes blind.
The apiserver `VMServiceScrape` endpoint now sets its own
`max_scrape_size` (128MB by default, `vmagent.apiserverMaxScrapeSize`).
vmagent uses a per-job value in place of the global flag for that job
only, so the 32MB guard for every other target is unchanged. The value
stays finite on purpose: vmagent reads the whole uncompressed response
into memory before parsing, `streamParse` included, so this limit is
also a memory ceiling.
A new `VMAgentScrapeSizeExceeded` warning fires on
`vm_promscrape_max_scrape_size_exceeded_errors_total`, so an oversized
target is reported as a size problem instead of only as a target that is
down. The counter has no target label, so the description points at the
vmagent targets page to find it.
Upgrade note: the per-job value replaces the global flag in both
directions. An install that raised `promscrape.maxScrapeSize` above
128MB through `vmagent.extraArgs` should raise
`vmagent.apiserverMaxScrapeSize` as well.
Verified on a dev cluster with the same operator (v0.68.4) and vmagent
(v1.136.0): a test vmagent with a 1MB global limit scraped all three API
servers with `max_scrape_size: 64MB` on the endpoint and refused all
three without it. The new counter reached storage with the `namespace`
and `pod` labels the alert uses, and the alert expression returned a
series. The `VMRule` passes the operator's admission webhook, which
rejects a malformed expression. Guest clusters get the same default,
since their HelmRelease overrides other `vmagent` keys only.
Fixes #4343
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(monitoring): kube-apiserver is scraped with its own 128MB size limit (`vmagent.apiserverMaxScrapeSize` in the monitoring-agents component) instead of the global 32MB, so a large `/metrics` no longer makes KubeAPIDown fire on a healthy API server. A new VMAgentScrapeSizeExceeded alert reports any target whose response exceeds the scrape size limit.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Kubernetes API server metrics can now use a configurable scrape-size
limit, set to 128 MB by default. Leaving the setting empty uses the
global limit.
* **Bug Fixes**
* Added a warning alert when vmagent reports scrape-size-limit errors.
The alert identifies the affected vmagent pod and service and provides
guidance for finding the target and adjusting its limit.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->4 files changed
Lines changed: 105 additions & 0 deletions
File tree
- packages/system/monitoring-agents
- alerts
- templates
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
12 | 15 | | |
13 | 16 | | |
14 | 17 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
283 | 283 | | |
284 | 284 | | |
285 | 285 | | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
286 | 293 | | |
287 | 294 | | |
288 | 295 | | |
| |||
0 commit comments