Skip to content

Commit 08c535c

Browse files
authored
fix(monitoring): give kube-apiserver its own scrape size limit (#4427)
## What this PR does kube-apiserver's `/metrics` grows with the number of resources and CRDs, and on a full install it outgrows the 32MB `promscrape.maxScrapeSize` that monitoring-agents sets globally. vmagent then refuses every scrape of the API server, `up` drops to 0, and `KubeAPIDown` and `TargetDown` fire against a healthy control plane while every rule over API server metrics goes blind. The apiserver `VMServiceScrape` endpoint now sets its own `max_scrape_size` (128MB by default, `vmagent.apiserverMaxScrapeSize`). vmagent uses a per-job value in place of the global flag for that job only, so the 32MB guard for every other target is unchanged. The value stays finite on purpose: vmagent reads the whole uncompressed response into memory before parsing, `streamParse` included, so this limit is also a memory ceiling. A new `VMAgentScrapeSizeExceeded` warning fires on `vm_promscrape_max_scrape_size_exceeded_errors_total`, so an oversized target is reported as a size problem instead of only as a target that is down. The counter has no target label, so the description points at the vmagent targets page to find it. Upgrade note: the per-job value replaces the global flag in both directions. An install that raised `promscrape.maxScrapeSize` above 128MB through `vmagent.extraArgs` should raise `vmagent.apiserverMaxScrapeSize` as well. Verified on a dev cluster with the same operator (v0.68.4) and vmagent (v1.136.0): a test vmagent with a 1MB global limit scraped all three API servers with `max_scrape_size: 64MB` on the endpoint and refused all three without it. The new counter reached storage with the `namespace` and `pod` labels the alert uses, and the alert expression returned a series. The `VMRule` passes the operator's admission webhook, which rejects a malformed expression. Guest clusters get the same default, since their HelmRelease overrides other `vmagent` keys only. Fixes #4343 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(monitoring): kube-apiserver is scraped with its own 128MB size limit (`vmagent.apiserverMaxScrapeSize` in the monitoring-agents component) instead of the global 32MB, so a large `/metrics` no longer makes KubeAPIDown fire on a healthy API server. A new VMAgentScrapeSizeExceeded alert reports any target whose response exceeds the scrape size limit. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Kubernetes API server metrics can now use a configurable scrape-size limit, set to 128 MB by default. Leaving the setting empty uses the global limit. * **Bug Fixes** * Added a warning alert when vmagent reports scrape-size-limit errors. The alert identifies the affected vmagent pod and service and provides guidance for finding the target and adjusting its limit. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 15035a9 + 380c446 commit 08c535c

4 files changed

Lines changed: 105 additions & 0 deletions

File tree

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
apiVersion: operator.victoriametrics.com/v1beta1
2+
kind: VMRule
3+
metadata:
4+
name: alerts-vmagent
5+
spec:
6+
groups:
7+
- name: vmagent
8+
params: {}
9+
rules:
10+
- alert: VMAgentScrapeSizeExceeded
11+
expr: increase(vm_promscrape_max_scrape_size_exceeded_errors_total[10m]) > 0
12+
for: 5m
13+
labels:
14+
severity: warning
15+
exported_instance: '{{ $labels.namespace }}/{{ $labels.pod }}'
16+
service: vmagent
17+
annotations:
18+
summary: vmagent refuses scrape responses larger than its size limit.
19+
description: >-
20+
vmagent {{ $labels.namespace }}/{{ $labels.pod }} in cluster {{ $labels.cluster }} dropped
21+
scrape responses that exceed -promscrape.maxScrapeSize or the scrape's max_scrape_size.
22+
The affected targets report up == 0, so TargetDown or KubeAPIDown may fire for a healthy
23+
target. The counter carries no target label: find the target on the vmagent /targets
24+
page, where the last error names the size limit, then raise max_scrape_size on that
25+
scrape. For kube-apiserver on the management cluster this is the
26+
vmagent.apiserverMaxScrapeSize value of the monitoring-agents component.

‎packages/system/monitoring-agents/templates/apiserver-scrape.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,9 @@ spec:
99
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
1010
port: https
1111
scheme: https
12+
{{- with .Values.vmagent.apiserverMaxScrapeSize }}
13+
max_scrape_size: {{ . | quote }}
14+
{{- end }}
1215
relabelConfigs:
1316
- action: labeldrop
1417
regex: (endpoint|namespace|pod|container)
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
suite: kube-apiserver scrape size limit and its alert
2+
3+
# kube-apiserver's /metrics grows with the number of resources and CRDs, and a
4+
# full install pushes it past the global 32MB -promscrape.maxScrapeSize. vmagent
5+
# then refuses every scrape of it, so up == 0 and KubeAPIDown fires on a
6+
# healthy API server. The apiserver endpoint carries its own max_scrape_size,
7+
# which replaces the global flag for that job only (upward too), so the guard
8+
# for every other target stays at 32MB. vmagent buffers the whole uncompressed
9+
# body even with streamParse, so the per-job limit is a memory ceiling and must
10+
# stay finite.
11+
release:
12+
name: monitoring-agents
13+
namespace: cozy-monitoring
14+
tests:
15+
- it: apiserver endpoint gets its own scrape size limit by default
16+
template: templates/apiserver-scrape.yaml
17+
asserts:
18+
- equal:
19+
path: spec.endpoints[0].max_scrape_size
20+
value: 128MB
21+
22+
- it: apiserver scrape size limit follows the value
23+
template: templates/apiserver-scrape.yaml
24+
set:
25+
vmagent.apiserverMaxScrapeSize: 512MB
26+
asserts:
27+
- equal:
28+
path: spec.endpoints[0].max_scrape_size
29+
value: 512MB
30+
31+
- it: a byte count renders as a string, as the CRD field requires
32+
template: templates/apiserver-scrape.yaml
33+
set:
34+
vmagent.apiserverMaxScrapeSize: 134217728
35+
asserts:
36+
- matchRegex:
37+
path: spec.endpoints[0].max_scrape_size
38+
pattern: ^[0-9.e+]+$
39+
40+
- it: an empty value falls back to the global flag
41+
template: templates/apiserver-scrape.yaml
42+
set:
43+
vmagent.apiserverMaxScrapeSize: ""
44+
asserts:
45+
- notExists:
46+
path: spec.endpoints[0].max_scrape_size
47+
48+
- it: the global limit for every other target is not loosened
49+
template: templates/vmagent.yaml
50+
asserts:
51+
- equal:
52+
path: spec.extraArgs["promscrape.maxScrapeSize"]
53+
value: 32MB
54+
55+
- it: a refused oversized scrape raises its own alert
56+
template: templates/alerts.yaml
57+
documentSelector:
58+
path: metadata.name
59+
value: alerts-vmagent
60+
asserts:
61+
- equal:
62+
path: spec.groups[0].rules[0].alert
63+
value: VMAgentScrapeSizeExceeded
64+
- equal:
65+
path: spec.groups[0].rules[0].expr
66+
value: increase(vm_promscrape_max_scrape_size_exceeded_errors_total[10m]) > 0
67+
- equal:
68+
path: spec.groups[0].rules[0].labels.severity
69+
value: warning

‎packages/system/monitoring-agents/values.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -283,6 +283,13 @@ vmagent:
283283
- http://vminsert-shortterm.{{ .Values.global.target }}.svc:8480/insert/0/prometheus
284284
- http://vminsert-longterm.{{ .Values.global.target }}.svc:8480/insert/0/prometheus
285285
extraArgs: {}
286+
# Replaces the global promscrape.maxScrapeSize (32MB) for the kube-apiserver
287+
# job only. Its /metrics grows with the number of CRDs and outgrows 32MB on a
288+
# full install. vmagent buffers the whole uncompressed response even with
289+
# streamParse, so keep this finite. It replaces the global in both
290+
# directions: if extraArgs raises promscrape.maxScrapeSize above this value,
291+
# raise this one too. An empty value falls back to the global.
292+
apiserverMaxScrapeSize: 128MB
286293

287294
fluent-bit:
288295
# 14d peak 319Mi; scales with log volume, so the limit carries wide headroom.

0 commit comments

Comments
 (0)