Skip to content

Commit c069a19

Browse files
lexfreilexfrei
authored
fix(monitoring-agents): give kube-state-metrics a sized memory limit (#3846)
`kube-state-metrics` ran BestEffort with no resources at all: the subchart ships `resources: {}` and our wrapper never set any. So it put no `memory.max` on its pod cgroup, and the Talos userspace OOM handler only ever selects cgroups that lack one, which left it permanently in the victim set. Any tenant workload driving node memory pressure could evict the source of every kube-state metric we alert on. Background is in `docs/operations/system-memory-limits.md`. ## The numbers are measured On a three node cluster with 322 pods and roughly 25.6k series, which is the smallest we run: 14 day peak working set 307Mi, peak RSS 280Mi, mean 176Mi. So the request sits above the mean at 192Mi rather than at the peak, and the limit carries wide headroom at 2Gi. The headroom is deliberate: this component's memory scales with the number of objects in the cluster, and the `--metric-labels-allowlist=pods=[*],deployments=[*]` already in this file widens every pod and deployment series further, so a cluster larger than the one measured lands well above what was seen here. `2Gi` is 6.7x that peak, which read as linear in object count is roughly 2100 pods and 170k series, so the limit stays a ceiling rather than a bound well past the largest cluster this sizing was drawn from. No CPU limit, following the node DaemonSet sizing in `c9ba7a3cc`, which set one zero times across nineteen files. CPU is irrelevant to the OOM ranking, and a ceiling that binds on a scraper turns into scrape timeouts rather than backpressure. ## This supersedes #2632 #2632 proposes 256Mi, which is below the 307Mi peak measured on the smallest cluster we have, so it would convert an eviction risk into a reliable OOM kill. It also adds a `cpu: 200m` limit, which is the half that buys nothing. The diagnosis there was right and it sat since May; the numbers are what needed replacing. ## Checks `packages/system/monitoring-agents/tests/kube_state_metrics_resources_test.yaml` pins the limit, both requests, and the absence of a CPU limit. Mutation checked: dropping the limit to 256Mi turns it red. The chart's suites are 3 files and 6 tests, all green. ### Release note ```release-note fix(monitoring-agents): kube-state-metrics now carries a memory request and limit sized from a measured peak, so it is no longer a permanent candidate for the node OOM handler ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Configured kube-state-metrics with requests of 50m CPU and 192Mi memory, plus a 2Gi memory limit. No CPU limit is configured, allowing CPU usage to scale as needed. These settings help ensure the service has the memory and CPU resources it needs while keeping memory use bounded. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 52aa837 + 361311c commit c069a19

2 files changed

Lines changed: 50 additions & 0 deletions

File tree

‎packages/system/monitoring-agents/tests/kube_state_metrics_resources_test.yaml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
suite: kube-state-metrics carries a memory limit sized above its measured peak
2+
# Without a memory limit the container sets no memory.max on its pod cgroup, and
3+
# the Talos userspace OOM handler only ever selects cgroups that lack one, so it
4+
# is an eviction candidate on every node regardless of what caused the pressure.
5+
# The numbers come from a measured 14d peak of 307Mi on the smallest cluster we
6+
# run; the limit carries headroom because this component scales with object
7+
# count. A CPU limit is deliberately absent: it does not affect the OOM ranking
8+
# and a binding ceiling turns into scrape timeouts.
9+
templates:
10+
- charts/kube-state-metrics/templates/deployment.yaml
11+
tests:
12+
- it: sets a memory limit above the measured peak and a request above the mean
13+
asserts:
14+
- equal:
15+
path: spec.template.spec.containers[0].resources.limits.memory
16+
value: 2Gi
17+
- equal:
18+
path: spec.template.spec.containers[0].resources.requests.memory
19+
value: 192Mi
20+
- equal:
21+
path: spec.template.spec.containers[0].resources.requests.cpu
22+
value: 50m
23+
- it: sets no CPU limit
24+
asserts:
25+
- notExists:
26+
path: spec.template.spec.containers[0].resources.limits.cpu

‎packages/system/monitoring-agents/values.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,30 @@ global:
22
target: cozy-monitoring
33

44
kube-state-metrics:
5+
# Sized from a measured 14d peak of 307Mi working set on a 3-node cluster with
6+
# 322 pods and roughly 25.6k series, which is the smallest cluster we run. The
7+
# request sits above the 176Mi mean rather than at the peak, and the limit
8+
# carries wide headroom on purpose: this component's memory scales with the
9+
# number of objects in the cluster, and --metric-labels-allowlist below widens
10+
# every pod and deployment series further, so a cluster larger than the one
11+
# measured will land well above the peak seen here.
12+
# 2Gi is 6.7x that peak, so scaling linearly with object count it stays a
13+
# ceiling rather than a bound up to roughly 2100 pods and 170k series.
14+
#
15+
# A memory limit is what sets memory.max on the pod cgroup, and the Talos
16+
# userspace OOM handler only ever selects cgroups that lack one, so running
17+
# this without a limit leaves it an eviction candidate on every node. See
18+
# docs/operations/system-memory-limits.md.
19+
#
20+
# No CPU limit: it is irrelevant to that OOM ranking, and a ceiling that binds
21+
# only buys CFS throttling, which for a metrics scraper turns into scrape
22+
# timeouts rather than backpressure.
23+
resources:
24+
requests:
25+
cpu: 50m
26+
memory: 192Mi
27+
limits:
28+
memory: 2Gi
529
extraArgs:
630
- --metric-labels-allowlist=pods=[*],deployments=[*]
731
rbac:

0 commit comments

Comments
 (0)