Skip to content

Commit 1c00ddb

Browse files
authored
fix(platform): give node DaemonSets memory requests and limits (#3495)
## What this PR does Sets memory requests and limits on the node DaemonSets that had none. Companion to the namespace-wide LimitRange; the reasoning for why a limit specifically (and not a request, and not a `PriorityClass`) is in that PR and in `docs/operations/system-memory-limits.md`. Requests come from 14-day peak working set measured on a 3-node dev cluster. Limits are deliberately loose, sized as a runaway backstop rather than a fitted ceiling, for two reasons: several of these scale with cluster size in ways a small dev cluster understates (cilium with endpoint and policy count, fluent-bit with log volume, multus and kube-ovn with pods per node, velero node-agent during a backup), and undersizing converts a rare pressure-driven kill into a deterministic one. This repository already carries a scar from that failure mode, recorded in the `vertical-pod-autoscaler` values where a 110Mi limit OOM-killed the updater and cascaded into failed HelmReleases. Two DaemonSets are not chart-rendered and needed their own seam. `linstor-satellite` goes through the `LinstorSatelliteConfiguration` podTemplate, including the two plunger sidecars — every container in the pod needs a limit for the pod cgroup to get `memory.max`, so one limit-free sidecar would put the whole pod back in the victim set. `virt-handler` goes through KubeVirt `customizeComponents.patches`, because virt-operator owns that DaemonSet and chart values cannot reach it. `linstor-controller` is deliberately left limit-free. It is a JVM that must not be OOM-killed mid-operation, and `cluster_test.yaml` pins that as an invariant. The satellite is a Go binary with a bounded working set, so the opposite call applies there. Not covered, because upstream hardcodes the containers with no `resources` knob: the four frr-k8s `cp-*` init containers, the kube-ovn `hostpath-init` and `install-cni` init containers, and `cozy-proxy`, whose vendored chart exposes no resources value at all. `hami` and `kilo` are left alone as optional components with no usage measurements behind them. All of these still receive a limit from the namespace LimitRange, so they are not left exposed — they simply carry no request. Worth stating plainly: this PR does **not** grant OOM immunity on its own, because the init containers it cannot reach leave the pod cgroup without `memory.max`. That only closes with the LimitRange PR in place. ### Verification Every touched package renders and its helm-unittest suite passes. New assertions cover the metallb speaker and all five frr-k8s containers, the linstor satellite and plunger sidecars, and the KubeVirt patch. The metallb suite was mutation-tested: removing the frr-k8s reloader limit fails it. One gap. The KubeVirt patch shape was validated against the live CRD and pushed through the real update path with `kubectl patch --dry-run=server` (accepted; live object confirmed unchanged), but virt-operator actually reconciling the DaemonSet with those resources has **not** been observed, because that means restarting virt-handler fleet-wide. Worth confirming in e2e or on a dedicated stand before merge. ### Downstream repositories Walked the trigger map against the diff. No entry is triggered: this changes chart values and cozystack-owned templates under `packages/system/`, not the `cozy-proxy` label and annotation contract, not node prerequisites, not an app schema or version enum. Leaving the boxes empty rather than ticking "none affected", because the diff surfaced a real gap in a downstream repository even though nothing here forces a change there: the vendored `cozy-proxy` chart exposes no `resources` value, so its DaemonSet cannot be given one from this side at all. That wants an issue or PR in cozystack/cozy-proxy followed by a re-vendor. Flagging it for a maintainer decision rather than filing speculatively. ### Release note ```release-note fix(platform): node DaemonSets (cilium, metallb speaker and frr-k8s, kube-ovn, linstor-satellite, virt-handler, fluent-bit, node-exporter, multus, velero node-agent, kubevirt-csi-node) now declare memory requests and limits, so they are no longer selected as victims by the Talos userspace OOM handler when an unrelated workload drives node memory pressure. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Resource Management** - Added CPU and memory requests and limits across Cilium, KubeVirt CSI, KubeVirt, Linstor, MetalLB, monitoring agents, Multus, and Velero workloads. - Improved workload stability with defined memory ceilings and resource reservations. - **Multus** - Moved resources to the dedicated `cozy-multus` namespace and configured Cilium as the master CNI. - Added optional CNI plugin staging with enhanced security settings. - **Tests** - Added and expanded Helm tests validating resource configurations and Multus behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents b7b5653 + c9ba7a3 commit 1c00ddb

20 files changed

Lines changed: 613 additions & 8 deletions

File tree

‎packages/system/cilium/Makefile‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,12 @@ export NAMESPACE=cozy-$(NAME)
66
include ../../../hack/common-envs.mk
77
include ../../../hack/package.mk
88

9+
# hack/helm-unit-tests.sh discovers work by probing for this target, so a
10+
# package without one is skipped in silence however many suites sit under
11+
# tests/.
12+
test:
13+
helm unittest .
14+
915
update:
1016
rm -rf charts
1117
helm repo add cilium https://helm.cilium.io/
Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
suite: cilium-agent memory requests and limits
2+
3+
# The cilium-agent DaemonSet ran with no resources at all, which put it in the
4+
# Talos userspace OOM handler's victim set: that handler (v1.12+) only ever
5+
# selects cgroups that lack memory.max, and picks by QoS class rather than by
6+
# what caused the pressure. Losing cilium-agent on a node takes that node's
7+
# datapath programming with it.
8+
#
9+
# The limit is what sets memory.max, and it is set on the POD cgroup only when
10+
# EVERY container in the pod has one — so all seven are asserted here, not just
11+
# the agent. Five of the init containers take their values from initResources;
12+
# install-cni-binaries carries upstream's own cni.resources and is asserted to
13+
# document that it is already covered rather than overlooked.
14+
#
15+
# Both keys are set from the umbrella values.yaml against the vendored subchart.
16+
# An upstream bump that renames or relocates either one renders nothing and
17+
# leaves no other trace, silently returning the pod to the victim set; catching
18+
# that is the main thing these assertions are for.
19+
release:
20+
name: cilium
21+
namespace: cozy-cilium
22+
tests:
23+
- it: cilium-agent declares a memory limit and request
24+
template: charts/cilium/templates/cilium-agent/daemonset.yaml
25+
documentSelector:
26+
path: kind
27+
value: DaemonSet
28+
asserts:
29+
- equal:
30+
path: spec.template.spec.containers[?(@.name=="cilium-agent")].resources.limits.memory
31+
value: 4Gi
32+
- equal:
33+
path: spec.template.spec.containers[?(@.name=="cilium-agent")].resources.requests.memory
34+
value: 256Mi
35+
36+
# Asserted one container at a time. A negated matcher over the init container
37+
# list reads as if it covers them all but passes vacuously, so each limit is
38+
# pinned explicitly instead.
39+
- it: every cilium-agent init container declares a memory limit
40+
template: charts/cilium/templates/cilium-agent/daemonset.yaml
41+
documentSelector:
42+
path: kind
43+
value: DaemonSet
44+
asserts:
45+
- equal:
46+
path: spec.template.spec.initContainers[?(@.name=="config")].resources.limits.memory
47+
value: 256Mi
48+
- equal:
49+
path: spec.template.spec.initContainers[?(@.name=="mount-cgroup")].resources.limits.memory
50+
value: 256Mi
51+
- equal:
52+
path: spec.template.spec.initContainers[?(@.name=="apply-sysctl-overwrites")].resources.limits.memory
53+
value: 256Mi
54+
- equal:
55+
path: spec.template.spec.initContainers[?(@.name=="mount-bpf-fs")].resources.limits.memory
56+
value: 256Mi
57+
- equal:
58+
path: spec.template.spec.initContainers[?(@.name=="clean-cilium-state")].resources.limits.memory
59+
value: 256Mi
60+
# Upstream's cni.resources, not initResources — pinned so a bump that
61+
# drops it is caught here rather than by a pod that quietly loses
62+
# memory.max.
63+
- equal:
64+
path: spec.template.spec.initContainers[?(@.name=="install-cni-binaries")].resources.limits.memory
65+
value: 1Gi
66+
67+
# cilium-envoy is a second DaemonSet on the same nodes. Its limit predates
68+
# this suite; it is pinned here because the pod-level argument above applies
69+
# to it identically.
70+
- it: cilium-envoy declares a memory limit and request
71+
template: charts/cilium/templates/cilium-envoy/daemonset.yaml
72+
documentSelector:
73+
path: kind
74+
value: DaemonSet
75+
asserts:
76+
- equal:
77+
path: spec.template.spec.containers[?(@.name=="cilium-envoy")].resources.limits.memory
78+
value: 1Gi
79+
- equal:
80+
path: spec.template.spec.containers[?(@.name=="cilium-envoy")].resources.requests.memory
81+
value: 512Mi

‎packages/system/cilium/values.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,30 @@ cilium:
3838
memory: 512Mi
3939
limits:
4040
memory: 1Gi
41+
# The cilium-agent DaemonSet had the same gap the envoy DaemonSet above
42+
# already closed. Beyond the node-headroom argument, a memory limit is what
43+
# sets memory.max on the pod cgroup, and the Talos userspace OOM handler
44+
# (v1.12+) only ever selects cgroups that lack it — so an unlimited agent is
45+
# an eviction candidate on every node regardless of what caused the pressure.
46+
# 14d peak on a 3-node dev cluster is 575Mi, and the agent grows with
47+
# endpoint and policy count, so the limit matches the 4Gi in the upstream
48+
# values' own commented example rather than being fitted to that measurement.
49+
resources:
50+
requests:
51+
cpu: 100m
52+
memory: 256Mi
53+
limits:
54+
memory: 4Gi
55+
# Shared by the agent's init containers (config, mount-cgroup, mount-bpf-fs,
56+
# apply-sysctl-overwrites, clean-cilium-state). All are short-lived setup
57+
# steps, but each one still has to carry a limit for the pod cgroup to get
58+
# memory.max at all.
59+
initResources:
60+
requests:
61+
cpu: 10m
62+
memory: 32Mi
63+
limits:
64+
memory: 256Mi
4165
gatewayAPI:
4266
enabled: true
4367
# Upstream default is false — Envoy then advertises no ALPN protocols on

‎packages/system/kubevirt-csi-node/templates/deploy.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -196,10 +196,17 @@ spec:
196196
timeoutSeconds: 3
197197
periodSeconds: 10
198198
failureThreshold: 5
199+
# Limits, not just requests: a memory limit is what sets memory.max on
200+
# the pod cgroup, and the Talos userspace OOM handler (v1.12+) only
201+
# ever kills cgroups that lack one. Requests alone left this DaemonSet
202+
# an eviction candidate on every node whatever caused the pressure.
203+
# Every container in the pod needs one for the pod cgroup to get it.
199204
resources:
200205
requests:
201206
memory: 50Mi
202207
cpu: 10m
208+
limits:
209+
memory: 512Mi
203210
- name: csi-node-driver-registrar
204211
image: quay.io/openshift/origin-csi-node-driver-registrar:latest
205212
args:
@@ -224,6 +231,8 @@ spec:
224231
requests:
225232
memory: 20Mi
226233
cpu: 5m
234+
limits:
235+
memory: 128Mi
227236
- name: csi-liveness-probe
228237
image: quay.io/openshift/origin-csi-livenessprobe:latest
229238
args:
@@ -237,6 +246,8 @@ spec:
237246
requests:
238247
memory: 20Mi
239248
cpu: 5m
249+
limits:
250+
memory: 128Mi
240251
volumes:
241252
- name: kubelet-dir
242253
hostPath:
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
suite: csi-node DaemonSet memory limits
2+
3+
# The node DaemonSet carried requests but no limits, which leaves it Burstable
4+
# rather than Guaranteed-of-a-cgroup-limit: a memory limit is what sets
5+
# memory.max on the pod cgroup, and the Talos userspace OOM handler (v1.12+)
6+
# only ever selects cgroups that lack it. Requests alone did not take the pod
7+
# out of the victim set.
8+
#
9+
# memory.max lands on the POD cgroup only when EVERY container has a limit, so
10+
# all three are asserted rather than the driver alone — one limit-free sidecar
11+
# would put the whole pod back in the victim set while the other two assertions
12+
# still passed, which is precisely the failure this suite exists to catch.
13+
14+
templates:
15+
- templates/deploy.yaml
16+
17+
tests:
18+
- it: every csi-node container declares a memory limit
19+
documentSelector:
20+
path: kind
21+
value: DaemonSet
22+
asserts:
23+
- equal:
24+
path: spec.template.spec.containers[?(@.name=="csi-driver")].resources.limits.memory
25+
value: 512Mi
26+
- equal:
27+
path: spec.template.spec.containers[?(@.name=="csi-node-driver-registrar")].resources.limits.memory
28+
value: 128Mi
29+
- equal:
30+
path: spec.template.spec.containers[?(@.name=="csi-liveness-probe")].resources.limits.memory
31+
value: 128Mi
32+
33+
# Pinned alongside the limits because they are what the limits were sized
34+
# against; a drift in either direction is worth catching.
35+
- it: csi-node containers keep their memory requests
36+
documentSelector:
37+
path: kind
38+
value: DaemonSet
39+
asserts:
40+
- equal:
41+
path: spec.template.spec.containers[?(@.name=="csi-driver")].resources.requests.memory
42+
value: 50Mi
43+
- equal:
44+
path: spec.template.spec.containers[?(@.name=="csi-node-driver-registrar")].resources.requests.memory
45+
value: 20Mi
46+
- equal:
47+
path: spec.template.spec.containers[?(@.name=="csi-liveness-probe")].resources.requests.memory
48+
value: 20Mi

‎packages/system/kubevirt/templates/kubevirt-cr.yaml‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,19 @@ spec:
4242
- Evict
4343
batchEvictionInterval: 1m
4444
batchEvictionSize: 10
45-
customizeComponents: {}
45+
# virt-handler is a node DaemonSet and ran BestEffort. The Talos userspace OOM
46+
# handler (v1.12+) only ever kills cgroups with no memory.max, and picks its
47+
# victim by QoS class rather than by what caused the pressure — so virt-handler
48+
# was an eviction candidate on every node, and losing it stops VM lifecycle
49+
# operations there. virt-operator owns the DaemonSet, so resources cannot be
50+
# set from chart values; customizeComponents.patches is the supported seam.
51+
# 14d peak 207Mi on a 3-node dev cluster; grows with VMs per node.
52+
customizeComponents:
53+
patches:
54+
- resourceName: virt-handler
55+
resourceType: DaemonSet
56+
type: strategic
57+
patch: |
58+
{"spec":{"template":{"spec":{"containers":[{"name":"virt-handler","resources":{"requests":{"cpu":"100m","memory":"128Mi"},"limits":{"memory":"2Gi"}}}]}}}}
4659
imagePullPolicy: IfNotPresent
4760
monitorNamespace: tenant-root

‎packages/system/kubevirt/tests/kubevirt_cr_test.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,3 +92,27 @@ tests:
9292
path: spec.configuration.mediatedDevicesConfiguration.mediatedDeviceTypes
9393
value:
9494
- nvidia-740
95+
96+
# virt-operator owns the virt-handler DaemonSet, so resources cannot come from
97+
# chart values — customizeComponents.patches is the only seam. virt-handler ran
98+
# BestEffort on every node, and the Talos userspace OOM handler (v1.12+) only
99+
# kills cgroups with no memory.max, so it was a standing eviction candidate.
100+
# Pin the patch: a malformed or dropped entry fails silently at runtime, since
101+
# virt-operator simply reconciles the DaemonSet without the resources.
102+
- it: patches virt-handler with memory requests and limits
103+
asserts:
104+
- equal:
105+
path: spec.customizeComponents.patches[0].resourceName
106+
value: virt-handler
107+
- equal:
108+
path: spec.customizeComponents.patches[0].resourceType
109+
value: DaemonSet
110+
- equal:
111+
path: spec.customizeComponents.patches[0].type
112+
value: strategic
113+
- matchRegex:
114+
path: spec.customizeComponents.patches[0].patch
115+
pattern: '"limits":\{"memory":"2Gi"\}'
116+
- matchRegex:
117+
path: spec.customizeComponents.patches[0].patch
118+
pattern: '"requests":\{"cpu":"100m","memory":"128Mi"\}'

‎packages/system/linstor/templates/satellites-cozy.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,24 @@ spec:
1414
containers:
1515
- name: linstor-satellite
1616
image: {{ .Values.piraeusServer.image.repository }}:{{ .Values.piraeusServer.image.tag }}
17+
# The satellite DaemonSet ran BestEffort on every storage node, which
18+
# made it a standing victim for the Talos userspace OOM handler
19+
# (v1.12+): that handler only ever kills cgroups with no memory.max, and
20+
# picks by QoS class rather than by what caused the pressure. A tenant
21+
# workload thrashing against its own limit would get the satellite
22+
# killed, taking DRBD connections down with it.
23+
#
24+
# Note this is the opposite call from linstor-controller, which is
25+
# deliberately left limit-free in values.yaml (a JVM that must not be
26+
# OOM-killed mid-operation). The satellite is a Go binary with a bounded
27+
# working set, so a generous limit costs nothing and buys immunity.
28+
# 14d peak 368Mi on a 3-node dev cluster; grows with resource count.
29+
resources:
30+
requests:
31+
cpu: 100m
32+
memory: 128Mi
33+
limits:
34+
memory: 2Gi
1735
startupProbe:
1836
tcpSocket:
1937
port: linstor

‎packages/system/linstor/templates/satellites-plunger.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,17 @@ spec:
1212
containers:
1313
- name: plunger
1414
image: {{ .Values.piraeusServer.image.repository }}:{{ .Values.piraeusServer.image.tag }}
15+
# Sidecars in the satellite pod. Every container needs a memory limit
16+
# for the pod cgroup to get memory.max, which is what keeps the pod out
17+
# of the Talos userspace OOM handler's victim set — one limit-free
18+
# sidecar is enough to put the whole satellite pod back in it.
19+
# 14d peaks on a 3-node dev cluster: plunger 18Mi, drbd-logger 5Mi.
20+
resources:
21+
requests:
22+
cpu: 10m
23+
memory: 32Mi
24+
limits:
25+
memory: 128Mi
1526
command:
1627
- "/scripts/plunger-satellite.sh"
1728
securityContext:
@@ -49,6 +60,12 @@ spec:
4960
readOnly: true
5061
- name: drbd-logger
5162
image: {{ .Values.piraeusServer.image.repository }}:{{ .Values.piraeusServer.image.tag }}
63+
resources:
64+
requests:
65+
cpu: 10m
66+
memory: 32Mi
67+
limits:
68+
memory: 128Mi
5269
command:
5370
- "/scripts/plunger-drbd-logger.sh"
5471
securityContext:
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
suite: satellite pod resource limits
2+
3+
# The satellite DaemonSet ran BestEffort, which made it a standing victim for the
4+
# Talos userspace OOM handler (v1.12+): that handler only ever kills cgroups with
5+
# no memory.max, and picks its victim by QoS class rather than by what caused the
6+
# pressure. A tenant workload thrashing against its own limit would get the
7+
# satellite killed, taking DRBD connections on that node down with it.
8+
#
9+
# Every container in the pod needs a limit for the pod cgroup to get memory.max,
10+
# so the plunger sidecars are pinned alongside the satellite itself — one
11+
# limit-free sidecar is enough to put the whole pod back in the victim set.
12+
#
13+
# Note this is the opposite call from linstor-controller, which cluster_test.yaml
14+
# pins as deliberately limit-free (a JVM that must not be OOM-killed mid-operation).
15+
16+
templates:
17+
- templates/satellites-cozy.yaml
18+
- templates/satellites-plunger.yaml
19+
20+
tests:
21+
- it: satellite declares a memory limit and request
22+
template: templates/satellites-cozy.yaml
23+
asserts:
24+
- equal:
25+
path: spec.podTemplate.spec.containers[?(@.name=="linstor-satellite")].resources.limits.memory
26+
value: 2Gi
27+
- equal:
28+
path: spec.podTemplate.spec.containers[?(@.name=="linstor-satellite")].resources.requests.memory
29+
value: 128Mi
30+
31+
- it: plunger sidecars declare memory limits
32+
template: templates/satellites-plunger.yaml
33+
asserts:
34+
- equal:
35+
path: spec.podTemplate.spec.containers[?(@.name=="plunger")].resources.limits.memory
36+
value: 128Mi
37+
- equal:
38+
path: spec.podTemplate.spec.containers[?(@.name=="drbd-logger")].resources.limits.memory
39+
value: 128Mi

0 commit comments

Comments
 (0)