Repository navigation
Commit 1c00ddb
authored
fix(platform): give node DaemonSets memory requests and limits (#3495)
## What this PR does
Sets memory requests and limits on the node DaemonSets that had none.
Companion to the namespace-wide LimitRange; the reasoning for why a
limit specifically (and not a request, and not a `PriorityClass`) is in
that PR and in `docs/operations/system-memory-limits.md`.
Requests come from 14-day peak working set measured on a 3-node dev
cluster. Limits are deliberately loose, sized as a runaway backstop
rather than a fitted ceiling, for two reasons: several of these scale
with cluster size in ways a small dev cluster understates (cilium with
endpoint and policy count, fluent-bit with log volume, multus and
kube-ovn with pods per node, velero node-agent during a backup), and
undersizing converts a rare pressure-driven kill into a deterministic
one. This repository already carries a scar from that failure mode,
recorded in the `vertical-pod-autoscaler` values where a 110Mi limit
OOM-killed the updater and cascaded into failed HelmReleases.
Two DaemonSets are not chart-rendered and needed their own seam.
`linstor-satellite` goes through the `LinstorSatelliteConfiguration`
podTemplate, including the two plunger sidecars — every container in the
pod needs a limit for the pod cgroup to get `memory.max`, so one
limit-free sidecar would put the whole pod back in the victim set.
`virt-handler` goes through KubeVirt `customizeComponents.patches`,
because virt-operator owns that DaemonSet and chart values cannot reach
it.
`linstor-controller` is deliberately left limit-free. It is a JVM that
must not be OOM-killed mid-operation, and `cluster_test.yaml` pins that
as an invariant. The satellite is a Go binary with a bounded working
set, so the opposite call applies there.
Not covered, because upstream hardcodes the containers with no
`resources` knob: the four frr-k8s `cp-*` init containers, the kube-ovn
`hostpath-init` and `install-cni` init containers, and `cozy-proxy`,
whose vendored chart exposes no resources value at all. `hami` and
`kilo` are left alone as optional components with no usage measurements
behind them. All of these still receive a limit from the namespace
LimitRange, so they are not left exposed — they simply carry no request.
Worth stating plainly: this PR does **not** grant OOM immunity on its
own, because the init containers it cannot reach leave the pod cgroup
without `memory.max`. That only closes with the LimitRange PR in place.
### Verification
Every touched package renders and its helm-unittest suite passes. New
assertions cover the metallb speaker and all five frr-k8s containers,
the linstor satellite and plunger sidecars, and the KubeVirt patch. The
metallb suite was mutation-tested: removing the frr-k8s reloader limit
fails it.
One gap. The KubeVirt patch shape was validated against the live CRD and
pushed through the real update path with `kubectl patch
--dry-run=server` (accepted; live object confirmed unchanged), but
virt-operator actually reconciling the DaemonSet with those resources
has **not** been observed, because that means restarting virt-handler
fleet-wide. Worth confirming in e2e or on a dedicated stand before
merge.
### Downstream repositories
Walked the trigger map against the diff. No entry is triggered: this
changes chart values and cozystack-owned templates under
`packages/system/`, not the `cozy-proxy` label and annotation contract,
not node prerequisites, not an app schema or version enum.
Leaving the boxes empty rather than ticking "none affected", because the
diff surfaced a real gap in a downstream repository even though nothing
here forces a change there: the vendored `cozy-proxy` chart exposes no
`resources` value, so its DaemonSet cannot be given one from this side
at all. That wants an issue or PR in cozystack/cozy-proxy followed by a
re-vendor. Flagging it for a maintainer decision rather than filing
speculatively.
### Release note
```release-note
fix(platform): node DaemonSets (cilium, metallb speaker and frr-k8s, kube-ovn, linstor-satellite, virt-handler, fluent-bit, node-exporter, multus, velero node-agent, kubevirt-csi-node) now declare memory requests and limits, so they are no longer selected as victims by the Talos userspace OOM handler when an unrelated workload drives node memory pressure.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Resource Management**
- Added CPU and memory requests and limits across Cilium, KubeVirt CSI,
KubeVirt, Linstor, MetalLB, monitoring agents, Multus, and Velero
workloads.
- Improved workload stability with defined memory ceilings and resource
reservations.
- **Multus**
- Moved resources to the dedicated `cozy-multus` namespace and
configured Cilium as the master CNI.
- Added optional CNI plugin staging with enhanced security settings.
- **Tests**
- Added and expanded Helm tests validating resource configurations and
Multus behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->20 files changed
Lines changed: 613 additions & 8 deletions
File tree
- packages/system
- cilium
- tests
- kubevirt-csi-node
- templates
- tests
- kubevirt
- templates
- tests
- linstor
- templates
- tests
- metallb
- tests
- monitoring-agents
- tests
- multus
- patches
- templates
- tests
- velero
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
9 | 15 | | |
10 | 16 | | |
11 | 17 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
41 | 65 | | |
42 | 66 | | |
43 | 67 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
196 | 196 | | |
197 | 197 | | |
198 | 198 | | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
199 | 204 | | |
200 | 205 | | |
201 | 206 | | |
202 | 207 | | |
| 208 | + | |
| 209 | + | |
203 | 210 | | |
204 | 211 | | |
205 | 212 | | |
| |||
224 | 231 | | |
225 | 232 | | |
226 | 233 | | |
| 234 | + | |
| 235 | + | |
227 | 236 | | |
228 | 237 | | |
229 | 238 | | |
| |||
237 | 246 | | |
238 | 247 | | |
239 | 248 | | |
| 249 | + | |
| 250 | + | |
240 | 251 | | |
241 | 252 | | |
242 | 253 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
45 | | - | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
46 | 59 | | |
47 | 60 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
92 | 92 | | |
93 | 93 | | |
94 | 94 | | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
17 | 35 | | |
18 | 36 | | |
19 | 37 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
15 | 26 | | |
16 | 27 | | |
17 | 28 | | |
| |||
49 | 60 | | |
50 | 61 | | |
51 | 62 | | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
52 | 69 | | |
53 | 70 | | |
54 | 71 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
0 commit comments