Skip to content

Commit 85ccadb

Browse files
committed
fix(kubernetes): add kubelet reserved resources and cgroup enforcement
Configure kubelet with proper resource reservations and cgroup enforcement to prevent OOM kills when memory pressure occurs. Configuration: - systemReserved: 100m CPU, 512Mi memory (OS daemons) - kubeReserved: 100m CPU, 512Mi memory (kubelet, containerd) - evictionHard: 512Mi threshold (immediate eviction) - evictionSoft: 1Gi threshold with 90s grace period - evictionMinimumReclaim: 256Mi (guaranteed reclaim after eviction) - enforce-node-allocatable: creates hard cgroup limits for reservations - cgroups-per-qos: enables QoS-based resource isolation - cgroup-driver: systemd (matches containerd configuration) - kernel-memcg-notification: faster memory pressure detection Allocatable calculation for 16Gi VM: VM capacity: 16.00 Gi - systemReserved: -0.50 Gi (enforced via cgroup) - kubeReserved: -0.50 Gi (enforced via cgroup) - evictionHard buffer: -0.50 Gi - kernel/page cache: -0.25 Gi (implicit) ---------------------------------------- Allocatable (safe): 14.25 Gi Key improvement: enforce-node-allocatable creates hard cgroup limits, preventing system/kubelet processes from exceeding reservations and killing kubelet via OOM. Related issue: tenant VMs experiencing kubelet crashes after 15-30min due to insufficient reserved memory (100Mi vs required 1.75Gi total) and lack of cgroup enforcement.
1 parent e26894e commit 85ccadb

1 file changed

Lines changed: 11 additions & 1 deletion

File tree

‎packages/apps/kubernetes/templates/cluster.yaml‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -250,7 +250,17 @@ spec:
250250
- systemctl start containerd.service
251251
joinConfiguration:
252252
nodeRegistration:
253-
kubeletExtraArgs: {}
253+
kubeletExtraArgs:
254+
system-reserved: "cpu=100m,memory=512Mi"
255+
kube-reserved: "cpu=100m,memory=512Mi"
256+
eviction-hard: "memory.available<512Mi"
257+
eviction-soft: "memory.available<1Gi"
258+
eviction-soft-grace-period: "memory.available=1m30s"
259+
eviction-minimum-reclaim: "memory.available=256Mi"
260+
enforce-node-allocatable: "pods,system-reserved,kube-reserved"
261+
cgroups-per-qos: "true"
262+
cgroup-driver: "systemd"
263+
kernel-memcg-notification: "true"
254264
# Ignore this for 1.31
255265
ignorePreflightErrors:
256266
- FileExisting-conntrack

0 commit comments

Comments
 (0)