Skip to content

Commit 5786afe

Browse files
authored
feat(kubernetes): add kubelet resource reservations for worker nodes (#2420)
**What this PR does** Adds configurable kubelet resource reservations and eviction thresholds for tenant cluster worker nodes. Without explicit reservations and cgroup enforcement, the scheduler treats nearly all node memory and CPU as allocatable. Under pressure, the OOM killer can target kubelet itself because no cgroup boundaries protect it, leading to "Kubelet stopped posting node status" failures and unrecoverable NotReady nodes. This PR auto-computes per-node-group kubelet reservations from the effective node capacity (instanceType or explicit `resources`): - **Memory**: `systemReservedMemory` and `kubeReservedMemory` each default to 5% of effective memory, clamped 256Mi-1Gi - **CPU**: `systemReservedCpu` and `kubeReservedCpu` each default to 5% of effective CPU, clamped 50m-500m - **Eviction**: configurable `evictionHardMemory` (default 7%) and `evictionSoftMemory` (default 10%) with validation that hard < soft - **Disk eviction**: `nodefs.available`, `imagefs.available`, and `nodefs.inodesFree` thresholds are preserved alongside memory signals - **Capacity annotations**: cluster-autoscaler annotations now report allocatable memory and CPU (total minus all reservations and eviction threshold) instead of raw totals - **Cgroup enforcement**: `enforce-node-allocatable=pods` configures advisory-only reservations — the scheduler accounts for reserved resources when computing allocatable capacity, but no cgroup boundaries are enforced for system or kube reserved resources - **Validation**: format checks, zero-value rejection, eviction ordering (hard < soft), same-unit-type requirement for eviction thresholds, and total reservation sanity check against effective node capacity All reservation fields are optional and overridable per node group via `.nodeGroups[].kubelet`. **Default rendered kubeletExtraArgs:** ```yaml kubeletExtraArgs: system-reserved: "memory=256Mi,cpu=50m" kube-reserved: "memory=256Mi,cpu=50m" eviction-hard: "memory.available<7%,nodefs.available<10%,imagefs.available<15%,nodefs.inodesFree<5%" eviction-soft: "memory.available<10%,nodefs.available<15%,imagefs.available<20%" eviction-soft-grace-period: "memory.available=1m30s,nodefs.available=1m30s,imagefs.available=1m30s" eviction-minimum-reclaim: "memory.available=256Mi" enforce-node-allocatable: "pods" ``` **New values schema:** ```yaml nodeGroups: md0: kubelet: systemReservedMemory: "" # auto-computed (5% of effective memory, 256Mi-1Gi) kubeReservedMemory: "" # auto-computed (5% of effective memory, 256Mi-1Gi) systemReservedCpu: "" # auto-computed (5% of effective CPU, 50m-500m) kubeReservedCpu: "" # auto-computed (5% of effective CPU, 50m-500m) evictionHardMemory: "7%" # absolute (e.g. 200Mi) or percentage evictionSoftMemory: "10%" # must be > evictionHardMemory, same unit type ``` **Release note:** ``` feat(kubernetes): add per-node-group kubelet resource reservations with auto-computed system/kube reserved memory (5% of effective memory, clamped 256Mi-1Gi) and CPU (5% of effective CPU, clamped 50m-500m), configurable memory eviction thresholds, preserved disk eviction signals (nodefs, imagefs, inodesFree), pods-only cgroup enforcement, and reservation-aware allocatable capacity annotations for cluster-autoscaler. ```
2 parents b5a1ae5 + 22d2a22 commit 5786afe

10 files changed

Lines changed: 2211 additions & 38 deletions

File tree

‎api/apps/v1alpha1/kubernetes/types.go‎

Lines changed: 20 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go‎

Lines changed: 16 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/kubernetes/Makefile‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ KUBERNETES_PKG_TAG = $(shell awk '$$1 == "version:" {print $$2}' Chart.yaml)
44
include ../../../hack/common-envs.mk
55
include ../../../hack/package.mk
66

7+
.PHONY: test generate update
8+
79
test:
810
helm unittest .
911

‎packages/apps/kubernetes/README.md‎

Lines changed: 81 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ Within a tenant cluster, users can take advantage of LoadBalancer services and e
1212
The control-plane operates within containers, while the worker nodes are deployed as virtual machines, all seamlessly managed by the application.
1313

1414
Kubernetes version in tenant clusters is independent of Kubernetes in the management cluster.
15-
Users can select the latest patch versions from 1.28 to 1.33.
15+
Users can select the supported patch versions from 1.30 to 1.35.
1616

1717
## Why Use a Managed Kubernetes Cluster?
1818

@@ -98,22 +98,29 @@ See the reference for components utilized in this service:
9898

9999
### Application-specific Parameters
100100

101-
| Name | Description | Type | Value |
102-
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ----------- |
103-
| `nodeGroups` | Worker nodes configuration map. | `map[string]object` | `{...}` |
104-
| `nodeGroups[name].minReplicas` | Minimum number of replicas. | `int` | `0` |
105-
| `nodeGroups[name].maxReplicas` | Maximum number of replicas. | `int` | `10` |
106-
| `nodeGroups[name].instanceType` | Virtual machine instance type. | `string` | `u1.medium` |
107-
| `nodeGroups[name].diskSize` | Persistent disk size for kubelet and containerd data. | `quantity` | `20Gi` |
108-
| `nodeGroups[name].storageClass` | StorageClass for worker node persistent disks. When empty, uses the management cluster default StorageClass (the one annotated storageclass.kubernetes.io/is-default-class: true). | `string` | `""` |
109-
| `nodeGroups[name].roles` | List of node roles. | `[]string` | `[]` |
110-
| `nodeGroups[name].resources` | CPU and memory resources for each worker node. | `object` | `{}` |
111-
| `nodeGroups[name].resources.cpu` | CPU available. | `quantity` | `""` |
112-
| `nodeGroups[name].resources.memory` | Memory (RAM) available. | `quantity` | `""` |
113-
| `nodeGroups[name].gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` |
114-
| `nodeGroups[name].gpus[i].name` | Name of GPU, such as "nvidia.com/AD102GL_L40S". | `string` | `""` |
115-
| `version` | Kubernetes major.minor version to deploy | `string` | `v1.35` |
116-
| `host` | External hostname for Kubernetes cluster. Defaults to `<cluster-name>.<tenant-host>` if empty. | `string` | `""` |
101+
| Name | Description | Type | Value |
102+
| ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ----------- |
103+
| `nodeGroups` | Worker nodes configuration map. | `map[string]object` | `{...}` |
104+
| `nodeGroups[name].minReplicas` | Minimum number of replicas. | `int` | `0` |
105+
| `nodeGroups[name].maxReplicas` | Maximum number of replicas. | `int` | `10` |
106+
| `nodeGroups[name].instanceType` | Virtual machine instance type. | `string` | `u1.medium` |
107+
| `nodeGroups[name].diskSize` | Persistent disk size for kubelet and containerd data. | `quantity` | `20Gi` |
108+
| `nodeGroups[name].storageClass` | StorageClass for worker node persistent disks. When empty, uses the management cluster default StorageClass (the one annotated storageclass.kubernetes.io/is-default-class: true). | `string` | `""` |
109+
| `nodeGroups[name].roles` | List of node roles. | `[]string` | `[]` |
110+
| `nodeGroups[name].resources` | CPU and memory resources for each worker node. | `object` | `{}` |
111+
| `nodeGroups[name].resources.cpu` | CPU available. | `quantity` | `""` |
112+
| `nodeGroups[name].resources.memory` | Memory (RAM) available. | `quantity` | `""` |
113+
| `nodeGroups[name].gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` |
114+
| `nodeGroups[name].gpus[i].name` | Name of GPU, such as "nvidia.com/AD102GL_L40S". | `string` | `""` |
115+
| `nodeGroups[name].kubelet` | Kubelet resource reservations for this node group. | `object` | `{}` |
116+
| `nodeGroups[name].kubelet.systemReservedMemory` | Memory reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` |
117+
| `nodeGroups[name].kubelet.kubeReservedMemory` | Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` |
118+
| `nodeGroups[name].kubelet.systemReservedCpu` | CPU reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` |
119+
| `nodeGroups[name].kubelet.kubeReservedCpu` | CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` |
120+
| `nodeGroups[name].kubelet.evictionHardMemory` | Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%). | `string` | `7%` |
121+
| `nodeGroups[name].kubelet.evictionSoftMemory` | Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%). | `string` | `10%` |
122+
| `version` | Kubernetes major.minor version to deploy | `string` | `v1.35` |
123+
| `host` | External hostname for Kubernetes cluster. Defaults to `<cluster-name>.<tenant-host>` if empty. | `string` | `""` |
117124

118125

119126
### Cluster Addons
@@ -266,6 +273,63 @@ The following instanceType resources are provided by Cozystack:
266273
| `u1.small` | 1 | 2Gi |
267274
| `u1.xlarge` | 4 | 16Gi |
268275

276+
### Kubelet Resource Reservations
277+
278+
Each node group supports a `kubelet` object that configures how much memory and CPU the kubelet reserves for the host OS and the Kubernetes/system components running on the worker node.
279+
280+
When `systemReservedMemory` or `kubeReservedMemory` are left empty, they are **auto-computed** using the following formula:
281+
282+
1. Determine the **effective memory** of the node:
283+
- If `resources.memory` is explicitly set, use that value.
284+
- Otherwise, look up the `instanceType` and use its `memory.guest` value.
285+
- If neither is available, the reservation falls back to the minimum (256Mi).
286+
2. Calculate **5%** of the effective memory (in MiB, rounded down).
287+
3. **Clamp** the result to the range **[256Mi, 1Gi]**:
288+
- Nodes with 5 GiB or less get the minimum 256Mi reservation.
289+
- Nodes with 20 GiB or more get the maximum 1Gi reservation.
290+
291+
Both `systemReservedMemory` and `kubeReservedMemory` receive the same auto-computed value by default.
292+
293+
CPU reservations (`systemReservedCpu`, `kubeReservedCpu`) follow the same pattern: **5%** of effective CPU, clamped to **[50m, 500m]**. Both are auto-computed when left empty.
294+
295+
#### Kubelet Defaults
296+
297+
| Parameter | Default | Description |
298+
| --- | --- | --- |
299+
| `systemReservedMemory` | auto-computed | Memory reserved for host OS |
300+
| `kubeReservedMemory` | auto-computed | Memory reserved for kubelet and container runtime |
301+
| `systemReservedCpu` | auto-computed | CPU reserved for host OS |
302+
| `kubeReservedCpu` | auto-computed | CPU reserved for kubelet and container runtime |
303+
| `evictionHardMemory` | `7%` | Hard eviction threshold for memory |
304+
| `evictionSoftMemory` | `10%` | Soft eviction threshold for memory |
305+
| `evictionSoftGracePeriod` | `1m30s` *(hardcoded)* | Duration a soft eviction threshold must be breached before triggering eviction |
306+
| `evictionMinimumReclaim` | `256Mi` *(hardcoded)* | Minimum amount of memory reclaimed per eviction action |
307+
308+
Eviction thresholds can be specified as percentages (e.g., `7%`) or absolute values (e.g., `200Mi`). Both thresholds must use the same unit type. The hard threshold must be strictly less than the soft threshold.
309+
310+
The `evictionSoftGracePeriod` and `evictionMinimumReclaim` parameters are currently hardcoded in the chart template and cannot be overridden through values.
311+
312+
#### Capacity Annotation
313+
314+
When kubelet resource reservations are configured, both the `capacity.cluster-autoscaler.kubernetes.io/memory` and `capacity.cluster-autoscaler.kubernetes.io/cpu` annotations on MachineDeployments report **allocatable** values instead of total node resources. Memory allocatable subtracts system-reserved, kube-reserved, and eviction-hard. CPU allocatable subtracts system-reserved and kube-reserved. This aligns the annotations with the values the cluster autoscaler uses in its scaling calculations.
315+
316+
Upgrading from a version without this feature may cause the autoscaler to see reduced per-node capacity after the annotations are updated, which can trigger additional scale-up operations. No action is typically required — the new values reflect the actual memory available for workload scheduling.
317+
318+
> **Note:** When neither `resources.memory` nor `instanceType` is set, eviction thresholds (default 7% hard / 10% soft) are still enforced by the kubelet at runtime, but the capacity annotation is not rendered. Without this annotation, the cluster-autoscaler has no visibility into these reservations and may over-schedule the node until evictions fire.
319+
320+
#### Example: Override kubelet reservations
321+
322+
```yaml
323+
nodeGroups:
324+
md0:
325+
instanceType: "u1.large"
326+
kubelet:
327+
systemReservedMemory: "256Mi"
328+
kubeReservedMemory: "256Mi"
329+
evictionHardMemory: "500Mi"
330+
evictionSoftMemory: "1Gi"
331+
```
332+
269333
### U Series: Universal
270334

271335
The U Series is quite neutral and provides resources for

‎packages/apps/kubernetes/templates/_resources.tpl‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,3 +47,16 @@ These presets are for basic testing and not meant to be used in production
4747
{{- printf "ERROR: Preset key '%s' invalid. Allowed values are %s" .type (join "," (keys $presets)) | fail -}}
4848
{{- end -}}
4949
{{- end -}}
50+
51+
{{/*
52+
Convert a CPU quantity string (e.g. "100m", "0.5", "1", "4") to millicores.
53+
Returns a number string: "100", "500", "1000", "4000".
54+
*/}}
55+
{{- define "kubernetes.cpuToMillicores" -}}
56+
{{- $str := . | toString -}}
57+
{{- if hasSuffix "m" $str -}}
58+
{{- trimSuffix "m" $str -}}
59+
{{- else -}}
60+
{{- mulf ($str | float64) 1000.0 | int | toString -}}
61+
{{- end -}}
62+
{{- end -}}

0 commit comments

Comments
 (0)