Skip to content

Commit 2ca7cc2

Browse files
committed
fix(kafka): size broker and operator heaps from their container limits
Strimzi sizes a broker heap by running the image's JDK 17 against the container memory and taking half of what it reports, capped at 5 GiB. On kernels built without CONFIG_MEMCG_V1 that JDK cannot see the limit and reports host RAM, so every broker and controller starts with -Xms5g -Xmx5g inside a 1Gi limit and is OOM-killed. The topic and user operators run the same JDK with percentage-based defaults, and the drain-time Cruise Control takes 75% of host RAM for both -Xms and -Xmx. The chart now applies Strimzi's rule, half the limit capped at 5 GiB, to the rendered limits and passes the result through jvmOptions, and the entity operators get their limit as -XX:MaxRAM. The node pools carry no -XX option on purpose: Strimzi puts every one except the RAM percentages into KAFKA_JVM_PERFORMANCE_OPTS, and kafka-run-class.sh applies its G1 settings only when that variable is empty. On an affected kernel their GC therefore still sizes its thread pools from the host CPU count. Existing clusters roll once on upgrade, because the pod environment changes. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 79d175d commit 2ca7cc2

4 files changed

Lines changed: 121 additions & 5 deletions

File tree

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{{- /*
2+
Heap options for a node pool, from its sanitized resources. Strimzi sizes the
3+
Kafka heap by asking the image's JDK 17 for the container memory, and on
4+
kernels built without CONFIG_MEMCG_V1 that JDK cannot see the limit
5+
(JDK-8347811): it answers with host RAM, and Strimzi starts every broker
6+
with up to a 5Gi heap. This applies Strimzi's own rule to the real limit,
7+
half of it capped at 5Gi. No -XX option here: Strimzi moves every one except
8+
the RAM percentages into KAFKA_JVM_PERFORMANCE_OPTS, and kafka-run-class.sh
9+
applies its G1 tuning only while that variable is empty.
10+
*/ -}}
11+
{{- define "kafka.jvmOptions" -}}
12+
{{- $memory := include "cozy-lib.resources.toFloat" .limits.memory | float64 | int64 -}}
13+
{{- $heap := printf "%dm" (min (div $memory 2097152) 5120) -}}
14+
{{- dict "-Xms" $heap "-Xmx" $heap | toYaml -}}
15+
{{- end -}}

‎packages/apps/kafka/templates/kafka.yaml‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
{{- $tlsEnabled := include "kafka.tls.enabled" . | trim -}}
1717
{{- $showExternal := .Values.external -}}
1818
{{- $kafkaVersion := include "kafka.versionMap" . -}}
19+
{{- $entityOperatorMemory := "512Mi" -}}
1920
apiVersion: kafka.strimzi.io/v1beta2
2021
kind: Kafka
2122
metadata:
@@ -75,21 +76,30 @@ spec:
7576
# The topic-operator and user-operator are JVM processes and OOMKill at
7677
# that default, leaving the entity-operator in CrashLoopBackOff so
7778
# KafkaTopics/KafkaUsers never reconcile. Set explicit resources so they
78-
# don't inherit the namespace default.
79+
# don't inherit the namespace default. MaxRAM hands the same limit to the
80+
# JVM, because the JDK in the operator image cannot read it on kernels
81+
# built without CONFIG_MEMCG_V1 (JDK-8347811) and sizes its heap from host
82+
# RAM instead.
7983
topicOperator:
8084
resources:
8185
requests:
8286
cpu: 100m
8387
memory: 256Mi
8488
limits:
85-
memory: 512Mi
89+
memory: {{ $entityOperatorMemory }}
90+
jvmOptions:
91+
-XX:
92+
MaxRAM: {{ include "cozy-lib.resources.toFloat" $entityOperatorMemory | float64 | int64 | quote }}
8693
userOperator:
8794
resources:
8895
requests:
8996
cpu: 100m
9097
memory: 256Mi
9198
limits:
92-
memory: 512Mi
99+
memory: {{ $entityOperatorMemory }}
100+
jvmOptions:
101+
-XX:
102+
MaxRAM: {{ include "cozy-lib.resources.toFloat" $entityOperatorMemory | float64 | int64 | quote }}
93103
template:
94104
# `pod` accepts metadata + a flat set of pod-spec fields
95105
# (enableServiceLinks, securityContext, affinity, …) directly.

‎packages/apps/kafka/templates/migration-hook.yaml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,7 @@ spec:
143143
roles:
144144
- broker
145145
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | nindent 18 }}
146+
jvmOptions: {{- include "kafka.jvmOptions" (include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | fromYaml) | nindent 18 }}
146147
storage:
147148
type: jbod
148149
volumes:
@@ -187,6 +188,7 @@ spec:
187188
roles:
188189
- controller
189190
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | nindent 18 }}
191+
jvmOptions: {{- include "kafka.jvmOptions" (include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | fromYaml) | nindent 18 }}
190192
storage:
191193
type: jbod
192194
volumes:
@@ -388,6 +390,7 @@ spec:
388390
roles:
389391
- broker
390392
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | nindent 18 }}
393+
jvmOptions: {{- include "kafka.jvmOptions" (include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | fromYaml) | nindent 18 }}
391394
storage:
392395
type: jbod
393396
volumes:
@@ -407,9 +410,11 @@ spec:
407410
# explicit resources: Cruise Control is a JVM, and a tenant namespace
408411
# LimitRange defaults containers to 128Mi, which OOMs it before it
409412
# can produce a proposal (same reason the entity-operator carries
410-
# explicit resources in kafka.yaml).
413+
# explicit resources in kafka.yaml). The heap is 75% of the limit,
414+
# what Strimzi picks when the JDK can read the limit; see
415+
# kafka.jvmOptions.
411416
echo "Enabling Cruise Control for the drain..."
412-
kubectl -n "$ns" patch "$kafka" "$name" --type merge -p '{"spec":{"cruiseControl":{"resources":{"requests":{"cpu":"200m","memory":"768Mi"},"limits":{"memory":"1Gi"}}}}}'
417+
kubectl -n "$ns" patch "$kafka" "$name" --type merge -p '{"spec":{"cruiseControl":{"resources":{"requests":{"cpu":"200m","memory":"768Mi"},"limits":{"memory":"1Gi"}},"jvmOptions":{"-Xms":"768m","-Xmx":"768m"}}}}'
413418
414419
# F3: drain all partitions off the "kafka" pool broker IDs.
415420
kafkaIds=$(kubectl -n "$ns" get "$np" kafka -o jsonpath='{.status.nodeIds}' 2>/dev/null || echo "")
Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
suite: Kafka JVMs are sized from their container limits
2+
3+
templates:
4+
- templates/migration-hook.yaml
5+
- templates/kafka.yaml
6+
7+
release:
8+
name: test-kafka
9+
namespace: tenant-test
10+
11+
tests:
12+
- it: sizes the heap of every node pool from the default preset
13+
template: templates/migration-hook.yaml
14+
documentIndex: 3
15+
asserts:
16+
- matchRegex:
17+
path: spec.template.spec.containers[0].args[0]
18+
pattern: "release-namespace: \\$\\{ns\\}\\s+spec:\\s+replicas: \\d+\\s+roles:\\s+- broker\\s+resources:\\s+limits:\\s+cpu: \"1\"\\s+ephemeral-storage: 2Gi\\s+memory: 1Gi\\s+requests:(\\s+\\S+: \\S+)+\\s+jvmOptions:\\s+-Xms: 512m\\s+-Xmx: 512m\\s+storage:"
19+
- matchRegex:
20+
path: spec.template.spec.containers[0].args[0]
21+
pattern: "roles:\\s+- controller\\s+resources:\\s+limits:\\s+cpu: \"1\"\\s+ephemeral-storage: 2Gi\\s+memory: 1Gi\\s+requests:(\\s+\\S+: \\S+)+\\s+jvmOptions:\\s+-Xms: 512m\\s+-Xmx: 512m\\s+storage:"
22+
- matchRegex:
23+
path: spec.template.spec.containers[0].args[0]
24+
pattern: "next-node-ids: \"\\[1000-1099\\]\"\\s+spec:\\s+replicas: \\d+\\s+roles:\\s+- broker\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+\\s+jvmOptions:\\s+-Xms: 512m\\s+-Xmx: 512m\\s+storage:"
25+
26+
# A -XX option other than the RAM percentages fills KAFKA_JVM_PERFORMANCE_OPTS,
27+
# and kafka-run-class.sh then drops its G1 settings.
28+
- it: leaves the broker GC options to Kafka
29+
template: templates/migration-hook.yaml
30+
documentIndex: 3
31+
asserts:
32+
- notMatchRegex:
33+
path: spec.template.spec.containers[0].args[0]
34+
pattern: "jvmOptions:(\\s+-Xm[sx]: \\S+)*\\s+-XX"
35+
36+
- it: caps the node pool heap at 5Gi like the Strimzi default
37+
template: templates/migration-hook.yaml
38+
documentIndex: 3
39+
set:
40+
kafka.resources:
41+
cpu: "4"
42+
memory: 16Gi
43+
asserts:
44+
- matchRegex:
45+
path: spec.template.spec.containers[0].args[0]
46+
pattern: "roles:\\s+- broker\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+\\s+jvmOptions:\\s+-Xms: 5120m\\s+-Xmx: 5120m\\s+storage:"
47+
48+
- it: gives a node pool half of a non-round memory limit
49+
template: templates/migration-hook.yaml
50+
documentIndex: 3
51+
set:
52+
kafka.resources:
53+
cpu: "1"
54+
memory: 3Gi
55+
asserts:
56+
- matchRegex:
57+
path: spec.template.spec.containers[0].args[0]
58+
pattern: "roles:\\s+- controller\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+\\s+jvmOptions:\\s+-Xms: 1536m\\s+-Xmx: 1536m\\s+storage:"
59+
60+
- it: gives the drain-time Cruise Control a heap that fits its limit
61+
template: templates/migration-hook.yaml
62+
documentIndex: 3
63+
asserts:
64+
- matchRegex:
65+
path: spec.template.spec.containers[0].args[0]
66+
pattern: "\"cruiseControl\":\\{\"resources\":\\{\"requests\":\\{\"cpu\":\"200m\",\"memory\":\"768Mi\"\\},\"limits\":\\{\"memory\":\"1Gi\"\\}\\},\"jvmOptions\":\\{\"-Xms\":\"768m\",\"-Xmx\":\"768m\"\\}\\}"
67+
68+
- it: passes the entity operator memory limit to its JVMs
69+
template: templates/kafka.yaml
70+
asserts:
71+
- equal:
72+
path: spec.entityOperator.topicOperator.resources.limits.memory
73+
value: 512Mi
74+
- equal:
75+
path: spec.entityOperator.topicOperator.jvmOptions
76+
value:
77+
-XX:
78+
MaxRAM: "536870912"
79+
- equal:
80+
path: spec.entityOperator.userOperator.resources.limits.memory
81+
value: 512Mi
82+
- equal:
83+
path: spec.entityOperator.userOperator.jvmOptions
84+
value:
85+
-XX:
86+
MaxRAM: "536870912"

0 commit comments

Comments
 (0)