Skip to content

Commit 40dade5

Browse files
committed
fix(kafka): pass the container limits to the Strimzi-managed JVMs
Strimzi sizes a broker heap by asking the image's JDK 17 what heap it would pick at MaxRAMPercentage=50, capped at 5 GiB. On kernels built without CONFIG_MEMCG_V1 that JDK cannot see the limit and reports host RAM, so every broker and controller starts with up to a 5 GiB heap inside a 1Gi limit and is OOM-killed. The topic and user operators run the same JDK with percentage-based defaults, and the drain-time Cruise Control takes 75% of host RAM for its heap. Each of these containers now gets JAVA_TOOL_OPTIONS with -XX:MaxRAM set to its memory limit, plus -XX:ActiveProcessorCount where the chart sets a CPU limit. The entity operators and Cruise Control have none in the chart, so a LimitRange default still leaves them counting host CPUs: Strimzi template env takes no resourceFieldRef. Every java in the container reads JAVA_TOOL_OPTIONS, Strimzi's heap probe included, so Strimzi's own heap rules and Kafka's G1 defaults apply to the real memory limit. jvmOptions would not do: Strimzi turns -XX options into KAFKA_JVM_PERFORMANCE_OPTS, and kafka-run-class.sh skips its G1 settings when that is set. Each JVM prints a "Picked up JAVA_TOOL_OPTIONS" line on start. Existing clusters roll once on upgrade, because the pod environment changes. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent ca27c0a commit 40dade5

4 files changed

Lines changed: 155 additions & 5 deletions

File tree

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
{{- /*
2+
JAVA_TOOL_OPTIONS for a Strimzi-managed container, from its resource limits.
3+
The JDK 17 in the Strimzi images cannot see the container limits on kernels
4+
built without CONFIG_MEMCG_V1 (JDK-8347811) and sizes itself from the host:
5+
Strimzi's heap probe then hands every broker up to a 5Gi heap. Every java in
6+
the container reads this variable, the probe included, so Strimzi's own
7+
heap rules work again. jvmOptions -XX would not do: Strimzi moves those into
8+
KAFKA_JVM_PERFORMANCE_OPTS, and kafka-run-class.sh drops its G1 tuning when
9+
that variable is set. ActiveProcessorCount only when the chart sets a CPU
10+
limit: a default a LimitRange adds at admission is not visible here, and
11+
Strimzi template env takes no resourceFieldRef, so such a JVM still counts
12+
the host CPUs.
13+
*/ -}}
14+
{{- define "kafka.javaToolOptions" -}}
15+
{{- $opts := list (printf "-XX:MaxRAM=%d" (include "cozy-lib.resources.toFloat" .memory | float64 | int64)) -}}
16+
{{- with .cpu -}}
17+
{{- $opts = append $opts (printf "-XX:ActiveProcessorCount=%d" (include "cozy-lib.resources.toFloat" . | float64 | ceil | int64)) -}}
18+
{{- end -}}
19+
{{- join " " $opts -}}
20+
{{- end -}}

‎packages/apps/kafka/templates/kafka.yaml‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
{{- $tlsEnabled := include "kafka.tls.enabled" . | trim -}}
1717
{{- $showExternal := .Values.external -}}
1818
{{- $kafkaVersion := include "kafka.versionMap" . -}}
19+
{{- $entityOperatorMemory := "512Mi" -}}
1920
apiVersion: kafka.strimzi.io/v1beta2
2021
kind: Kafka
2122
metadata:
@@ -75,21 +76,22 @@ spec:
7576
# The topic-operator and user-operator are JVM processes and OOMKill at
7677
# that default, leaving the entity-operator in CrashLoopBackOff so
7778
# KafkaTopics/KafkaUsers never reconcile. Set explicit resources so they
78-
# don't inherit the namespace default.
79+
# don't inherit the namespace default. JAVA_TOOL_OPTIONS: see
80+
# kafka.javaToolOptions.
7981
topicOperator:
8082
resources:
8183
requests:
8284
cpu: 100m
8385
memory: 256Mi
8486
limits:
85-
memory: 512Mi
87+
memory: {{ $entityOperatorMemory }}
8688
userOperator:
8789
resources:
8890
requests:
8991
cpu: 100m
9092
memory: 256Mi
9193
limits:
92-
memory: 512Mi
94+
memory: {{ $entityOperatorMemory }}
9395
template:
9496
# `pod` accepts metadata + a flat set of pod-spec fields
9597
# (enableServiceLinks, securityContext, affinity, …) directly.
@@ -101,3 +103,11 @@ spec:
101103
labels:
102104
policy.cozystack.io/allow-to-apiserver: "true"
103105
enableServiceLinks: false
106+
topicOperatorContainer:
107+
env:
108+
- name: JAVA_TOOL_OPTIONS
109+
value: {{ include "kafka.javaToolOptions" (dict "memory" $entityOperatorMemory) | quote }}
110+
userOperatorContainer:
111+
env:
112+
- name: JAVA_TOOL_OPTIONS
113+
value: {{ include "kafka.javaToolOptions" (dict "memory" $entityOperatorMemory) | quote }}

‎packages/apps/kafka/templates/migration-hook.yaml‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
{{- $controllerReplicas := include "kafka.controllerReplicas" . }}
2727
{{- $brokerPool := include "kafka.brokerPoolName" . }}
2828
{{- $freshBrokerPool := printf "b-%s" (include "kafka.poolHash" .) }}
29+
{{- $cruiseControlMemory := "1Gi" }}
2930
---
3031
apiVersion: v1
3132
kind: ServiceAccount
@@ -143,6 +144,14 @@ spec:
143144
roles:
144145
- broker
145146
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | nindent 18 }}
147+
# A pool template replaces spec.kafka.template whole, for every
148+
# pool here: nothing set on the Kafka CR's template reaches these
149+
# pods (KafkaPool.java at 0.45.1-rc1: https://github.com/strimzi/strimzi-kafka-operator/blob/0.45.1-rc1/cluster-operator/src/main/java/io/strimzi/operator/cluster/model/KafkaPool.java#L196-L218).
150+
template:
151+
kafkaContainer:
152+
env:
153+
- name: JAVA_TOOL_OPTIONS
154+
value: {{ include "kafka.javaToolOptions" (include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | fromYaml).limits | quote }}
146155
storage:
147156
type: jbod
148157
volumes:
@@ -187,6 +196,11 @@ spec:
187196
roles:
188197
- controller
189198
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | nindent 18 }}
199+
template:
200+
kafkaContainer:
201+
env:
202+
- name: JAVA_TOOL_OPTIONS
203+
value: {{ include "kafka.javaToolOptions" (include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | fromYaml).limits | quote }}
190204
storage:
191205
type: jbod
192206
volumes:
@@ -388,6 +402,11 @@ spec:
388402
roles:
389403
- broker
390404
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | nindent 18 }}
405+
template:
406+
kafkaContainer:
407+
env:
408+
- name: JAVA_TOOL_OPTIONS
409+
value: {{ include "kafka.javaToolOptions" (include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | fromYaml).limits | quote }}
391410
storage:
392411
type: jbod
393412
volumes:
@@ -407,9 +426,13 @@ spec:
407426
# explicit resources: Cruise Control is a JVM, and a tenant namespace
408427
# LimitRange defaults containers to 128Mi, which OOMs it before it
409428
# can produce a proposal (same reason the entity-operator carries
410-
# explicit resources in kafka.yaml).
429+
# explicit resources in kafka.yaml). JAVA_TOOL_OPTIONS: see
430+
# kafka.javaToolOptions.
411431
echo "Enabling Cruise Control for the drain..."
412-
kubectl -n "$ns" patch "$kafka" "$name" --type merge -p '{"spec":{"cruiseControl":{"resources":{"requests":{"cpu":"200m","memory":"768Mi"},"limits":{"memory":"1Gi"}}}}}'
432+
{{- $ccEnv := list (dict "name" "JAVA_TOOL_OPTIONS" "value" (include "kafka.javaToolOptions" (dict "memory" $cruiseControlMemory))) }}
433+
{{- $ccResources := dict "requests" (dict "cpu" "200m" "memory" "768Mi") "limits" (dict "memory" $cruiseControlMemory) }}
434+
{{- $ccPatch := dict "spec" (dict "cruiseControl" (dict "resources" $ccResources "template" (dict "cruiseControlContainer" (dict "env" $ccEnv)))) }}
435+
kubectl -n "$ns" patch "$kafka" "$name" --type merge -p '{{ $ccPatch | toJson }}'
413436
414437
# F3: drain all partitions off the "kafka" pool broker IDs.
415438
kafkaIds=$(kubectl -n "$ns" get "$np" kafka -o jsonpath='{.status.nodeIds}' 2>/dev/null || echo "")
Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
suite: Kafka JVMs are sized from their container limits
2+
3+
templates:
4+
- templates/migration-hook.yaml
5+
- templates/kafka.yaml
6+
7+
release:
8+
name: test-kafka
9+
namespace: tenant-test
10+
11+
tests:
12+
- it: passes the default preset limits to every node pool JVM
13+
template: templates/migration-hook.yaml
14+
documentIndex: 3
15+
asserts:
16+
- matchRegex:
17+
path: spec.template.spec.containers[0].args[0]
18+
pattern: "release-namespace: \\$\\{ns\\}\\s+spec:\\s+replicas: \\d+\\s+roles:\\s+- broker\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+(\\s+#[^\\n]*)*\\s+template:\\s+kafkaContainer:\\s+env:\\s+- name: JAVA_TOOL_OPTIONS\\s+value: \"-XX:MaxRAM=1073741824 -XX:ActiveProcessorCount=1\"\\s+storage:"
19+
- matchRegex:
20+
path: spec.template.spec.containers[0].args[0]
21+
pattern: "roles:\\s+- controller\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+(\\s+#[^\\n]*)*\\s+template:\\s+kafkaContainer:\\s+env:\\s+- name: JAVA_TOOL_OPTIONS\\s+value: \"-XX:MaxRAM=1073741824 -XX:ActiveProcessorCount=1\"\\s+storage:"
22+
- matchRegex:
23+
path: spec.template.spec.containers[0].args[0]
24+
pattern: "next-node-ids: \"\\[1000-1099\\]\"\\s+spec:\\s+replicas: \\d+\\s+roles:\\s+- broker\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+(\\s+#[^\\n]*)*\\s+template:\\s+kafkaContainer:\\s+env:\\s+- name: JAVA_TOOL_OPTIONS\\s+value: \"-XX:MaxRAM=1073741824 -XX:ActiveProcessorCount=1\"\\s+storage:"
25+
26+
# Strimzi moves jvmOptions -XX into KAFKA_JVM_PERFORMANCE_OPTS, and
27+
# kafka-run-class.sh then drops its G1 settings.
28+
- it: leaves the node pool JVM options to Strimzi and Kafka
29+
template: templates/migration-hook.yaml
30+
documentIndex: 3
31+
asserts:
32+
- notMatchRegex:
33+
path: spec.template.spec.containers[0].args[0]
34+
pattern: "jvmOptions"
35+
36+
- it: rounds a fractional CPU limit up
37+
template: templates/migration-hook.yaml
38+
documentIndex: 3
39+
set:
40+
kafka.resources:
41+
cpu: 2500m
42+
memory: 3Gi
43+
asserts:
44+
- matchRegex:
45+
path: spec.template.spec.containers[0].args[0]
46+
pattern: "roles:\\s+- controller\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+(\\s+#[^\\n]*)*\\s+template:\\s+kafkaContainer:\\s+env:\\s+- name: JAVA_TOOL_OPTIONS\\s+value: \"-XX:MaxRAM=3221225472 -XX:ActiveProcessorCount=3\"\\s+storage:"
47+
48+
- it: takes the limits, not the requests, when the allocation ratios shrink the requests
49+
template: templates/migration-hook.yaml
50+
documentIndex: 3
51+
set:
52+
_cluster:
53+
cpu-allocation-ratio: "4"
54+
memory-allocation-ratio: "2"
55+
kafka.resources:
56+
cpu: "2"
57+
memory: 2Gi
58+
asserts:
59+
- matchRegex:
60+
path: spec.template.spec.containers[0].args[0]
61+
pattern: "release-namespace: \\$\\{ns\\}\\s+spec:\\s+replicas: \\d+\\s+roles:\\s+- broker\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+(\\s+#[^\\n]*)*\\s+template:\\s+kafkaContainer:\\s+env:\\s+- name: JAVA_TOOL_OPTIONS\\s+value: \"-XX:MaxRAM=2147483648 -XX:ActiveProcessorCount=2\"\\s+storage:"
62+
- matchRegex:
63+
path: spec.template.spec.containers[0].args[0]
64+
pattern: "roles:\\s+- controller\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+(\\s+#[^\\n]*)*\\s+template:\\s+kafkaContainer:\\s+env:\\s+- name: JAVA_TOOL_OPTIONS\\s+value: \"-XX:MaxRAM=2147483648 -XX:ActiveProcessorCount=2\"\\s+storage:"
65+
- matchRegex:
66+
path: spec.template.spec.containers[0].args[0]
67+
pattern: "next-node-ids: \"\\[1000-1099\\]\"\\s+spec:\\s+replicas: \\d+\\s+roles:\\s+- broker\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+(\\s+#[^\\n]*)*\\s+template:\\s+kafkaContainer:\\s+env:\\s+- name: JAVA_TOOL_OPTIONS\\s+value: \"-XX:MaxRAM=2147483648 -XX:ActiveProcessorCount=2\"\\s+storage:"
68+
69+
# The whole patch, closing brace included: a merge patch kubectl cannot
70+
# parse would leave the drain without Cruise Control.
71+
- it: passes the drain-time Cruise Control memory limit to its JVM
72+
template: templates/migration-hook.yaml
73+
documentIndex: 3
74+
asserts:
75+
- matchRegex:
76+
path: spec.template.spec.containers[0].args[0]
77+
pattern: "\\-\\-type\\ merge\\ \\-p\\ '\\{\"spec\":\\{\"cruiseControl\":\\{\"resources\":\\{\"limits\":\\{\"memory\":\"1Gi\"\\},\"requests\":\\{\"cpu\":\"200m\",\"memory\":\"768Mi\"\\}\\},\"template\":\\{\"cruiseControlContainer\":\\{\"env\":\\[\\{\"name\":\"JAVA_TOOL_OPTIONS\",\"value\":\"\\-XX:MaxRAM=1073741824\"\\}\\]\\}\\}\\}\\}\\}'\n"
78+
79+
- it: passes the entity operator memory limit to its JVMs
80+
template: templates/kafka.yaml
81+
asserts:
82+
- equal:
83+
path: spec.entityOperator.topicOperator.resources.limits.memory
84+
value: 512Mi
85+
- equal:
86+
path: spec.entityOperator.template.topicOperatorContainer.env
87+
value:
88+
- name: JAVA_TOOL_OPTIONS
89+
value: -XX:MaxRAM=536870912
90+
- equal:
91+
path: spec.entityOperator.userOperator.resources.limits.memory
92+
value: 512Mi
93+
- equal:
94+
path: spec.entityOperator.template.userOperatorContainer.env
95+
value:
96+
- name: JAVA_TOOL_OPTIONS
97+
value: -XX:MaxRAM=536870912

0 commit comments

Comments
 (0)