Skip to content

Commit d060c1d

Browse files
authored
fix(kafka): pass the container limits to the Kafka JVMs (#4865)
## What this PR does The Kafka JVMs we ship size themselves from host memory instead of their container limits, and get OOMKilled. The Strimzi 0.45.1-rc1 images carry JDK 17.0.15, which reads `/proc/cgroups` to decide whether it runs in a container. Kernels 6.12 and newer built without `CONFIG_MEMCG_V1` no longer list the memory controller there, so the JVM turns container support off. The JDK fix is JDK-8347811, backported to 17.0.20 and 21.0.10. In CI this shows up in two places. The cluster operator gets a 25 GiB max heap inside its 512Mi limit. In 107 of the last 150 e2e reports the kernel OOM-killed a Java process at about 500M, the limit the cluster operator shares with the topic and user operators. Brokers and controllers start with a 5 GiB heap inside a 1Gi limit, because Strimzi asks the same JDK for the container memory and takes half of what it reports. This PR passes the limits to every Kafka JVM explicitly: - the cluster operator gets `-XX:MaxRAM` and `-XX:ActiveProcessorCount` from its own limits through the downward API; - brokers, controllers, the topic and user operators and the drain-time Cruise Control get `JAVA_TOOL_OPTIONS` with `-XX:MaxRAM` set to their memory limit, plus `-XX:ActiveProcessorCount` where the chart sets a CPU limit. The chart sets one only for brokers and controllers. A LimitRange default on the entity operators or Cruise Control is invisible to the chart, and Strimzi template env takes no `resourceFieldRef`, so those JVMs still count host CPUs. Every java in a container reads `JAVA_TOOL_OPTIONS`, including the heap probe Strimzi runs before starting Kafka. So Strimzi's own heap rules and Kafka's G1 defaults apply to the real memory limit, and the chart does no heap arithmetic. `jvmOptions` with `-XX` would not work here: Strimzi turns those into `KAFKA_JVM_PERFORMANCE_OPTS`, and Kafka skips its G1 settings when that is set. Each JVM now prints a "Picked up JAVA_TOOL_OPTIONS" line on start. In a kind cluster on such a kernel the operator then runs Serial GC with a heap of about 100 MiB, and peaks at 249Mi while creating a Kafka, against 440Mi before. A broker with a 1Gi limit and one CPU gets the same `-Xmx518982205` that Strimzi computes when the JDK sees the limit, with G1 and one concurrent GC thread. Existing Kafka clusters roll once on upgrade, Talos ones included, because the pod environment changes. Their heap does not change on a kernel where the JDK already sees the limit. The chart has no values key for JVM options or heap, so there is no user setting this could override. The Kafka migration hook now applies with `--openapi-patch=false`. Its container runs under a tenant LimitRange default of 128Mi, and a plain `kubectl apply` of an existing pool downloads the cluster's whole OpenAPI document, about 300Mi with the platform's CRDs, so every upgrade in such a tenant failed on an OOM-killed hook. With the flag kubectl builds the same merge patch from the last-applied annotation and peaks at about 35Mi. A larger memory limit would count against the tenant quota, and a server-side apply would stop removing fields that `kubectl create` set on the `kafka` pool. Other Jobs without resources are tracked in #4868. The operator OOMKill is what triggered the e2e failure in #4855. The other half is a Strimzi race. After a restart, the PodSet controller drops the ADD event of a PodSet that has no pods yet and only picks it up at the 5-minute resync. That one is upstream and is described in the issue. The other JVMs we ship are tracked in #4864. Talos nodes are not affected, the Talos kernel keeps `CONFIG_MEMCG_V1=y`. Generic-variant clusters on newer distribution kernels are. CI is affected even though it runs Talos, because there the Talos nodes are containers on the runner's Ubuntu 6.17 kernel. Closes #4855 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(kafka): the Strimzi cluster operator, Kafka brokers and controllers, the entity operators and the drain-time Cruise Control no longer size their JVM heap from host memory on kernels built without cgroup v1 memory support, which got them OOMKilled. The Kafka migration hook no longer runs out of memory on upgrade in a tenant with resource quotas. Existing Kafka clusters roll once on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Kafka components now size JVM memory and processor settings based on their configured resource limits, including during migrations. - The Kafka cluster operator now receives container resource limits for JVM sizing. - **Bug Fixes** - Kafka migration operations avoid OpenAPI patch lookups that could use excessive memory. - **Tests** - Added coverage for JVM sizing, resource limits, and migration behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 1f74ad4 + 6ba440f commit d060c1d

9 files changed

Lines changed: 234 additions & 12 deletions

File tree

‎hack/kafka-migration-hook.bats‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ case "$args" in
6060
[ -n "${STUB_CREATE_FAIL:-}" ] && { echo "AlreadyExists" >&2; exit 1; }
6161
exit 0
6262
;;
63-
*"apply -f -"*)
63+
*"apply "*"-f -"*)
6464
cat >> "${KUBECTL_LOG:-/dev/null}"
6565
exit 0
6666
;;
@@ -206,7 +206,7 @@ run_hook() {
206206

207207
[ "$rc" != 0 ]
208208
grep -qF 'already belongs to cluster' "$tmp/out"
209-
if grep -qE 'apply -f -|create -f -|kraft=|node-pools=' "$tmp/kubectl.log"; then
209+
if grep -qE 'apply .*-f -|create -f -|kraft=|node-pools=' "$tmp/kubectl.log"; then
210210
echo "FAIL: a foreign-owned kafka pool did not abort before mutating"
211211
cat "$tmp/kubectl.log"
212212
return 1
@@ -223,7 +223,7 @@ run_hook() {
223223

224224
[ "$rc" != 0 ]
225225
grep -qF 'owner label failed' "$tmp/out"
226-
if grep -qE 'apply -f -|create -f -|kraft=|node-pools=' "$tmp/kubectl.log"; then
226+
if grep -qE 'apply .*-f -|create -f -|kraft=|node-pools=' "$tmp/kubectl.log"; then
227227
echo "FAIL: proceeded to mutate the cluster after a failed owner read"
228228
cat "$tmp/kubectl.log"
229229
return 1
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
{{- /*
2+
JAVA_TOOL_OPTIONS for a Strimzi-managed container, from its resource limits.
3+
The JDK 17 in the Strimzi images cannot see the container limits on kernels
4+
built without CONFIG_MEMCG_V1 (JDK-8347811) and sizes itself from the host:
5+
Strimzi's heap probe then hands every broker up to a 5Gi heap. Every java in
6+
the container reads this variable, the probe included, so Strimzi's own
7+
heap rules work again. jvmOptions -XX would not do: Strimzi moves those into
8+
KAFKA_JVM_PERFORMANCE_OPTS, and kafka-run-class.sh drops its G1 tuning when
9+
that variable is set. ActiveProcessorCount only when the chart sets a CPU
10+
limit: a default a LimitRange adds at admission is not visible here, and
11+
Strimzi template env takes no resourceFieldRef, so such a JVM still counts
12+
the host CPUs.
13+
*/ -}}
14+
{{- define "kafka.javaToolOptions" -}}
15+
{{- $opts := list (printf "-XX:MaxRAM=%d" (include "cozy-lib.resources.toFloat" .memory | float64 | int64)) -}}
16+
{{- with .cpu -}}
17+
{{- $opts = append $opts (printf "-XX:ActiveProcessorCount=%d" (include "cozy-lib.resources.toFloat" . | float64 | ceil | int64)) -}}
18+
{{- end -}}
19+
{{- join " " $opts -}}
20+
{{- end -}}

‎packages/apps/kafka/templates/kafka.yaml‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
{{- $tlsEnabled := include "kafka.tls.enabled" . | trim -}}
1717
{{- $showExternal := .Values.external -}}
1818
{{- $kafkaVersion := include "kafka.versionMap" . -}}
19+
{{- $entityOperatorMemory := "512Mi" -}}
1920
apiVersion: kafka.strimzi.io/v1beta2
2021
kind: Kafka
2122
metadata:
@@ -75,21 +76,22 @@ spec:
7576
# The topic-operator and user-operator are JVM processes and OOMKill at
7677
# that default, leaving the entity-operator in CrashLoopBackOff so
7778
# KafkaTopics/KafkaUsers never reconcile. Set explicit resources so they
78-
# don't inherit the namespace default.
79+
# don't inherit the namespace default. JAVA_TOOL_OPTIONS: see
80+
# kafka.javaToolOptions.
7981
topicOperator:
8082
resources:
8183
requests:
8284
cpu: 100m
8385
memory: 256Mi
8486
limits:
85-
memory: 512Mi
87+
memory: {{ $entityOperatorMemory }}
8688
userOperator:
8789
resources:
8890
requests:
8991
cpu: 100m
9092
memory: 256Mi
9193
limits:
92-
memory: 512Mi
94+
memory: {{ $entityOperatorMemory }}
9395
template:
9496
# `pod` accepts metadata + a flat set of pod-spec fields
9597
# (enableServiceLinks, securityContext, affinity, …) directly.
@@ -101,3 +103,11 @@ spec:
101103
labels:
102104
policy.cozystack.io/allow-to-apiserver: "true"
103105
enableServiceLinks: false
106+
topicOperatorContainer:
107+
env:
108+
- name: JAVA_TOOL_OPTIONS
109+
value: {{ include "kafka.javaToolOptions" (dict "memory" $entityOperatorMemory) | quote }}
110+
userOperatorContainer:
111+
env:
112+
- name: JAVA_TOOL_OPTIONS
113+
value: {{ include "kafka.javaToolOptions" (dict "memory" $entityOperatorMemory) | quote }}

‎packages/apps/kafka/templates/migration-hook.yaml‎

Lines changed: 38 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
{{- $controllerReplicas := include "kafka.controllerReplicas" . }}
2727
{{- $brokerPool := include "kafka.brokerPoolName" . }}
2828
{{- $freshBrokerPool := printf "b-%s" (include "kafka.poolHash" .) }}
29+
{{- $cruiseControlMemory := "1Gi" }}
2930
---
3031
apiVersion: v1
3132
kind: ServiceAccount
@@ -121,6 +122,15 @@ spec:
121122
np=kafkanodepools.kafka.strimzi.io
122123
kr=kafkarebalances.kafka.strimzi.io
123124
125+
# Every apply here passes --openapi-patch=false. Without it, applying
126+
# an object that already exists makes kubectl download the cluster's
127+
# whole OpenAPI v2 document only to learn that a custom resource
128+
# takes a JSON merge patch: about 300Mi with the platform's CRDs,
129+
# while this container runs under a tenant LimitRange default of
130+
# 128Mi. With it kubectl builds the same merge patch from the
131+
# last-applied annotation. Not --server-side: fields that
132+
# "kubectl create" set on the "kafka" pool stay co-owned by that
133+
# manager, so a server-side apply would never remove them.
124134
# ---- Node-pool manifests (broker pool reused for the create-CAS) ----
125135
broker_manifest=$(cat <<MANIFEST
126136
apiVersion: kafka.strimzi.io/v1beta2
@@ -143,6 +153,14 @@ spec:
143153
roles:
144154
- broker
145155
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | nindent 18 }}
156+
# A pool template replaces spec.kafka.template whole, for every
157+
# pool here: nothing set on the Kafka CR's template reaches these
158+
# pods (KafkaPool.java at 0.45.1-rc1: https://github.com/strimzi/strimzi-kafka-operator/blob/0.45.1-rc1/cluster-operator/src/main/java/io/strimzi/operator/cluster/model/KafkaPool.java#L196-L218).
159+
template:
160+
kafkaContainer:
161+
env:
162+
- name: JAVA_TOOL_OPTIONS
163+
value: {{ include "kafka.javaToolOptions" (include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | fromYaml).limits | quote }}
146164
storage:
147165
type: jbod
148166
volumes:
@@ -187,6 +205,11 @@ spec:
187205
roles:
188206
- controller
189207
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | nindent 18 }}
208+
template:
209+
kafkaContainer:
210+
env:
211+
- name: JAVA_TOOL_OPTIONS
212+
value: {{ include "kafka.javaToolOptions" (include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | fromYaml).limits | quote }}
190213
storage:
191214
type: jbod
192215
volumes:
@@ -242,10 +265,10 @@ spec:
242265
fi
243266
fi
244267
echo "Ensuring broker KafkaNodePool \"$brokerPool\" exists..."
245-
echo "$broker_manifest" | kubectl apply -f -
268+
echo "$broker_manifest" | kubectl apply --openapi-patch=false -f -
246269
247270
echo "Ensuring controller KafkaNodePool exists..."
248-
echo "$controller_manifest" | kubectl apply -f -
271+
echo "$controller_manifest" | kubectl apply --openapi-patch=false -f -
249272
250273
# ---- Decide what to do from the Kafka CR ----
251274
# Fail closed. --ignore-not-found makes kubectl exit 0 with empty
@@ -370,7 +393,7 @@ spec:
370393
# F1: stand up this cluster's own release-hashed broker pool with
371394
# node IDs pinned high, clear of the existing broker/controller IDs.
372395
echo "Creating target broker pool \"$freshBrokerPool\"..."
373-
cat <<POOL | kubectl apply -f -
396+
cat <<POOL | kubectl apply --openapi-patch=false -f -
374397
apiVersion: kafka.strimzi.io/v1beta2
375398
kind: KafkaNodePool
376399
metadata:
@@ -388,6 +411,11 @@ spec:
388411
roles:
389412
- broker
390413
resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | nindent 18 }}
414+
template:
415+
kafkaContainer:
416+
env:
417+
- name: JAVA_TOOL_OPTIONS
418+
value: {{ include "kafka.javaToolOptions" (include "cozy-lib.resources.defaultingSanitize" (list .Values.kafka.resourcesPreset .Values.kafka.resources $) | fromYaml).limits | quote }}
391419
storage:
392420
type: jbod
393421
volumes:
@@ -407,9 +435,13 @@ spec:
407435
# explicit resources: Cruise Control is a JVM, and a tenant namespace
408436
# LimitRange defaults containers to 128Mi, which OOMs it before it
409437
# can produce a proposal (same reason the entity-operator carries
410-
# explicit resources in kafka.yaml).
438+
# explicit resources in kafka.yaml). JAVA_TOOL_OPTIONS: see
439+
# kafka.javaToolOptions.
411440
echo "Enabling Cruise Control for the drain..."
412-
kubectl -n "$ns" patch "$kafka" "$name" --type merge -p '{"spec":{"cruiseControl":{"resources":{"requests":{"cpu":"200m","memory":"768Mi"},"limits":{"memory":"1Gi"}}}}}'
441+
{{- $ccEnv := list (dict "name" "JAVA_TOOL_OPTIONS" "value" (include "kafka.javaToolOptions" (dict "memory" $cruiseControlMemory))) }}
442+
{{- $ccResources := dict "requests" (dict "cpu" "200m" "memory" "768Mi") "limits" (dict "memory" $cruiseControlMemory) }}
443+
{{- $ccPatch := dict "spec" (dict "cruiseControl" (dict "resources" $ccResources "template" (dict "cruiseControlContainer" (dict "env" $ccEnv)))) }}
444+
kubectl -n "$ns" patch "$kafka" "$name" --type merge -p '{{ $ccPatch | toJson }}'
413445
414446
# F3: drain all partitions off the "kafka" pool broker IDs.
415447
kafkaIds=$(kubectl -n "$ns" get "$np" kafka -o jsonpath='{.status.nodeIds}' 2>/dev/null || echo "")
@@ -418,7 +450,7 @@ spec:
418450
exit 1
419451
fi
420452
echo "Draining brokers $kafkaIds via Cruise Control rebalance..."
421-
cat <<REB | kubectl apply -f -
453+
cat <<REB | kubectl apply --openapi-patch=false -f -
422454
apiVersion: kafka.strimzi.io/v1beta2
423455
kind: KafkaRebalance
424456
metadata:
Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
suite: Kafka JVMs are sized from their container limits
2+
3+
templates:
4+
- templates/migration-hook.yaml
5+
- templates/kafka.yaml
6+
7+
release:
8+
name: test-kafka
9+
namespace: tenant-test
10+
11+
tests:
12+
- it: passes the default preset limits to every node pool JVM
13+
template: templates/migration-hook.yaml
14+
documentIndex: 3
15+
asserts:
16+
- matchRegex:
17+
path: spec.template.spec.containers[0].args[0]
18+
pattern: "release-namespace: \\$\\{ns\\}\\s+spec:\\s+replicas: \\d+\\s+roles:\\s+- broker\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+(\\s+#[^\\n]*)*\\s+template:\\s+kafkaContainer:\\s+env:\\s+- name: JAVA_TOOL_OPTIONS\\s+value: \"-XX:MaxRAM=1073741824 -XX:ActiveProcessorCount=1\"\\s+storage:"
19+
- matchRegex:
20+
path: spec.template.spec.containers[0].args[0]
21+
pattern: "roles:\\s+- controller\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+(\\s+#[^\\n]*)*\\s+template:\\s+kafkaContainer:\\s+env:\\s+- name: JAVA_TOOL_OPTIONS\\s+value: \"-XX:MaxRAM=1073741824 -XX:ActiveProcessorCount=1\"\\s+storage:"
22+
- matchRegex:
23+
path: spec.template.spec.containers[0].args[0]
24+
pattern: "next-node-ids: \"\\[1000-1099\\]\"\\s+spec:\\s+replicas: \\d+\\s+roles:\\s+- broker\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+(\\s+#[^\\n]*)*\\s+template:\\s+kafkaContainer:\\s+env:\\s+- name: JAVA_TOOL_OPTIONS\\s+value: \"-XX:MaxRAM=1073741824 -XX:ActiveProcessorCount=1\"\\s+storage:"
25+
26+
# Strimzi moves jvmOptions -XX into KAFKA_JVM_PERFORMANCE_OPTS, and
27+
# kafka-run-class.sh then drops its G1 settings.
28+
- it: leaves the node pool JVM options to Strimzi and Kafka
29+
template: templates/migration-hook.yaml
30+
documentIndex: 3
31+
asserts:
32+
- notMatchRegex:
33+
path: spec.template.spec.containers[0].args[0]
34+
pattern: "jvmOptions"
35+
36+
- it: rounds a fractional CPU limit up
37+
template: templates/migration-hook.yaml
38+
documentIndex: 3
39+
set:
40+
kafka.resources:
41+
cpu: 2500m
42+
memory: 3Gi
43+
asserts:
44+
- matchRegex:
45+
path: spec.template.spec.containers[0].args[0]
46+
pattern: "roles:\\s+- controller\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+(\\s+#[^\\n]*)*\\s+template:\\s+kafkaContainer:\\s+env:\\s+- name: JAVA_TOOL_OPTIONS\\s+value: \"-XX:MaxRAM=3221225472 -XX:ActiveProcessorCount=3\"\\s+storage:"
47+
48+
- it: takes the limits, not the requests, when the allocation ratios shrink the requests
49+
template: templates/migration-hook.yaml
50+
documentIndex: 3
51+
set:
52+
_cluster:
53+
cpu-allocation-ratio: "4"
54+
memory-allocation-ratio: "2"
55+
kafka.resources:
56+
cpu: "2"
57+
memory: 2Gi
58+
asserts:
59+
- matchRegex:
60+
path: spec.template.spec.containers[0].args[0]
61+
pattern: "release-namespace: \\$\\{ns\\}\\s+spec:\\s+replicas: \\d+\\s+roles:\\s+- broker\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+(\\s+#[^\\n]*)*\\s+template:\\s+kafkaContainer:\\s+env:\\s+- name: JAVA_TOOL_OPTIONS\\s+value: \"-XX:MaxRAM=2147483648 -XX:ActiveProcessorCount=2\"\\s+storage:"
62+
- matchRegex:
63+
path: spec.template.spec.containers[0].args[0]
64+
pattern: "roles:\\s+- controller\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+(\\s+#[^\\n]*)*\\s+template:\\s+kafkaContainer:\\s+env:\\s+- name: JAVA_TOOL_OPTIONS\\s+value: \"-XX:MaxRAM=2147483648 -XX:ActiveProcessorCount=2\"\\s+storage:"
65+
- matchRegex:
66+
path: spec.template.spec.containers[0].args[0]
67+
pattern: "next-node-ids: \"\\[1000-1099\\]\"\\s+spec:\\s+replicas: \\d+\\s+roles:\\s+- broker\\s+resources:\\s+limits:(\\s+\\S+: \\S+)+\\s+requests:(\\s+\\S+: \\S+)+(\\s+#[^\\n]*)*\\s+template:\\s+kafkaContainer:\\s+env:\\s+- name: JAVA_TOOL_OPTIONS\\s+value: \"-XX:MaxRAM=2147483648 -XX:ActiveProcessorCount=2\"\\s+storage:"
68+
69+
# The whole patch, closing brace included: a merge patch kubectl cannot
70+
# parse fails the hook before the drain starts.
71+
- it: passes the drain-time Cruise Control memory limit to its JVM
72+
template: templates/migration-hook.yaml
73+
documentIndex: 3
74+
asserts:
75+
- matchRegex:
76+
path: spec.template.spec.containers[0].args[0]
77+
pattern: "\\-\\-type\\ merge\\ \\-p\\ '\\{\"spec\":\\{\"cruiseControl\":\\{\"resources\":\\{\"limits\":\\{\"memory\":\"1Gi\"\\},\"requests\":\\{\"cpu\":\"200m\",\"memory\":\"768Mi\"\\}\\},\"template\":\\{\"cruiseControlContainer\":\\{\"env\":\\[\\{\"name\":\"JAVA_TOOL_OPTIONS\",\"value\":\"\\-XX:MaxRAM=1073741824\"\\}\\]\\}\\}\\}\\}\\}'\n"
78+
79+
- it: passes the entity operator memory limit to its JVMs
80+
template: templates/kafka.yaml
81+
asserts:
82+
- equal:
83+
path: spec.entityOperator.topicOperator.resources.limits.memory
84+
value: 512Mi
85+
- equal:
86+
path: spec.entityOperator.template.topicOperatorContainer.env
87+
value:
88+
- name: JAVA_TOOL_OPTIONS
89+
value: -XX:MaxRAM=536870912
90+
- equal:
91+
path: spec.entityOperator.userOperator.resources.limits.memory
92+
value: 512Mi
93+
- equal:
94+
path: spec.entityOperator.template.userOperatorContainer.env
95+
value:
96+
- name: JAVA_TOOL_OPTIONS
97+
value: -XX:MaxRAM=536870912

‎packages/apps/kafka/tests/migration-hook_test.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -447,3 +447,16 @@ tests:
447447
path: spec.template.spec.containers[0].args[0]
448448
pattern: "roles:\\s+- controller[\\s\\S]*?resources:\\s+limits:"
449449
documentIndex: 3
450+
451+
- it: applies without the OpenAPI patch lookup, so an upgrade fits the tenant LimitRange default
452+
release:
453+
name: test-kafka
454+
namespace: tenant-test
455+
documentIndex: 3
456+
asserts:
457+
- notMatchRegex:
458+
path: spec.template.spec.containers[0].args[0]
459+
pattern: "kubectl[^|\\n]* apply (-f|--filename|--server-side)"
460+
- matchRegex:
461+
path: spec.template.spec.containers[0].args[0]
462+
pattern: "echo \"\\$broker_manifest\" \\| kubectl apply --openapi-patch=false -f -\\n[\\s\\S]*echo \"\\$controller_manifest\" \\| kubectl apply --openapi-patch=false -f -\\n[\\s\\S]*<<POOL \\| kubectl apply --openapi-patch=false -f -\\n[\\s\\S]*<<REB \\| kubectl apply --openapi-patch=false -f -\\n"

‎packages/system/kafka-operator/Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,9 @@ export NAMESPACE=cozy-$(NAME)
33

44
include ../../../hack/package.mk
55

6+
test:
7+
helm unittest .
8+
69
update:
710
rm -rf charts
811
helm pull oci://quay.io/strimzi-helm/strimzi-kafka-operator --untar --untardir charts --version 0.45.1-rc1
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
suite: cluster operator JVM is sized from the container limits
2+
release:
3+
name: kafka-operator
4+
namespace: cozy-kafka-operator
5+
templates:
6+
- charts/strimzi-kafka-operator/templates/060-Deployment-strimzi-cluster-operator.yaml
7+
tests:
8+
- it: passes the container limits to the JVM, which cannot read them on kernels without cgroup v1 controllers
9+
asserts:
10+
# $(VAR) expands only from entries earlier in the list.
11+
- equal:
12+
path: spec.template.spec.containers[0].env[19]
13+
value:
14+
name: CONTAINER_MEMORY_LIMIT
15+
valueFrom:
16+
resourceFieldRef:
17+
resource: limits.memory
18+
- equal:
19+
path: spec.template.spec.containers[0].env[20]
20+
value:
21+
name: CONTAINER_CPU_LIMIT
22+
valueFrom:
23+
resourceFieldRef:
24+
resource: limits.cpu
25+
- equal:
26+
path: spec.template.spec.containers[0].env[21]
27+
value:
28+
name: JAVA_OPTS
29+
value: -XX:MaxRAM=$(CONTAINER_MEMORY_LIMIT) -XX:ActiveProcessorCount=$(CONTAINER_CPU_LIMIT)
30+
- exists:
31+
path: spec.template.spec.containers[0].resources.limits.cpu

‎packages/system/kafka-operator/values.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,19 @@ strimzi-kafka-operator:
55
resources:
66
limits:
77
memory: 512Mi
8+
# The JDK 17 in the operator image decides whether to read the container
9+
# limits by looking at /proc/cgroups, which loses its memory line on kernels
10+
# built without CONFIG_MEMCG_V1 (6.12+). The JVM then sizes its heap from
11+
# host RAM and the container is OOMKilled. Drop this once the image ships a
12+
# JDK with JDK-8347811 (17.0.20, 21.0.10).
13+
extraEnvs:
14+
- name: CONTAINER_MEMORY_LIMIT
15+
valueFrom:
16+
resourceFieldRef:
17+
resource: limits.memory
18+
- name: CONTAINER_CPU_LIMIT
19+
valueFrom:
20+
resourceFieldRef:
21+
resource: limits.cpu
22+
- name: JAVA_OPTS
23+
value: -XX:MaxRAM=$(CONTAINER_MEMORY_LIMIT) -XX:ActiveProcessorCount=$(CONTAINER_CPU_LIMIT)

0 commit comments

Comments
 (0)