Skip to content

[Backport release-1.7] fix(kafka): pass the container limits to the Kafka JVMs - #5019

Merged
myasnikovdaniil merged 3 commits into
release-1.7from
backport-4865-to-release-1.7
Oct 21, 2026
Merged

myasnikovdaniil merged 3 commits into
release-1.7from
backport-4865-to-release-1.7

Conversation

@github-actions

Copy link
Copy Markdown

Description

Backport of #4865 to release-1.7.

The JDK 17 in the Strimzi 0.45.1-rc1 operator image decides whether to
read the container limits by looking at /proc/cgroups. Kernels 6.12 and
newer built without CONFIG_MEMCG_V1 no longer list the memory controller
there, so the JVM turns container support off and takes host RAM and
host CPUs as its budget: on a 128 GiB node it picks G1 with a 25 GiB max
heap and a 2 GiB initial heap inside a 512Mi limit, and the kernel
OOM-kills the operator under load. A restart at the wrong moment leaves
a freshly created StrimziPodSet without pods for up to the five-minute
informer resync, which is how a KRaft controller pool ends up never
starting within an e2e timeout.

The limits now reach the JVM through the downward API as -XX:MaxRAM and
-XX:ActiveProcessorCount, so they follow any later change to the
container resources. In a test cluster the operator then picks Serial
GC and a heap of about 100 MiB. The upstream JDK fix is JDK-8347811,
backported to 17.0.20 and 21.0.10.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
(cherry picked from commit b255dc4)
Strimzi sizes a broker heap by asking the image's JDK 17 what heap it
would pick at MaxRAMPercentage=50, capped at 5 GiB.
On kernels built without CONFIG_MEMCG_V1 that JDK cannot see the limit
and reports host RAM, so every broker and controller starts with up to
a 5 GiB heap inside a 1Gi limit and is OOM-killed. The topic and user
operators run the same JDK with percentage-based defaults, and the
drain-time Cruise Control takes 75% of host RAM for its heap.

Each of these containers now gets JAVA_TOOL_OPTIONS with -XX:MaxRAM set
to its memory limit, plus -XX:ActiveProcessorCount where the chart sets
a CPU limit. The entity operators and Cruise Control have none in the
chart, so a LimitRange default still leaves them counting host CPUs:
Strimzi template env takes no resourceFieldRef. Every java in the
container reads JAVA_TOOL_OPTIONS, Strimzi's heap probe included, so
Strimzi's own heap rules and Kafka's G1 defaults apply to the real
memory limit. jvmOptions would not do: Strimzi turns -XX options into
KAFKA_JVM_PERFORMANCE_OPTS, and kafka-run-class.sh skips its G1 settings
when that is set. Each JVM prints a "Picked up JAVA_TOOL_OPTIONS" line
on start. Existing clusters roll once on upgrade, because the pod
environment changes.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
(cherry picked from commit 479d17a)
The pre-install/pre-upgrade hook applies the node pools and the drain
KafkaRebalance with kubectl, and its container has no resources of its
own, so in a tenant with resource quotas it runs under the LimitRange
default of 128Mi. When it applies an object that already exists,
kubectl downloads the cluster's whole OpenAPI v2 document only to
decide that a custom resource takes a JSON merge patch: with the 374
CRDs of an e2e cluster that peaks at about 300Mi, so every upgrade of a
Kafka in such a tenant OOM-kills the hook and fails with "pre-upgrade
hooks failed".

--openapi-patch=false skips that lookup and builds the same three-way
merge patch from the last-applied annotation, peaking at about 35Mi on
the same cluster. A server-side apply would be as small but changes
what gets removed: the "kafka" pool is created with "kubectl create",
whose manager keeps co-owning every field it set, so a later
server-side apply leaves those fields in place when the chart drops
them. A larger memory limit would charge every upgrade against the
tenant quota instead.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
(cherry picked from commit 6ba440f)
@coderabbitai

coderabbitai Bot commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: cozystack/cozystack/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: b5a49558-1ab7-4243-ba40-299084bed6fe

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@myasnikovdaniil
myasnikovdaniil merged commit 2ccca66 into release-1.7 Oct 21, 2026
2 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the backport-4865-to-release-1.7 branch October 21, 2026 09:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants