You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d060c1d
Browse filesBrowse the repository at this point in the historyBrowse files
fix(kafka): pass the container limits to the Kafka JVMs (#4865)
## What this PR does
The Kafka JVMs we ship size themselves from host memory instead of their
container limits, and get OOMKilled. The Strimzi 0.45.1-rc1 images carry
JDK 17.0.15, which reads `/proc/cgroups` to decide whether it runs in a
container. Kernels 6.12 and newer built without `CONFIG_MEMCG_V1` no
longer list the memory controller there, so the JVM turns container
support off. The JDK fix is JDK-8347811, backported to 17.0.20 and
21.0.10.
In CI this shows up in two places. The cluster operator gets a 25 GiB
max heap inside its 512Mi limit. In 107 of the last 150 e2e reports the
kernel OOM-killed a Java process at about 500M, the limit the cluster
operator shares with the topic and user operators. Brokers and
controllers start with a 5 GiB heap inside a 1Gi limit, because Strimzi
asks the same JDK for the container memory and takes half of what it
reports.
This PR passes the limits to every Kafka JVM explicitly:
- the cluster operator gets `-XX:MaxRAM` and `-XX:ActiveProcessorCount`
from its own limits through the downward API;
- brokers, controllers, the topic and user operators and the drain-time
Cruise Control get `JAVA_TOOL_OPTIONS` with `-XX:MaxRAM` set to their
memory limit, plus `-XX:ActiveProcessorCount` where the chart sets a CPU
limit. The chart sets one only for brokers and controllers. A LimitRange
default on the entity operators or Cruise Control is invisible to the
chart, and Strimzi template env takes no `resourceFieldRef`, so those
JVMs still count host CPUs.
Every java in a container reads `JAVA_TOOL_OPTIONS`, including the heap
probe Strimzi runs before starting Kafka. So Strimzi's own heap rules
and Kafka's G1 defaults apply to the real memory limit, and the chart
does no heap arithmetic. `jvmOptions` with `-XX` would not work here:
Strimzi turns those into `KAFKA_JVM_PERFORMANCE_OPTS`, and Kafka skips
its G1 settings when that is set. Each JVM now prints a "Picked up
JAVA_TOOL_OPTIONS" line on start.
In a kind cluster on such a kernel the operator then runs Serial GC with
a heap of about 100 MiB, and peaks at 249Mi while creating a Kafka,
against 440Mi before. A broker with a 1Gi limit and one CPU gets the
same `-Xmx518982205` that Strimzi computes when the JDK sees the limit,
with G1 and one concurrent GC thread.
Existing Kafka clusters roll once on upgrade, Talos ones included,
because the pod environment changes. Their heap does not change on a
kernel where the JDK already sees the limit.
The chart has no values key for JVM options or heap, so there is no user
setting this could override.
The Kafka migration hook now applies with `--openapi-patch=false`. Its
container runs under a tenant LimitRange default of 128Mi, and a plain
`kubectl apply` of an existing pool downloads the cluster's whole
OpenAPI document, about 300Mi with the platform's CRDs, so every upgrade
in such a tenant failed on an OOM-killed hook. With the flag kubectl
builds the same merge patch from the last-applied annotation and peaks
at about 35Mi. A larger memory limit would count against the tenant
quota, and a server-side apply would stop removing fields that `kubectl
create` set on the `kafka` pool. Other Jobs without resources are
tracked in #4868.
The operator OOMKill is what triggered the e2e failure in #4855. The
other half is a Strimzi race. After a restart, the PodSet controller
drops the ADD event of a PodSet that has no pods yet and only picks it
up at the 5-minute resync. That one is upstream and is described in the
issue. The other JVMs we ship are tracked in #4864.
Talos nodes are not affected, the Talos kernel keeps
`CONFIG_MEMCG_V1=y`. Generic-variant clusters on newer distribution
kernels are. CI is affected even though it runs Talos, because there the
Talos nodes are containers on the runner's Ubuntu 6.17 kernel.
Closes#4855
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(kafka): the Strimzi cluster operator, Kafka brokers and controllers, the entity operators and the drain-time Cruise Control no longer size their JVM heap from host memory on kernels built without cgroup v1 memory support, which got them OOMKilled. The Kafka migration hook no longer runs out of memory on upgrade in a tenant with resource quotas. Existing Kafka clusters roll once on upgrade.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Kafka components now size JVM memory and processor settings based on
their configured resource limits, including during migrations.
- The Kafka cluster operator now receives container resource limits for
JVM sizing.
- **Bug Fixes**
- Kafka migration operations avoid OpenAPI patch lookups that could use
excessive memory.
- **Tests**
- Added coverage for JVM sizing, resource limits, and migration
behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
0 commit comments