Skip to content

Commit fc38a54

Browse files
lexfreigithub-actions[bot]
authored andcommitted
fix(kafka): skip the OpenAPI patch lookup in the migration hook
The pre-install/pre-upgrade hook applies the node pools and the drain KafkaRebalance with kubectl, and its container has no resources of its own, so in a tenant with resource quotas it runs under the LimitRange default of 128Mi. When it applies an object that already exists, kubectl downloads the cluster's whole OpenAPI v2 document only to decide that a custom resource takes a JSON merge patch: with the 374 CRDs of an e2e cluster that peaks at about 300Mi, so every upgrade of a Kafka in such a tenant OOM-kills the hook and fails with "pre-upgrade hooks failed". --openapi-patch=false skips that lookup and builds the same three-way merge patch from the last-applied annotation, peaking at about 35Mi on the same cluster. A server-side apply would be as small but changes what gets removed: the "kafka" pool is created with "kubectl create", whose manager keeps co-owning every field it set, so a later server-side apply leaves those fields in place when the chart drops them. A larger memory limit would charge every upgrade against the tenant quota instead. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]> (cherry picked from commit 6ba440f)
1 parent f6f5c7a commit fc38a54

3 files changed

Lines changed: 29 additions & 7 deletions

File tree

‎hack/kafka-migration-hook.bats‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ case "$args" in
6060
[ -n "${STUB_CREATE_FAIL:-}" ] && { echo "AlreadyExists" >&2; exit 1; }
6161
exit 0
6262
;;
63-
*"apply -f -"*)
63+
*"apply "*"-f -"*)
6464
cat >> "${KUBECTL_LOG:-/dev/null}"
6565
exit 0
6666
;;
@@ -206,7 +206,7 @@ run_hook() {
206206

207207
[ "$rc" != 0 ]
208208
grep -qF 'already belongs to cluster' "$tmp/out"
209-
if grep -qE 'apply -f -|create -f -|kraft=|node-pools=' "$tmp/kubectl.log"; then
209+
if grep -qE 'apply .*-f -|create -f -|kraft=|node-pools=' "$tmp/kubectl.log"; then
210210
echo "FAIL: a foreign-owned kafka pool did not abort before mutating"
211211
cat "$tmp/kubectl.log"
212212
return 1
@@ -223,7 +223,7 @@ run_hook() {
223223

224224
[ "$rc" != 0 ]
225225
grep -qF 'owner label failed' "$tmp/out"
226-
if grep -qE 'apply -f -|create -f -|kraft=|node-pools=' "$tmp/kubectl.log"; then
226+
if grep -qE 'apply .*-f -|create -f -|kraft=|node-pools=' "$tmp/kubectl.log"; then
227227
echo "FAIL: proceeded to mutate the cluster after a failed owner read"
228228
cat "$tmp/kubectl.log"
229229
return 1

‎packages/apps/kafka/templates/migration-hook.yaml‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,15 @@ spec:
122122
np=kafkanodepools.kafka.strimzi.io
123123
kr=kafkarebalances.kafka.strimzi.io
124124
125+
# Every apply here passes --openapi-patch=false. Without it, applying
126+
# an object that already exists makes kubectl download the cluster's
127+
# whole OpenAPI v2 document only to learn that a custom resource
128+
# takes a JSON merge patch: about 300Mi with the platform's CRDs,
129+
# while this container runs under a tenant LimitRange default of
130+
# 128Mi. With it kubectl builds the same merge patch from the
131+
# last-applied annotation. Not --server-side: fields that
132+
# "kubectl create" set on the "kafka" pool stay co-owned by that
133+
# manager, so a server-side apply would never remove them.
125134
# ---- Node-pool manifests (broker pool reused for the create-CAS) ----
126135
broker_manifest=$(cat <<MANIFEST
127136
apiVersion: kafka.strimzi.io/v1beta2
@@ -256,10 +265,10 @@ spec:
256265
fi
257266
fi
258267
echo "Ensuring broker KafkaNodePool \"$brokerPool\" exists..."
259-
echo "$broker_manifest" | kubectl apply -f -
268+
echo "$broker_manifest" | kubectl apply --openapi-patch=false -f -
260269
261270
echo "Ensuring controller KafkaNodePool exists..."
262-
echo "$controller_manifest" | kubectl apply -f -
271+
echo "$controller_manifest" | kubectl apply --openapi-patch=false -f -
263272
264273
# ---- Decide what to do from the Kafka CR ----
265274
# Fail closed. --ignore-not-found makes kubectl exit 0 with empty
@@ -384,7 +393,7 @@ spec:
384393
# F1: stand up this cluster's own release-hashed broker pool with
385394
# node IDs pinned high, clear of the existing broker/controller IDs.
386395
echo "Creating target broker pool \"$freshBrokerPool\"..."
387-
cat <<POOL | kubectl apply -f -
396+
cat <<POOL | kubectl apply --openapi-patch=false -f -
388397
apiVersion: kafka.strimzi.io/v1beta2
389398
kind: KafkaNodePool
390399
metadata:
@@ -441,7 +450,7 @@ spec:
441450
exit 1
442451
fi
443452
echo "Draining brokers $kafkaIds via Cruise Control rebalance..."
444-
cat <<REB | kubectl apply -f -
453+
cat <<REB | kubectl apply --openapi-patch=false -f -
445454
apiVersion: kafka.strimzi.io/v1beta2
446455
kind: KafkaRebalance
447456
metadata:

‎packages/apps/kafka/tests/migration-hook_test.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -447,3 +447,16 @@ tests:
447447
path: spec.template.spec.containers[0].args[0]
448448
pattern: "roles:\\s+- controller[\\s\\S]*?resources:\\s+limits:"
449449
documentIndex: 3
450+
451+
- it: applies without the OpenAPI patch lookup, so an upgrade fits the tenant LimitRange default
452+
release:
453+
name: test-kafka
454+
namespace: tenant-test
455+
documentIndex: 3
456+
asserts:
457+
- notMatchRegex:
458+
path: spec.template.spec.containers[0].args[0]
459+
pattern: "kubectl[^|\\n]* apply (-f|--filename|--server-side)"
460+
- matchRegex:
461+
path: spec.template.spec.containers[0].args[0]
462+
pattern: "echo \"\\$broker_manifest\" \\| kubectl apply --openapi-patch=false -f -\\n[\\s\\S]*echo \"\\$controller_manifest\" \\| kubectl apply --openapi-patch=false -f -\\n[\\s\\S]*<<POOL \\| kubectl apply --openapi-patch=false -f -\\n[\\s\\S]*<<REB \\| kubectl apply --openapi-patch=false -f -\\n"

0 commit comments

Comments
 (0)