Skip to content

Commit b113cc7

Browse files
authored
fix(kafka): delete the release's node pools on uninstall (#4490)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Deleting a Kafka application left its brokers running. The migration hook creates the KafkaNodePools with `kubectl apply`, so they are not in the release manifest and `helm uninstall` does not remove them, and each pool owns the StrimziPodSet that runs its brokers. The pre-delete hook now deletes the release's pools, selected by `strimzi.io/cluster=<release>`, which is the label the migration hook stamps. Its Role gains `list` and `delete` on `kafkanodepools`, which is what a label-selected delete issues. Fixes #4488. The pools are deleted after the KafkaTopics, also when the topic wait timed out. The topic operator clears topic finalizers through the brokers, so the brokers have to outlive it. Deleting the pools before Helm deletes the Kafka CR does not bring brokers back. For the few seconds in between, Strimzi 0.45.1-rc1 (the version the operator chart pins) fails the reconcile with `KafkaNodePools are enabled, but no KafkaNodePools found` before it builds any StrimziPodSet. A regular pool is the ownerReference of its StrimziPodSet (`NodePoolUtils.createKafkaPools` at that tag), so the pods go away with the pool by garbage collection. If the Kafka CR is already gone when the hook runs, the pools are still found by label. An absent CRD exits 0, and so does a release with no pools, because a label-selected delete that matches nothing succeeds. Any other failure exits 1. The call uses `--wait=false` and no request timeout, so the #4276 invariants and the 270s deadline arithmetic in `delete.yaml` stay as they were. The cleanup lives in the pre-delete hook because only there a failure gets retried. In Helm v4.1.1, which helm-controller v1.5.0 vendors, a pre-delete hook error returns from `Uninstall.Run` before anything is deleted, so the next reconcile runs the hook again. A post-delete error is only collected: the release is still purged and Helm never runs its hooks again, so a failure there would leave the pools for good. An ownerReference from each pool to the Kafka CR would also work. I did not take it because it changes what the migration hook applies on every install and upgrade, including the ZooKeeper migration path that adopts a namespace-wide `kafka` pool. PVCs are out of scope. The pools set `deleteClaim: false`, so Strimzi leaves the broker and controller volumes in place when a pool is deleted, before and after this change. Not covered by this change: - Pools leaked by releases deleted before this fix stay where they are. Find them with `kubectl get kafkanodepools.kafka.strimzi.io -A -L strimzi.io/cluster`; a pool whose cluster has no Kafka CR in its namespace (`kubectl get kafkas.kafka.strimzi.io -A`) is leaked, and `kubectl -n <namespace> delete kafkanodepools.kafka.strimzi.io -l strimzi.io/cluster=<cluster>` removes it and its brokers. - `examples/backups/kafka-metadata/cleanup.sh` returns without waiting for the Kafka uninstalls, unlike `examples/backups/kafka/cleanup.sh`, which is why the metadata suite's new checks carry a 12m budget. The new step already holds the suite until the pools and pods are gone, so the next suite no longer starts into a namespace that is still uninstalling. Adding the same `wait_deleted` calls to the script would let those checks drop to the default timeout; left as a follow-up because the example is outside this change. - The "Deletion and PVC retention" section of `packages/apps/kafka/README.md` does not yet say that the pre-delete hook now removes the node pools while `deleteClaim: false` keeps the PVCs. That one sentence is left as a follow-up. - Helm runs the hooks stored in the release's latest revision, so a release uninstalled before it has been upgraded onto this chart still runs the old hook and still leaks its pools. Tests: - `packages/apps/kafka/tests/delete_test.yaml`: a new case pins, in one pattern, that the pool delete follows the topic delete, is scoped to this release, and does not wait, and pins the exact Role rule. - `hack/kafka-pre-delete-hook.bats` runs the rendered hook against a fake kubectl. Its fake now answers the pool call separately, and four new cases pin the exit codes and ordering: pools are deleted after the topics, still deleted after a topic-wait timeout, an absent pool CRD exits 0, and a failed pool delete exits non-zero. - Each new pin was checked by a mutation that turned it red: widening the selector, moving the pool delete before the topics, waiting on it, exiting on an absent CRD, not exiting on a failure, skipping the pools on the timeout branch, and removing or widening the Role rule. - E2E: the `kafka`, `kafka-2-backup-roundtrip` and `kafka-3-metadata-roundtrip` tests each gain a step, `verify-uninstall-removes-node-pools`, that checks, after the apps are deleted, that no KafkaNodePool and no pod labelled `strimzi.io/cluster=<cluster>` remains. The pod check covers every Strimzi pod of the cluster, not only the brokers, on purpose. The `kafka` test now deletes its app itself so it can look afterwards. E2E success criterion, registered before the run: in this PR's E2E run, every `verify-uninstall-removes-node-pools` step (one in each of the tests `kafka`, `kafka-2-backup-roundtrip` and `kafka-3-metadata-roundtrip`) passes on the first attempt. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> No UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches one chart template and its unit tests, one hack unit test, and two e2e suites. No values, schema, README, package list, hack tooling, CRD or node requirement changes, so no entry of the trigger map applies. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(kafka): deleting a Kafka application now also deletes its KafkaNodePools, so its brokers stop instead of running on after the application is gone. Volumes are kept, as before. Node pools left behind by applications deleted before this fix are not removed automatically: delete them with `kubectl -n <namespace> delete kafkanodepools.kafka.strimzi.io -l strimzi.io/cluster=<cluster>`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Kafka app uninstall now removes associated Kafka node pools after attempting to remove Kafka topics. If topic cleanup times out, uninstall continues; node pool cleanup failures are reported. * End-to-end checks now verify that Kafka node pools and pods are removed after uninstall and backup cleanup. If resources remain, diagnostic details are provided. * **Tests** * Added coverage for node pool cleanup outcomes, including missing resource definitions and deletion failures. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 2c2c030 + 7002397 commit b113cc7

5 files changed

Lines changed: 355 additions & 3 deletions

File tree

‎hack/e2e-chainsaw/kafka-metadata/chainsaw-test.yaml‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,3 +99,61 @@ spec:
9999
set -eu
100100
cd ../../..
101101
NAMESPACE="$NAMESPACE" examples/backups/kafka-metadata/cleanup.sh
102+
103+
# cleanup.sh above deleted both Kafka apps; the node pools the migration hook
104+
# created outside the release manifest must go with their uninstalls. The pod
105+
# checks select on strimzi.io/cluster on purpose, so they cover every pod
106+
# Strimzi runs for each cluster, not only the brokers.
107+
#
108+
# 12m per check, because cleanup.sh returns without waiting for the
109+
# uninstalls. The pools are deleted inside each release's pre-delete Job,
110+
# which activeDeadlineSeconds 270 plus terminationGracePeriodSeconds 10 bound
111+
# at 280s; the pods then take Strimzi's default 30s termination grace period.
112+
# If the two uninstalls run one after the other, 2 x 280 + 30 = 590s, and the
113+
# remaining 130s cover helm-controller picking up each deletion and garbage
114+
# collection. Every check carries the whole budget, since the order in which
115+
# the two finish is not fixed; a passing run waits only as long as the
116+
# uninstalls take.
117+
- name: verify-uninstall-removes-node-pools
118+
try:
119+
- error:
120+
timeout: 12m
121+
resource:
122+
apiVersion: kafka.strimzi.io/v1beta2
123+
kind: KafkaNodePool
124+
metadata:
125+
labels:
126+
strimzi.io/cluster: kafka-kafka-meta-src
127+
- error:
128+
timeout: 12m
129+
resource:
130+
apiVersion: kafka.strimzi.io/v1beta2
131+
kind: KafkaNodePool
132+
metadata:
133+
labels:
134+
strimzi.io/cluster: kafka-kafka-meta-target
135+
- error:
136+
timeout: 12m
137+
resource:
138+
apiVersion: v1
139+
kind: Pod
140+
metadata:
141+
labels:
142+
strimzi.io/cluster: kafka-kafka-meta-src
143+
- error:
144+
timeout: 12m
145+
resource:
146+
apiVersion: v1
147+
kind: Pod
148+
metadata:
149+
labels:
150+
strimzi.io/cluster: kafka-kafka-meta-target
151+
catch:
152+
# A failed error op already names each object still matching. This adds
153+
# the StrimziPodSets between the pools and the pods, and the pods' phase.
154+
- script:
155+
timeout: 1m
156+
content: |
157+
kubectl -n "$NAMESPACE" --request-timeout=15s \
158+
get kafkanodepools.kafka.strimzi.io,strimzipodsets.core.strimzi.io,pods \
159+
-l 'strimzi.io/cluster in (kafka-kafka-meta-src,kafka-kafka-meta-target)' -o wide || true

‎hack/e2e-chainsaw/kafka/chainsaw-test.yaml‎

Lines changed: 93 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@
44
# of separate broker + controller KafkaNodePools. This test asserts the KRaft
55
# topology renders and converges (both node pools present, the Strimzi Kafka CR
66
# reaches kafkaMetadataState=KRaft) and that no ZooKeeper resources remain.
7-
# Chainsaw's built-in cleanup deletes the applied Kafka CR and fails the test if
8-
# the teardown hangs, so no explicit delete step is needed.
7+
# The last step deletes the app itself and checks that no node pool or Strimzi
8+
# pod of this cluster survives the uninstall.
99
apiVersion: chainsaw.kyverno.io/v1alpha1
1010
kind: Test
1111
metadata:
@@ -121,6 +121,51 @@ spec:
121121
metadata:
122122
name: kafka-test-kafka-bootstrap
123123
(subsets[0].addresses[0].ip != null): true
124+
125+
# The node pools are created by the migration hook, outside the release
126+
# manifest, and each owns the StrimziPodSet running its brokers, so only the
127+
# chart's pre-delete hook removes them. Deleting the app explicitly here,
128+
# rather than leaving it to Chainsaw's cleanup, is what lets the test look at
129+
# the namespace afterwards. The pod check selects on strimzi.io/cluster on
130+
# purpose, so it covers every pod Strimzi runs for this cluster (brokers,
131+
# controllers, entity operator), not only the brokers.
132+
- name: verify-uninstall-removes-node-pools
133+
try:
134+
# Waits for the app, and so the HelmRelease uninstall behind it, to finish.
135+
# 8m, the same budget and for the same reason as timeouts.cleanup above:
136+
# this delete is the teardown that cleanup used to perform.
137+
- delete:
138+
timeout: 8m
139+
ref:
140+
apiVersion: apps.cozystack.io/v1alpha1
141+
kind: Kafka
142+
name: test
143+
# The pre-delete hook deleted the pools before the delete above returned;
144+
# the pods then go through garbage collection and their termination grace
145+
# period, which the default error timeout covers.
146+
- error:
147+
resource:
148+
apiVersion: kafka.strimzi.io/v1beta2
149+
kind: KafkaNodePool
150+
metadata:
151+
labels:
152+
strimzi.io/cluster: kafka-test
153+
- error:
154+
resource:
155+
apiVersion: v1
156+
kind: Pod
157+
metadata:
158+
labels:
159+
strimzi.io/cluster: kafka-test
160+
catch:
161+
# A failed error op already names each object still matching. This adds
162+
# the StrimziPodSets between the pools and the pods, and the pods' phase.
163+
- script:
164+
timeout: 1m
165+
content: |
166+
kubectl -n "$NAMESPACE" --request-timeout=15s \
167+
get kafkanodepools.kafka.strimzi.io,strimzipodsets.core.strimzi.io,pods \
168+
-l strimzi.io/cluster=kafka-test -o wide || true
124169
---
125170
# Logical topic-data backup -> in-place + to-copy RestoreJob round-trip,
126171
# driving the validated example scripts (examples/backups/kafka) as the harness
@@ -254,3 +299,49 @@ spec:
254299
set -eu
255300
cd ../../..
256301
NAMESPACE="$NAMESPACE" examples/backups/kafka/cleanup.sh
302+
303+
# cleanup.sh above deleted both Kafka apps and waited for them to go, so their
304+
# uninstalls have run, and the node pools the migration hook created outside
305+
# the release manifest must be gone with them. The pod checks select on
306+
# strimzi.io/cluster on purpose, so they cover every pod Strimzi runs for each
307+
# cluster, not only the brokers. What is left is garbage collection and the
308+
# pods' termination grace period, which the default error timeout covers.
309+
- name: verify-uninstall-removes-node-pools
310+
try:
311+
- error:
312+
resource:
313+
apiVersion: kafka.strimzi.io/v1beta2
314+
kind: KafkaNodePool
315+
metadata:
316+
labels:
317+
strimzi.io/cluster: kafka-kafka-test
318+
- error:
319+
resource:
320+
apiVersion: kafka.strimzi.io/v1beta2
321+
kind: KafkaNodePool
322+
metadata:
323+
labels:
324+
strimzi.io/cluster: kafka-kafka-restore
325+
- error:
326+
resource:
327+
apiVersion: v1
328+
kind: Pod
329+
metadata:
330+
labels:
331+
strimzi.io/cluster: kafka-kafka-test
332+
- error:
333+
resource:
334+
apiVersion: v1
335+
kind: Pod
336+
metadata:
337+
labels:
338+
strimzi.io/cluster: kafka-kafka-restore
339+
catch:
340+
# A failed error op already names each object still matching. This adds
341+
# the StrimziPodSets between the pools and the pods, and the pods' phase.
342+
- script:
343+
timeout: 1m
344+
content: |
345+
kubectl -n "$NAMESPACE" --request-timeout=15s \
346+
get kafkanodepools.kafka.strimzi.io,strimzipodsets.core.strimzi.io,pods \
347+
-l 'strimzi.io/cluster in (kafka-kafka-test,kafka-kafka-restore)' -o wide || true

‎hack/kafka-pre-delete-hook.bats‎

Lines changed: 141 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,14 @@
2525
# must turn this file red;
2626
# * any OTHER kubectl failure exits non-zero, so a Forbidden or an apiserver
2727
# error cannot be mistaken for the timeout case and silently proceed;
28-
# * the selector names this release only.
28+
# * the selector names this release only;
29+
# * the release's KafkaNodePools are deleted after the topics, including on
30+
# the timeout branch. The migration hook creates them outside the release
31+
# manifest, and each owns the StrimziPodSet running its brokers, so a pool
32+
# the hook skips keeps its brokers running after the release is gone;
33+
# * a KafkaNodePool CRD that is absent exits 0, and any other failure of
34+
# that delete exits non-zero, so the uninstall is retried rather than
35+
# finishing with the pools still in place.
2936
#
3037
# Run via hack/cozytest.sh from the repo root (make bats-unit-tests); relative
3138
# paths resolve against that cwd. That runner implements `@test` and little
@@ -59,11 +66,20 @@ render_hook() {
5966
# CRD-absent strings are the two RESTMapper wordings.
6067
# DELETE_RC deliberately has NO default: an unset value would make every
6168
# failure-path test exit 0 and assert nothing. Each test sets it.
69+
# The KafkaNodePool delete answers from POOL_OUT / POOL_RC instead, so a test
70+
# can fail one call without failing the other. POOL_RC defaults to a clean
71+
# delete, because the topic-path tests above are about the topic call; every
72+
# pool-path test sets it explicitly.
6273
# No line below is a bare column-0 `}`, so cozytest.sh's parser leaves it alone.
6374
write_fake_kubectl() {
6475
cat > "$1/kubectl" <<'KEOF'
6576
#!/bin/sh
6677
echo "$*" >> "$KLOG"
78+
case "$*" in
79+
*kafkanodepools*)
80+
printf '%s\n' "${POOL_OUT:-kafkanodepool.kafka.strimzi.io \"b-0\" deleted}"
81+
exit "${POOL_RC:-0}" ;;
82+
esac
6783
if [ -z "${DELETE_RC:-}" ]; then
6884
echo "fake kubectl: DELETE_RC unset; the test would assert nothing" >&2
6985
exit 64
@@ -206,3 +222,127 @@ KEOF
206222
fi
207223
rm -rf "$tmp"
208224
}
225+
226+
@test "kafka pre-delete hook deletes this release's node pools after its topics" {
227+
tmp=$(mktemp -d)
228+
write_fake_kubectl "$tmp"
229+
render_hook "$KAFKA_CHART" kafka-test tenant-test kafka-test-pre-delete "$tmp/hook.sh"
230+
231+
export KLOG="$tmp/calls"
232+
export DELETE_RC=0
233+
export DELETE_OUT='kafkatopic.kafka.strimzi.io "t" deleted'
234+
export POOL_RC=0
235+
export POOL_OUT='kafkanodepool.kafka.strimzi.io "b-0" deleted'
236+
rc=0
237+
PATH="$tmp:$PATH" sh "$tmp/hook.sh" > "$tmp/out" 2> "$tmp/err" || rc=$?
238+
if [ "$rc" -ne 0 ]; then
239+
echo "FAIL: hook exited $rc on a clean delete"
240+
cat "$tmp/out" "$tmp/err"
241+
false
242+
fi
243+
# Anchored like the topic selector above: the trailing space keeps the
244+
# sibling release kafka-test2 out.
245+
pool_line=$(grep -n -- "delete kafkanodepools.kafka.strimzi.io -l strimzi.io/cluster=kafka-test " "$tmp/calls" | cut -d: -f1)
246+
topic_line=$(grep -n -- "delete kafkatopics.kafka.strimzi.io -l strimzi.io/cluster=kafka-test " "$tmp/calls" | cut -d: -f1)
247+
if [ -z "$pool_line" ] || [ -z "$topic_line" ]; then
248+
echo "FAIL: expected one release-scoped delete of each kind"
249+
cat "$tmp/calls"
250+
false
251+
fi
252+
# The topic operator clears topic finalizers through the brokers, so the
253+
# brokers have to outlive the topic delete.
254+
if [ "$pool_line" -le "$topic_line" ]; then
255+
echo "FAIL: the node pools were deleted before the topics"
256+
cat "$tmp/calls"
257+
false
258+
fi
259+
if ! grep -q -- "kafkanodepools.* --wait=false" "$tmp/calls"; then
260+
echo "FAIL: the node pool delete waits, which spends the Job's deadline"
261+
cat "$tmp/calls"
262+
false
263+
fi
264+
rm -rf "$tmp"
265+
}
266+
267+
@test "kafka pre-delete hook still deletes the node pools after a topic wait timeout" {
268+
# The timeout branch lets the uninstall finish. Skipping the pools there
269+
# would leave the brokers running with nothing left to remove them.
270+
tmp=$(mktemp -d)
271+
write_fake_kubectl "$tmp"
272+
render_hook "$KAFKA_CHART" kafka-test tenant-test kafka-test-pre-delete "$tmp/hook.sh"
273+
274+
export KLOG="$tmp/calls"
275+
export DELETE_RC=1
276+
export DELETE_OUT='error: timed out waiting for the condition on kafkatopics/kafka-test-orders'
277+
export POOL_RC=0
278+
export POOL_OUT='kafkanodepool.kafka.strimzi.io "b-0" deleted'
279+
rc=0
280+
PATH="$tmp:$PATH" sh "$tmp/hook.sh" > "$tmp/out" 2> "$tmp/err" || rc=$?
281+
if [ "$rc" -ne 0 ]; then
282+
echo "FAIL: hook exited $rc on a wait timeout"
283+
cat "$tmp/out" "$tmp/err"
284+
false
285+
fi
286+
if ! grep -q -- "delete kafkanodepools.kafka.strimzi.io -l strimzi.io/cluster=kafka-test " "$tmp/calls"; then
287+
echo "FAIL: the node pools were skipped after the topic wait timed out"
288+
cat "$tmp/calls"
289+
false
290+
fi
291+
rm -rf "$tmp"
292+
}
293+
294+
@test "kafka pre-delete hook exits 0 when the KafkaNodePool CRD is absent" {
295+
for msg in 'error: the server doesn'"'"'t have a resource type "kafkanodepools"' \
296+
'error: no matches for kind "KafkaNodePool" in version "kafka.strimzi.io/v1beta2"'; do
297+
tmp=$(mktemp -d)
298+
write_fake_kubectl "$tmp"
299+
render_hook "$KAFKA_CHART" kafka-test tenant-test kafka-test-pre-delete "$tmp/hook.sh"
300+
301+
export KLOG="$tmp/calls"
302+
export DELETE_RC=0
303+
export DELETE_OUT='kafkatopic.kafka.strimzi.io "t" deleted'
304+
export POOL_RC=1
305+
export POOL_OUT="$msg"
306+
rc=0
307+
PATH="$tmp:$PATH" sh "$tmp/hook.sh" > "$tmp/out" 2> "$tmp/err" || rc=$?
308+
if [ "$rc" -ne 0 ]; then
309+
echo "FAIL: hook exited $rc on an absent KafkaNodePool CRD, wedging the uninstall"
310+
cat "$tmp/out" "$tmp/err"
311+
false
312+
fi
313+
if ! grep -q 'KafkaNodePool CRD absent' "$tmp/out"; then
314+
echo "FAIL: the absent CRD was not reported as such"
315+
cat "$tmp/out"
316+
false
317+
fi
318+
rm -rf "$tmp"
319+
done
320+
}
321+
322+
@test "kafka pre-delete hook exits non-zero when the node pool delete fails" {
323+
# Once the uninstall finishes, Helm never runs this hook for the release
324+
# again, so a pool left here is left for good. A non-zero exit stops the
325+
# uninstall before it deletes anything, and helm-controller retries it.
326+
tmp=$(mktemp -d)
327+
write_fake_kubectl "$tmp"
328+
render_hook "$KAFKA_CHART" kafka-test tenant-test kafka-test-pre-delete "$tmp/hook.sh"
329+
330+
export KLOG="$tmp/calls"
331+
export DELETE_RC=0
332+
export DELETE_OUT='kafkatopic.kafka.strimzi.io "t" deleted'
333+
export POOL_RC=1
334+
export POOL_OUT='Error from server (Forbidden): kafkanodepools.kafka.strimzi.io is forbidden'
335+
rc=0
336+
PATH="$tmp:$PATH" sh "$tmp/hook.sh" > "$tmp/out" 2> "$tmp/err" || rc=$?
337+
if [ "$rc" -eq 0 ]; then
338+
echo "FAIL: hook exited 0 on a Forbidden, so the uninstall would finish and leave the pools"
339+
cat "$tmp/out" "$tmp/err"
340+
false
341+
fi
342+
if ! grep -q '^ERROR: deleting KafkaNodePools failed' "$tmp/err"; then
343+
echo "FAIL: the failure was not reported as an error"
344+
cat "$tmp/err"
345+
false
346+
fi
347+
rm -rf "$tmp"
348+
}

‎packages/apps/kafka/templates/hooks/delete.yaml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,36 @@ spec:
9696
else
9797
printf '%s\n' "${out}"
9898
fi
99+
100+
# The migration hook creates the node pools with kubectl apply, so
101+
# they are not in the release manifest and the uninstall does not
102+
# remove them, and each pool owns the StrimziPodSet that runs its
103+
# brokers. After the topics, because the topic operator clears
104+
# their finalizers through those brokers; also after the timeout
105+
# branch above, since this is the last chance to remove them.
106+
# --wait=false: the pods go by garbage collection, and waiting
107+
# would spend the headroom the deadline above leaves. No request
108+
# timeout either, for the reason given on the topic call.
109+
echo "Deleting KafkaNodePools for {{ .Release.Name }}"
110+
if ! out=$(kubectl -n {{ .Release.Namespace }} delete kafkanodepools.kafka.strimzi.io \
111+
-l 'strimzi.io/cluster={{ .Release.Name }}' \
112+
--ignore-not-found=true \
113+
--wait=false 2>&1); then
114+
printf '%s\n' "${out}"
115+
case "${out}" in
116+
*"the server doesn't have a resource type"*|*"no matches for kind"*)
117+
echo "KafkaNodePool CRD absent; nothing to delete" ;;
118+
*)
119+
# Non-zero, unlike the topic timeout: once the uninstall
120+
# completes, Helm never runs this hook for the release
121+
# again, so a pool left now keeps its brokers for good.
122+
# Failing stops the uninstall before it deletes anything,
123+
# and helm-controller retries it.
124+
echo "ERROR: deleting KafkaNodePools failed: ${out}" >&2; exit 1 ;;
125+
esac
126+
else
127+
printf '%s\n' "${out}"
128+
fi
99129
volumeMounts:
100130
- name: tmp
101131
mountPath: /tmp
@@ -133,6 +163,9 @@ rules:
133163
- apiGroups: ["kafka.strimzi.io"]
134164
resources: ["kafkatopics"]
135165
verbs: ["get", "list", "watch", "delete"]
166+
- apiGroups: ["kafka.strimzi.io"]
167+
resources: ["kafkanodepools"]
168+
verbs: ["list", "delete"]
136169
---
137170
apiVersion: rbac.authorization.k8s.io/v1
138171
kind: RoleBinding

0 commit comments

Comments
 (0)