Skip to content

Commit ece050c

Browse files
[Backport release-1.6] fix(kafka): delete release topics before the topic operator is removed (#4456)
Backport of #3938 and #4280 to `release-1.6`, together, because #3938 alone breaks every Kafka deletion (#4276) and #4280 is what fixes that. Kafka suite on this branch fails the delete step on 2 of the last 3 E2E runs on the container lane (#4328, #4377), both times with `UninstallFailed ... KafkaTopic/tenant-test/kafka-test-test-results termination timeout: context deadline exceeded`. Helm removes the topic operator together with the release, a KafkaTopic that still carries the `strimzi.io/topic-operator` finalizer then has nobody to release it, so uninstall waits out its timeout and leaves the topic behind, which is what #3793 hits on reinstall. When the operator wins the race the same delete takes 3s. The pre-delete hook from #3938 deletes the release topics while the operator is still running, #4280 makes it actually reach the apiserver and reaps its credentials. All ten commits cherry-picked clean with `-x`. Kafka on this branch is still ZooKeeper-based, the hook does not depend on that: it selects topics by `strimzi.io/cluster=<release>`, the label the 1.6 chart puts on them. ### Testing - `make unit-tests` green (kafka helm unit tests and `hack/kafka-pre-delete-hook.bats` included), POSIX sh sweep clean. - E2E on this PR runs the kafka suite, which now asserts both release topics are gone after deletion. ```release-note fix(kafka): delete release-owned KafkaTopics before removing the Strimzi topic operator ```
2 parents d32dc78 + 9dabf19 commit ece050c

4 files changed

Lines changed: 534 additions & 0 deletions

File tree

‎hack/e2e-chainsaw/kafka/chainsaw-test.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,11 +107,39 @@ spec:
107107

108108
- name: verify-pvc-reclaim-on-delete
109109
try:
110+
- assert:
111+
timeout: 2m
112+
resource:
113+
apiVersion: kafka.strimzi.io/v1beta2
114+
kind: KafkaTopic
115+
metadata:
116+
name: kafka-test-test-results
117+
- assert:
118+
timeout: 2m
119+
resource:
120+
apiVersion: kafka.strimzi.io/v1beta2
121+
kind: KafkaTopic
122+
metadata:
123+
name: kafka-test-test-orders
110124
- delete:
111125
ref:
112126
apiVersion: apps.cozystack.io/v1alpha1
113127
kind: Kafka
114128
name: test
129+
- error:
130+
timeout: 5m
131+
resource:
132+
apiVersion: kafka.strimzi.io/v1beta2
133+
kind: KafkaTopic
134+
metadata:
135+
name: kafka-test-test-results
136+
- error:
137+
timeout: 5m
138+
resource:
139+
apiVersion: kafka.strimzi.io/v1beta2
140+
kind: KafkaTopic
141+
metadata:
142+
name: kafka-test-test-orders
115143
- error:
116144
timeout: 2m
117145
resource:

‎hack/kafka-pre-delete-hook.bats‎

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
#!/usr/bin/env bats
2+
# Behavioural tests for the Kafka pre-delete hook:
3+
#
4+
# packages/apps/kafka/templates/hooks/delete.yaml
5+
#
6+
# The script is extracted from the rendered chart and RUN against a fake
7+
# kubectl, because every property worth pinning here is an EXIT CODE and no
8+
# amount of matching against the manifest text establishes one. The
9+
# helm-unittest suite in packages/apps/kafka/tests/delete_test.yaml matches the
10+
# source text instead, and stayed green through a mutation that put the timeout
11+
# branch back to `exit 1` — which is the whole subject of this file.
12+
#
13+
# Why the exit code carries so much: a non-zero hook returns from Uninstall.Run
14+
# before any release resource goes, so Helm leaves the topic operator running
15+
# and helm-controller retries. That is the difference between "the uninstall
16+
# waits" and "the topic operator is gone while topics still hold
17+
# strimzi.io/topic-operator", which is issue #3793.
18+
#
19+
# What is pinned here:
20+
# * a delete that succeeds exits 0;
21+
# * a CRD that is absent is not a failure: the release may never have created
22+
# a topic, and the uninstall must not wedge on that;
23+
# * a wait that timed out exits 0 DELIBERATELY, so the uninstall can finish,
24+
# and says so on stderr. This is the branch 2b47c5a5 added; reverting it
25+
# must turn this file red;
26+
# * any OTHER kubectl failure exits non-zero, so a Forbidden or an apiserver
27+
# error cannot be mistaken for the timeout case and silently proceed;
28+
# * the selector names this release only.
29+
#
30+
# Run via hack/cozytest.sh from the repo root (make bats-unit-tests); relative
31+
# paths resolve against that cwd. That runner implements `@test` and little
32+
# else: it has NO `run` helper and therefore no $status or $output, so every
33+
# test below saves `$?` by hand the way hack/app-cleanup-hooks_test.bats does.
34+
# Under the bats binary `run` would work and the file would pass either way,
35+
# which is exactly how a suite that dies on `run: command not found` under the
36+
# real runner got committed once. docs/agents/e2e-testing.md:66 says it too.
37+
# Each @test builds its own fixture and removes it in the body: no
38+
# setup/teardown, and no EXIT trap.
39+
40+
KAFKA_CHART=packages/apps/kafka
41+
42+
# render_hook <chart> <release> <namespace> <job-name> <out>
43+
# Extract the named Job's shell script from the rendered chart into <out>. The
44+
# non-empty check must be an early `return 1`: hack/cozytest.sh rewrites every
45+
# line matching ^}$ into `return 0` plus `}`, helpers included, so a check left
46+
# in last-command position has its status discarded.
47+
render_hook() {
48+
chart=$1 release=$2 namespace=$3 job=$4 out=$5
49+
helm template "$release" "$chart" --namespace "$namespace" \
50+
| yq eval "select(.kind == \"Job\" and .metadata.name == \"$job\") | .spec.template.spec.containers[0].command[2]" - > "$out"
51+
[ -s "$out" ] || return 1
52+
}
53+
54+
# write_fake_kubectl <dir>
55+
# Emit a fake `kubectl` that logs its call to $KLOG and replays whatever
56+
# $DELETE_OUT holds, exiting with $DELETE_RC. The hook reads kubectl's combined
57+
# output and branches on its text, so the fixture is that text verbatim: the
58+
# timeout string is what kubectl v1.32 prints from the wait path, and the
59+
# CRD-absent strings are the two RESTMapper wordings.
60+
# DELETE_RC deliberately has NO default: an unset value would make every
61+
# failure-path test exit 0 and assert nothing. Each test sets it.
62+
# No line below is a bare column-0 `}`, so cozytest.sh's parser leaves it alone.
63+
write_fake_kubectl() {
64+
cat > "$1/kubectl" <<'KEOF'
65+
#!/bin/sh
66+
echo "$*" >> "$KLOG"
67+
if [ -z "${DELETE_RC:-}" ]; then
68+
echo "fake kubectl: DELETE_RC unset; the test would assert nothing" >&2
69+
exit 64
70+
fi
71+
printf '%s\n' "${DELETE_OUT:-kafkatopic.kafka.strimzi.io \"t\" deleted}"
72+
exit "$DELETE_RC"
73+
KEOF
74+
chmod +x "$1/kubectl"
75+
}
76+
77+
@test "a render that produces no script fails instead of passing as a no-op" {
78+
# Guards the guard: every test below runs whatever render_hook produced, so an
79+
# empty render would satisfy the exit-0 assertions without executing a line of
80+
# the hook.
81+
tmp=$(mktemp -d)
82+
if render_hook "$KAFKA_CHART" kafka-test tenant-test no-such-job "$tmp/hook.sh"; then
83+
echo "FAIL: render_hook reported success for a Job name the chart never emits"
84+
false
85+
fi
86+
rm -rf "$tmp"
87+
}
88+
89+
@test "kafka pre-delete hook exits 0 and deletes this release's topics only" {
90+
tmp=$(mktemp -d)
91+
write_fake_kubectl "$tmp"
92+
render_hook "$KAFKA_CHART" kafka-test tenant-test kafka-test-pre-delete "$tmp/hook.sh"
93+
94+
export KLOG="$tmp/calls"
95+
export DELETE_RC=0
96+
rc=0
97+
PATH="$tmp:$PATH" sh "$tmp/hook.sh" > "$tmp/out" 2> "$tmp/err" || rc=$?
98+
if [ "$rc" -ne 0 ]; then
99+
echo "FAIL: hook exited $rc on a clean delete"
100+
cat "$tmp/out" "$tmp/err"
101+
false
102+
fi
103+
# Anchored on both sides. Bare "strimzi.io/cluster=kafka-test" is a prefix of
104+
# the sibling release kafka-test2 and of kafka-test-topics, so a selector
105+
# widened to either stays green against it: verified by rendering the hook
106+
# with {{ .Release.Name }}2 and watching this test pass. The leading "-l " and
107+
# the trailing space are what make it name this release and no other.
108+
if ! grep -q -- "-l strimzi.io/cluster=kafka-test " "$tmp/calls"; then
109+
echo "FAIL: the delete was not scoped to this release"
110+
cat "$tmp/calls"
111+
false
112+
fi
113+
if ! grep -q -- "--ignore-not-found=true" "$tmp/calls"; then
114+
echo "FAIL: --ignore-not-found is gone, so an already-deleted topic fails the hook"
115+
cat "$tmp/calls"
116+
false
117+
fi
118+
rm -rf "$tmp"
119+
}
120+
121+
@test "kafka pre-delete hook exits 0 when the KafkaTopic CRD is absent" {
122+
# A release that never created a topic, or a cluster where Strimzi is already
123+
# gone. Failing here would wedge the uninstall over nothing to clean up. Both
124+
# RESTMapper wordings are covered because kubectl picks between them.
125+
for msg in 'error: the server doesn'"'"'t have a resource type "kafkatopics"' \
126+
'error: no matches for kind "KafkaTopic" in version "kafka.strimzi.io/v1beta2"'; do
127+
tmp=$(mktemp -d)
128+
write_fake_kubectl "$tmp"
129+
render_hook "$KAFKA_CHART" kafka-test tenant-test kafka-test-pre-delete "$tmp/hook.sh"
130+
131+
export KLOG="$tmp/calls"
132+
export DELETE_RC=1
133+
export DELETE_OUT="$msg"
134+
rc=0
135+
PATH="$tmp:$PATH" sh "$tmp/hook.sh" > "$tmp/out" 2> "$tmp/err" || rc=$?
136+
if [ "$rc" -ne 0 ]; then
137+
echo "FAIL: hook exited $rc on an absent CRD, wedging the uninstall"
138+
cat "$tmp/out" "$tmp/err"
139+
false
140+
fi
141+
if ! grep -q 'CRD absent' "$tmp/out"; then
142+
echo "FAIL: the absent CRD was not reported as such"
143+
cat "$tmp/out"
144+
false
145+
fi
146+
rm -rf "$tmp"
147+
done
148+
}
149+
150+
@test "kafka pre-delete hook exits 0 on a wait timeout and says the finalizer is still held" {
151+
# The branch 2b47c5a5 added. Exiting non-zero here wedges the uninstall
152+
# forever, because helm-controller retries it; exiting 0 lets the uninstall
153+
# finish and leaves the topics for a human. Reverting that branch to `exit 1`
154+
# turns this test red, which is the point: the helm-unittest suite does not
155+
# notice.
156+
tmp=$(mktemp -d)
157+
write_fake_kubectl "$tmp"
158+
render_hook "$KAFKA_CHART" kafka-test tenant-test kafka-test-pre-delete "$tmp/hook.sh"
159+
160+
export KLOG="$tmp/calls"
161+
export DELETE_RC=1
162+
export DELETE_OUT='error: timed out waiting for the condition on kafkatopics/kafka-test-orders'
163+
rc=0
164+
PATH="$tmp:$PATH" sh "$tmp/hook.sh" > "$tmp/out" 2> "$tmp/err" || rc=$?
165+
if [ "$rc" -ne 0 ]; then
166+
echo "FAIL: hook exited $rc on a wait timeout, which wedges the uninstall"
167+
cat "$tmp/out" "$tmp/err"
168+
false
169+
fi
170+
if ! grep -q 'still hold the strimzi.io/topic-operator finalizer' "$tmp/err"; then
171+
echo "FAIL: the degraded path said nothing about the finalizer"
172+
cat "$tmp/err"
173+
false
174+
fi
175+
if ! grep -q 'clear the finalizer by hand' "$tmp/err"; then
176+
echo "FAIL: the degraded path did not say what to do about it"
177+
cat "$tmp/err"
178+
false
179+
fi
180+
rm -rf "$tmp"
181+
}
182+
183+
@test "kafka pre-delete hook exits non-zero on any other kubectl failure" {
184+
# A Forbidden, an unreachable apiserver or a webhook rejection must NOT be
185+
# read as the timeout case. Here the non-zero exit is the safety net: Helm
186+
# stops before deleting the topic operator, so the topics keep something that
187+
# can clear their finalizer.
188+
tmp=$(mktemp -d)
189+
write_fake_kubectl "$tmp"
190+
render_hook "$KAFKA_CHART" kafka-test tenant-test kafka-test-pre-delete "$tmp/hook.sh"
191+
192+
export KLOG="$tmp/calls"
193+
export DELETE_RC=1
194+
export DELETE_OUT='Error from server (Forbidden): kafkatopics.kafka.strimzi.io is forbidden'
195+
rc=0
196+
PATH="$tmp:$PATH" sh "$tmp/hook.sh" > "$tmp/out" 2> "$tmp/err" || rc=$?
197+
if [ "$rc" -eq 0 ]; then
198+
echo "FAIL: hook exited 0 on a Forbidden, so Helm would remove the topic operator"
199+
cat "$tmp/out" "$tmp/err"
200+
false
201+
fi
202+
if ! grep -q '^ERROR: deleting KafkaTopics failed' "$tmp/err"; then
203+
echo "FAIL: the failure was not reported as an error"
204+
cat "$tmp/err"
205+
false
206+
fi
207+
rm -rf "$tmp"
208+
}

0 commit comments

Comments
 (0)