Skip to content

Commit b323d5d

Browse files
authored
fix(kafka): keep the pre-delete hook's credentials as long as its Job (cozystack#4280)
## What this PR does Fixes cozystack#4276. Since cozystack#3938 landed, no Kafka release finishes deleting: the pre-delete hook cannot reach the apiserver, exits non-zero, and a non-zero pre-delete hook returns from `Uninstall.Run` before a single resource is removed. It is the inverse of what the hook was written for. cozystack#3793 was one release that would not delete; this made every release refuse to. The cause is `--request-timeout=20s` on the delete call. Any non-zero value for that flag makes kubectl skip its in-cluster credentials altogether: client-go consults them only while the assembled config still equals the default, and the flag lands in that config, so the branch that would load the ServiceAccount token never runs. kubectl talks to its built-in default cluster, `localhost:8080`, is refused, and the script reads the refusal as a failed delete. Measured on the pinned image rather than argued from documentation. With the token and `ca.crt` mounted and `KUBERNETES_SERVICE_HOST` set, the same delete call reaches the cluster with no flag, goes to `localhost:8080` with `--request-timeout=20s`, goes to `localhost:8080` with `1m`, and reaches the cluster again with `0`. The tree agrees. None of the three hooks that work pass it: `apps/kubernetes` names it in a comment only, `apps/harbor` and `apps/mariadb` never mention it, and `apps/tenant/templates/cleanup-job.yaml` records that its DELETE calls "carry kubectl's default `--request-timeout` of 0". The watch is still bounded by `--timeout`, and the Job by `activeDeadlineSeconds`, which is that same shape. The second commit was a separate change found on the way, and review turned it around. It matched the credentials to the Job on `before-hook-creation` alone, and a ServiceAccount, Role or RoleBinding has nothing like `ttlSecondsAfterFinished` to fall back on: helm removes a hook resource only when a policy says so, and `before-hook-creation` fires on a next activation that after the last uninstall never comes. All three would have sat in the tenant namespace for the life of the cluster, still granting delete over every KafkaTopic in it. They now carry `before-hook-creation,hook-succeeded`, the shape `apps/clickhouse` and `apps/kubernetes-nodes` already use, while the Job keeps `before-hook-creation` alone because the script exits 0 on the degraded path too and the Job's log is the only record that a finalizer outlived the wait. Reaping the credentials cannot strand a running pod: helm applies `hook-succeeded` only once the hook is terminal, and `activeDeadlineSeconds: 270` ends the Job inside the 5m uninstall hook timeout. Neither of these caused cozystack#4276, and a cluster run confirmed as much. Both are pinned. Restoring the flag turns the suite red, with `0` as well as with `20s`; the assertion matches `--request-timeout=` rather than the bare name, because the script's own comment names the flag. Dropping the reaping policy from the ServiceAccount alone, or from the Role alone, turns it red too, so the three are three assertions rather than one. Worth stating plainly, because it is the reason this reached `main`: none of this is visible in a rendered manifest. The asymmetry and the flag both rendered cleanly, passed 37 helm-unittest cases and 5 bats cases through two rounds of review and per-branch mutation testing, and only a live uninstall showed either. The e2e lane never ran on cozystack#3938 while it was open, because its workflows sat at `action_required` on all five heads a fork PR had. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff. This changes hook annotations and test assertions inside `packages/apps/kafka`; no values key, schema, API, default or naming contract moves, and the app itself is neither added nor renamed. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(kafka): deleting a Kafka release no longer wedges on a pre-delete hook that cannot reach the apiserver ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved Kafka app uninstall cleanup by removing an unnecessary request-timeout constraint while retaining existing overall timeout safeguards. - Updated cleanup resource policies to remove failed-hook retention and preserve resources through successful hook completion, supporting reliable repeated uninstall attempts. - **Tests** - Added coverage for cleanup command timeout behavior and resource deletion policies. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 94885c2 + 3a43bcf commit b323d5d

2 files changed

Lines changed: 59 additions & 5 deletions

File tree

‎packages/apps/kafka/templates/hooks/delete.yaml‎

Lines changed: 23 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -65,12 +65,24 @@ spec:
6565
- |
6666
set -eu
6767
echo "Deleting KafkaTopics for {{ .Release.Name }} before the topic-operator is removed"
68+
# The call carries no request timeout, deliberately. Any non-zero
69+
# value for that flag makes kubectl skip its in-cluster
70+
# credentials: client-go consults them only while the assembled
71+
# config still equals the default, and the flag lands in that
72+
# config. kubectl then talks to its built-in default cluster,
73+
# localhost:8080, is refused, and this script reads the refusal as
74+
# a failed delete and exits 1 -- which returns from Uninstall.Run
75+
# before anything is removed and wedges every uninstall of this
76+
# release. It leaves no trace in the rendered manifest; only a live
77+
# uninstall shows it (#4276). The call is bounded by --timeout
78+
# above for the watch and by activeDeadlineSeconds for the whole
79+
# Job, the shape packages/apps/tenant/templates/cleanup-job.yaml
80+
# uses and documents.
6881
if ! out=$(kubectl -n {{ .Release.Namespace }} delete kafkatopics.kafka.strimzi.io \
6982
-l 'strimzi.io/cluster={{ .Release.Name }}' \
7083
--ignore-not-found=true \
7184
--wait=true \
72-
--timeout=180s \
73-
--request-timeout=20s 2>&1); then
85+
--timeout=180s 2>&1); then
7486
printf '%s\n' "${out}"
7587
case "${out}" in
7688
*"the server doesn't have a resource type"*|*"no matches for kind"*)
@@ -100,7 +112,11 @@ metadata:
100112
annotations:
101113
"helm.sh/hook": pre-delete
102114
"helm.sh/hook-weight": "0"
103-
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded,hook-failed
115+
# hook-succeeded, unlike the Job above: the Job reclaims itself through
116+
# ttlSecondsAfterFinished, this has no such clock and would otherwise
117+
# outlive the release. Reaping runs only once the Job is terminal, which
118+
# activeDeadlineSeconds guarantees happens first.
119+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
104120
---
105121
apiVersion: rbac.authorization.k8s.io/v1
106122
kind: Role
@@ -111,7 +127,8 @@ metadata:
111127
annotations:
112128
"helm.sh/hook": pre-delete
113129
"helm.sh/hook-weight": "5"
114-
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded,hook-failed
130+
# Reaped with the ServiceAccount above, for the reason recorded there.
131+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
115132
rules:
116133
- apiGroups: ["kafka.strimzi.io"]
117134
resources: ["kafkatopics"]
@@ -126,7 +143,8 @@ metadata:
126143
annotations:
127144
"helm.sh/hook": pre-delete
128145
"helm.sh/hook-weight": "5"
129-
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded,hook-failed
146+
# Reaped with the ServiceAccount above, for the reason recorded there.
147+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
130148
roleRef:
131149
apiGroup: rbac.authorization.k8s.io
132150
kind: Role

‎packages/apps/kafka/tests/delete_test.yaml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,19 @@ tests:
5454
path: spec.template.spec.containers[0].command[2]
5555
pattern: "--wait=true"
5656
documentIndex: 0
57+
# A non-zero --request-timeout makes kubectl ignore its in-cluster
58+
# credentials: client-go consults them only while the assembled config
59+
# still equals the default, and the flag lands in that config. The call
60+
# then goes to localhost:8080, is refused, and the hook exits 1, which
61+
# wedges every uninstall of the release (#4276). Nothing in the rendered
62+
# manifest looks wrong, so this assertion is the only thing standing
63+
# between that flag and another cluster-only failure. The pattern needs
64+
# the "=": the script's own comment names the flag, and matching the bare
65+
# name would fail on the explanation rather than on a real use.
66+
- notMatchRegex:
67+
path: spec.template.spec.containers[0].command[2]
68+
pattern: "--request-timeout="
69+
documentIndex: 0
5770
# Pinned by shape, not by literal: Renovate rewrites both the tag and the
5871
# digest under packages/.+/templates/hooks/.+\.yaml and nothing rewrites
5972
# tests/, so a literal of either turns the next bump red for no defect.
@@ -69,9 +82,28 @@ tests:
6982
- isKind:
7083
of: ServiceAccount
7184
documentIndex: 1
85+
# Pinned as an asymmetry, not as a value: the Job above carries no
86+
# hook-succeeded because the script exits 0 on the degraded path too and
87+
# its log is the only record that a finalizer outlived the wait, while
88+
# these three have no ttlSecondsAfterFinished and without a terminal
89+
# policy would sit in the tenant namespace for the life of the cluster,
90+
# still granting delete over every KafkaTopic in it. Reaping them cannot
91+
# strand a running pod: helm reaps on hook-succeeded only once the Job is
92+
# terminal, and activeDeadlineSeconds 270 ends it inside the uninstall
93+
# hook timeout, HelmReleaseSpec.Timeout, 5m by default. hook-failed adds
94+
# nothing, since a failing hook already reaps the earlier successful ones
95+
# under hook-succeeded.
96+
- equal:
97+
path: metadata.annotations["helm.sh/hook-delete-policy"]
98+
value: before-hook-creation,hook-succeeded
99+
documentIndex: 1
72100
- isKind:
73101
of: Role
74102
documentIndex: 2
103+
- equal:
104+
path: metadata.annotations["helm.sh/hook-delete-policy"]
105+
value: before-hook-creation,hook-succeeded
106+
documentIndex: 2
75107
- contains:
76108
path: rules
77109
content:
@@ -82,6 +114,10 @@ tests:
82114
- isKind:
83115
of: RoleBinding
84116
documentIndex: 3
117+
- equal:
118+
path: metadata.annotations["helm.sh/hook-delete-policy"]
119+
value: before-hook-creation,hook-succeeded
120+
documentIndex: 3
85121
- equal:
86122
path: roleRef.name
87123
value: kafka-test-pre-delete

0 commit comments

Comments
 (0)