Skip to content

Commit 64d5689

Browse files
committed
fix(kafka): keep the pre-delete hook's credentials as long as its Job
The Job carries no hook-succeeded, so it outlives the hook run and ttlSecondsAfterFinished reclaims it, which is what leaves a readable record when the finalizer outlives the wait. Its ServiceAccount, Role and RoleBinding still carried hook-succeeded and hook-failed, so Helm reaped them at the end of the run either way and left the Job behind them. A Job whose ServiceAccount is gone still starts a pod. kubelet mounts the projected volume without a token, client-go finds no in-cluster config, kubectl falls back to localhost:8080, and the script takes its catch-all branch and exits 1. A non-zero pre-delete hook returns from Uninstall.Run before any resource goes, so the release never finishes deleting and helm-controller retries it: the e2e run on the merge commit shows six such pods in five minutes, in two namespaces, every one of them dead the same way. The hook meant to prevent one stuck Kafka release was instead stopping all of them. Give all four objects the same before-hook-creation policy, so the credentials live exactly as long as the consumer and the next run replaces them. Both other in-tree hooks that reach the apiserver, apps/kubernetes and apps/harbor, are symmetric this way already. The suite now pins the policy on each of the three, because nothing compared them: the asymmetry rendered cleanly, passed 37 helm-unittest cases and 5 bats cases, and only a cluster showed it. Assisted-by: LLM Signed-off-by: IvanHunters <[email protected]>
1 parent 6818795 commit 64d5689

2 files changed

Lines changed: 46 additions & 3 deletions

File tree

‎packages/apps/kafka/templates/hooks/delete.yaml‎

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,15 @@ metadata:
100100
annotations:
101101
"helm.sh/hook": pre-delete
102102
"helm.sh/hook-weight": "0"
103-
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded,hook-failed
103+
# Matches the Job's policy deliberately. The Job carries no hook-succeeded:
104+
# it outlives the run so the degraded path leaves a readable record, and
105+
# ttlSecondsAfterFinished reclaims it. Reaping the ServiceAccount, Role or
106+
# RoleBinding on hook-succeeded or hook-failed would leave that Job able to
107+
# start a pod whose ServiceAccount no longer exists: kubelet then mounts the
108+
# projected volume without a token, kubectl finds no in-cluster config and
109+
# falls back to localhost:8080, and the pre-delete hook fails every uninstall
110+
# from then on.
111+
"helm.sh/hook-delete-policy": before-hook-creation
104112
---
105113
apiVersion: rbac.authorization.k8s.io/v1
106114
kind: Role
@@ -111,7 +119,15 @@ metadata:
111119
annotations:
112120
"helm.sh/hook": pre-delete
113121
"helm.sh/hook-weight": "5"
114-
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded,hook-failed
122+
# Matches the Job's policy deliberately. The Job carries no hook-succeeded:
123+
# it outlives the run so the degraded path leaves a readable record, and
124+
# ttlSecondsAfterFinished reclaims it. Reaping the ServiceAccount, Role or
125+
# RoleBinding on hook-succeeded or hook-failed would leave that Job able to
126+
# start a pod whose ServiceAccount no longer exists: kubelet then mounts the
127+
# projected volume without a token, kubectl finds no in-cluster config and
128+
# falls back to localhost:8080, and the pre-delete hook fails every uninstall
129+
# from then on.
130+
"helm.sh/hook-delete-policy": before-hook-creation
115131
rules:
116132
- apiGroups: ["kafka.strimzi.io"]
117133
resources: ["kafkatopics"]
@@ -126,7 +142,15 @@ metadata:
126142
annotations:
127143
"helm.sh/hook": pre-delete
128144
"helm.sh/hook-weight": "5"
129-
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded,hook-failed
145+
# Matches the Job's policy deliberately. The Job carries no hook-succeeded:
146+
# it outlives the run so the degraded path leaves a readable record, and
147+
# ttlSecondsAfterFinished reclaims it. Reaping the ServiceAccount, Role or
148+
# RoleBinding on hook-succeeded or hook-failed would leave that Job able to
149+
# start a pod whose ServiceAccount no longer exists: kubelet then mounts the
150+
# projected volume without a token, kubectl finds no in-cluster config and
151+
# falls back to localhost:8080, and the pre-delete hook fails every uninstall
152+
# from then on.
153+
"helm.sh/hook-delete-policy": before-hook-creation
130154
roleRef:
131155
apiGroup: rbac.authorization.k8s.io
132156
kind: Role

‎packages/apps/kafka/tests/delete_test.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,9 +69,24 @@ tests:
6969
- isKind:
7070
of: ServiceAccount
7171
documentIndex: 1
72+
# The Job carries no hook-succeeded, so it outlives the hook run. Every
73+
# credential it needs has to outlive it too. A ServiceAccount reaped on
74+
# hook-succeeded or hook-failed leaves the Job able to start a pod whose
75+
# ServiceAccount is gone: the projected volume mounts without a token,
76+
# kubectl falls back to localhost:8080, and the uninstall wedges on a hook
77+
# that can no longer reach the apiserver. That is #4276, and it reached
78+
# main because nothing here compared the two policies.
79+
- equal:
80+
path: metadata.annotations["helm.sh/hook-delete-policy"]
81+
value: before-hook-creation
82+
documentIndex: 1
7283
- isKind:
7384
of: Role
7485
documentIndex: 2
86+
- equal:
87+
path: metadata.annotations["helm.sh/hook-delete-policy"]
88+
value: before-hook-creation
89+
documentIndex: 2
7590
- contains:
7691
path: rules
7792
content:
@@ -82,6 +97,10 @@ tests:
8297
- isKind:
8398
of: RoleBinding
8499
documentIndex: 3
100+
- equal:
101+
path: metadata.annotations["helm.sh/hook-delete-policy"]
102+
value: before-hook-creation
103+
documentIndex: 3
85104
- equal:
86105
path: roleRef.name
87106
value: kafka-test-pre-delete

0 commit comments

Comments
 (0)