Skip to content

Commit b60c08f

Browse files
committed
fix(kubernetes): bound the remaining pre-delete hook deletions
Three of the six steps in the pre-delete hook call kubectl delete with neither --wait=false nor --timeout. kubectl defaults --wait to true and reads --timeout=0 as "wait forever", which it implements as a week, so the KamajiControlPlane delete in step 3, the DataVolume delete in step 5 and the LoadBalancer Service delete in step 6 each park the hook for 168 hours on a single stuck finalizer. A DataVolume held by a CDI importer or a PVC finalizer is the reachable case. None of the three waits now. Nothing downstream reads the result of steps 5 and 6, and what matters after step 3 is the TenantControlPlane, which step 4 deletes and waits for directly; blocking on the KamajiControlPlane first only adds a second place for a CAPI finalizer to stall. An expired wait and --wait=false leave the same cluster state either way, with the same objects finalizing in the background once the hook has moved on. Not waiting also keeps these steps under `set -e` with no guard. A guard cannot tell an expired wait from a rejected request, so it would report a failed delete as cleanup that happened; with no timeout to expire, every non-zero exit left is a real failure. Drop the watch verb from the three resources those steps delete, and pin the whole rule set. kubectl opens a watch only on the wait path, so the grant was unexercised for the three; for helmreleases and tenantcontrolplanes its absence would not error but hang, which is the failure mode hardest to read from the outside. Set backoffLimit to 0, against a default of 6 that restartPolicy Never turns into seven pods in series. Zero rather than one because both remaining waits are guarded, so an attempt can spend its whole budget without failing and a retry would start from step 1 with the clock already there; on a nodeless tenant it would also re-pay step 4, since attempt one leaves the TenantControlPlane still terminating. The failure a retry would cover is an eviction, which is not bounded in time and is worst exactly when it arrives late. The price is that such an eviction fails the whole attempt and flux repeats it on its own interval. No activeDeadlineSeconds: the waits and helm-controller's uninstall budget leave no gap a deadline could sit in, once a Docker Hub image pull is counted. That needs the budget raised first, through spec.uninstall.timeout on the generated HelmRelease, and is left out rather than set to a number that would cut runs which were going to finish. These bounds cover the waiting, not the request: --request-timeout is still 0 on every call in the hook, so an unresponsive apiserver hangs it as before. That is a different failure from the stuck finalizer, and it wants one answer across every call rather than a flag on the deletes. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 6c82988 commit b60c08f

2 files changed

Lines changed: 179 additions & 6 deletions

File tree

‎packages/apps/kubernetes/templates/delete.yaml‎

Lines changed: 58 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,37 @@ metadata:
88
"helm.sh/hook-delete-policy": hook-succeeded,before-hook-creation
99
name: {{ .Release.Name }}-pre-cleanup
1010
spec:
11+
# One attempt. restartPolicy Never turns the default of 6 into seven pods in
12+
# series, each restarting the script from Step 1 and each paying its own image
13+
# pull, which outlives the uninstall budget: helm then times out on its own
14+
# instead of reporting which step failed.
15+
#
16+
# Zero and not one, because a second attempt does not fit. Both waits below
17+
# are guarded, so an attempt can spend all of its budget without failing and a
18+
# retry starts from Step 1 with the clock already there; on a nodeless tenant
19+
# it also re-pays Step 4, since attempt one leaves the TenantControlPlane
20+
# terminating. The failure a retry would cover is an eviction (this pod
21+
# tolerates control-plane taints), which is not bounded in time and costs most
22+
# when it arrives late. The price is that such an eviction fails the whole
23+
# attempt and flux repeats it on its own interval — slower, and predictable.
24+
backoffLimit: 0
25+
# No activeDeadlineSeconds, for arithmetic rather than oversight. A deadline
26+
# has to sit above the sum of the waits below, or it cuts runs that were going
27+
# to finish, and far enough under helm-controller's uninstall budget to be
28+
# what helm reports rather than what it waits through. Those two bounds do not
29+
# currently leave a gap, and the image pull comes from Docker Hub with no
30+
# mirror in front of it.
31+
#
32+
# Note that the sum is a whole-Job figure only while backoffLimit is 0:
33+
# activeDeadlineSeconds runs from status.startTime and a retry does not reset
34+
# it, so allowing retries means the deadline has to clear all of them.
35+
#
36+
# Room comes from raising spec.uninstall.timeout on the generated HelmRelease,
37+
# not from squeezing the waits further.
38+
# packages/apps/bucket/templates/hooks/cleanup-acme.yaml is the shape to copy
39+
# then: same image, with the deadline and the retry budget set together, and
40+
# --request-timeout on each call so an unresponsive apiserver is bounded too,
41+
# which --wait=false below does not cover.
1142
template:
1243
metadata:
1344
labels:
@@ -90,8 +121,21 @@ spec:
90121
fi
91122
92123
echo "Step 3: Deleting KamajiControlPlane {{ .Release.Name }}"
124+
# --wait is spelled out because kubectl defaults it to true, and a
125+
# --timeout it does not carry defaults to 0, which kubectl reads
126+
# as "wait forever" and turns into a week.
127+
#
128+
# false rather than a bound, because there is nothing here worth
129+
# waiting for. What matters downstream is the TenantControlPlane,
130+
# and Step 4 deletes that one directly and waits for it; blocking
131+
# on the KamajiControlPlane first only adds a second place for a
132+
# CAPI finalizer to stall the hook. Not waiting also keeps this
133+
# step under `set -e` with no guard: with no timeout to expire,
134+
# every remaining non-zero exit is a real failure — the API
135+
# rejecting the request, discovery breaking, RBAC being wrong —
136+
# and none of those should be reported as a cleanup that happened.
93137
kubectl -n {{ .Release.Namespace }} delete kamajicontrolplanes.controlplane.cluster.x-k8s.io {{ .Release.Name }} \
94-
--ignore-not-found=true
138+
--ignore-not-found=true --wait=false
95139
96140
echo "Step 4: Deleting TenantControlPlane {{ .Release.Name }} and waiting for it to be removed"
97141
# --wait=true blocks until the TenantControlPlane object is fully
@@ -137,16 +181,24 @@ spec:
137181
echo " Secret {{ .Release.Name }}-datastore-config not present; nothing to do"
138182
fi
139183
184+
# Steps 5 and 6 do not wait. Both defaulted to --wait=true with no
185+
# --timeout, so either one parked this hook for a week on a single
186+
# stuck finalizer: a DataVolume held by a CDI importer or a PVC
187+
# finalizer is the reachable case. Waiting buys nothing here.
188+
# Nothing after these steps reads their result, and the DELETE has
189+
# already been accepted either way — an expired wait and
190+
# --wait=false leave the same cluster state, with the same objects
191+
# finalizing in the background once the hook has moved on.
140192
echo "Step 5: Cleaning up DataVolumes"
141193
kubectl -n {{ .Release.Namespace }} delete datavolumes \
142194
-l "cluster.x-k8s.io/cluster-name={{ .Release.Name }}" \
143-
--ignore-not-found=true
195+
--ignore-not-found=true --wait=false
144196
145197
echo "Step 6: Cleaning up LoadBalancer Services"
146198
kubectl -n {{ .Release.Namespace }} delete services \
147199
-l "cluster.x-k8s.io/cluster-name={{ .Release.Name }}" \
148200
--field-selector spec.type=LoadBalancer \
149-
--ignore-not-found=true
201+
--ignore-not-found=true --wait=false
150202
151203
echo "Cleanup completed successfully"
152204
volumeMounts:
@@ -184,14 +236,16 @@ rules:
184236
- watch
185237
- patch
186238
- delete
239+
# watch is granted only where a delete still waits: Step 2 on helmreleases and
240+
# Step 4 on tenantcontrolplanes. The other three deletes pass --wait=false, so
241+
# kubectl never opens a watch for them and the verb would be unexercised.
187242
- apiGroups:
188243
- "controlplane.cluster.x-k8s.io"
189244
resources:
190245
- kamajicontrolplanes
191246
verbs:
192247
- get
193248
- list
194-
- watch
195249
- delete
196250
- apiGroups:
197251
- "kamaji.clastix.io"
@@ -209,7 +263,6 @@ rules:
209263
verbs:
210264
- get
211265
- list
212-
- watch
213266
- delete
214267
- apiGroups:
215268
- ""
@@ -218,7 +271,6 @@ rules:
218271
verbs:
219272
- get
220273
- list
221-
- watch
222274
- delete
223275
# Scoped to the single Kamaji datastore connection Secret so Step 4b can
224276
# strip the leftover finalizer.kamaji.clastix.io/datastore-secret finalizer

‎packages/apps/kubernetes/tests/delete_hook_test.yaml‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,3 +68,124 @@ tests:
6868
- matchRegex:
6969
path: spec.template.spec.containers[0].command[2]
7070
pattern: 'get tenantcontrolplanes\.kamaji\.clastix\.io test-k8s[\s\S]*still present[\s\S]*patch secret test-k8s-datastore-config'
71+
72+
# The remaining tests pin the wall-clock bounds on every delete in the hook.
73+
# kubectl defaults --wait to true and reads --timeout=0 as a week, so a delete
74+
# written with neither flag is one stuck finalizer away from parking this hook
75+
# for 168 hours.
76+
77+
- it: every delete in the hook carries a wall-clock bound
78+
documentSelector:
79+
path: kind
80+
value: Job
81+
asserts:
82+
# Each pattern walks only continuation lines — the ones ending in a
83+
# backslash — so it stays inside the command it names. A cross-line
84+
# wildcard reaches a flag belonging to the next command instead, and RE2,
85+
# which helm-unittest matches with, has no lookahead to exclude the echo
86+
# in between.
87+
#
88+
# None of the three waits, so none of them needs a guard to swallow an
89+
# expired one — which matters, because a guard cannot tell a timeout from
90+
# a rejected request and would report both as cleanup that happened.
91+
#
92+
# Step 3. What matters downstream is the TenantControlPlane, which Step 4
93+
# deletes and waits for; waiting here only adds a second place for a CAPI
94+
# finalizer to stall.
95+
- matchRegex:
96+
path: spec.template.spec.containers[0].command[2]
97+
pattern: 'delete kamajicontrolplanes\.controlplane\.cluster\.x-k8s\.io test-k8s(?:[^\n]*\\\n)*[^\n]*--wait=false'
98+
# Steps 5 and 6: nothing after them reads the result, and a DELETE that
99+
# has been accepted is not made more complete by watching it. --wait=false
100+
# and an expired --wait leave the same cluster state, one of them sooner.
101+
- matchRegex:
102+
path: spec.template.spec.containers[0].command[2]
103+
pattern: 'delete datavolumes(?:[^\n]*\\\n)*[^\n]*--wait=false'
104+
- matchRegex:
105+
path: spec.template.spec.containers[0].command[2]
106+
pattern: 'delete services(?:[^\n]*\\\n)*[^\n]*--wait=false'
107+
# And the two that were already bounded stay bounded. --wait=false and
108+
# --timeout bound the waiting, not the request: --request-timeout is still
109+
# 0 on every call here, so an apiserver that stops answering hangs this
110+
# hook the same as before. That is a different failure from the stuck
111+
# finalizer these bounds are for, and it wants one answer across all the
112+
# calls rather than a flag on the deletes.
113+
- matchRegex:
114+
path: spec.template.spec.containers[0].command[2]
115+
pattern: 'delete helmreleases\.helm\.toolkit\.fluxcd\.io(?:[^\n]*\\\n)*[^\n]*--timeout=[1-9][0-9]*s'
116+
- matchRegex:
117+
path: spec.template.spec.containers[0].command[2]
118+
pattern: 'delete tenantcontrolplanes\.kamaji\.clastix\.io test-k8s(?:[^\n]*\\\n)*[^\n]*--timeout=[1-9][0-9]*s'
119+
120+
- it: cleanup Role keeps watch on exactly the two resources whose deletes wait
121+
documentSelector:
122+
path: kind
123+
value: Role
124+
asserts:
125+
# kubectl opens a watch only on the wait path, so watch is granted to the
126+
# two resources Steps 2 and 4 wait on and to no others. Removing it from
127+
# either of those does not error — the delete hangs instead, which is the
128+
# failure mode that is hardest to read from the outside, so it is pinned
129+
# rather than left to the reviewer of the next RBAC change.
130+
#
131+
# Whole-rule contains, not per-index verb assertions. notContains passes
132+
# when its path does not resolve, so an index-addressed "and nowhere else"
133+
# goes quiet the moment the rules list is reordered or shortened — the
134+
# failure it is written to prevent would take the assertion with it.
135+
# Matching the entire rule pins the verb set in both directions and names
136+
# the resource when it fails; lengthEqual is what closes the remaining
137+
# hole, a sixth rule appended past everything asserted here.
138+
- lengthEqual:
139+
path: rules
140+
count: 6
141+
- contains:
142+
path: rules
143+
content:
144+
apiGroups: ["helm.toolkit.fluxcd.io"]
145+
resources: ["helmreleases"]
146+
verbs: ["get", "list", "watch", "patch", "delete"]
147+
- contains:
148+
path: rules
149+
content:
150+
apiGroups: ["kamaji.clastix.io"]
151+
resources: ["tenantcontrolplanes"]
152+
verbs: ["get", "list", "watch", "delete"]
153+
- contains:
154+
path: rules
155+
content:
156+
apiGroups: ["controlplane.cluster.x-k8s.io"]
157+
resources: ["kamajicontrolplanes"]
158+
verbs: ["get", "list", "delete"]
159+
- contains:
160+
path: rules
161+
content:
162+
apiGroups: ["cdi.kubevirt.io"]
163+
resources: ["datavolumes"]
164+
verbs: ["get", "list", "delete"]
165+
- contains:
166+
path: rules
167+
content:
168+
apiGroups: [""]
169+
resources: ["services"]
170+
verbs: ["get", "list", "delete"]
171+
172+
- it: pre-delete Job makes a single attempt
173+
documentSelector:
174+
path: kind
175+
value: Job
176+
asserts:
177+
# restartPolicy is Never, so the default backoffLimit of 6 allows seven
178+
# pods in series, each re-running every step from the top and each paying
179+
# its own image pull. That outlives the uninstall budget, and helm gives
180+
# up on its own timeout without reporting which step failed.
181+
#
182+
# Zero and not one: both remaining waits are guarded, so an attempt can
183+
# spend its full 240s without failing, and a retry starts from Step 1 with
184+
# the clock already there. On a nodeless tenant it also re-pays Step 4's
185+
# 120s, because attempt 1 leaves the TenantControlPlane still terminating.
186+
- equal:
187+
path: spec.backoffLimit
188+
value: 0
189+
- equal:
190+
path: spec.template.spec.restartPolicy
191+
value: Never

0 commit comments

Comments
 (0)