Skip to content

Commit bf26ce9

Browse files
authored
fix(apps): let the cleanup hooks watch the objects they delete (#4135)
## What this PR does Fixes #4085. `kubectl delete` waits for the object to disappear unless `--wait=false` is given, and that wait is a watch on the object. The post-delete hook Roles of `harbor`, `clickhouse`, `mariadb` and `qdrant` grant `get`, `list` and `delete` only, so every uninstall logs ``` Failed to watch *unstructured.Unstructured: persistentvolumeclaims "..." is forbidden: User "system:serviceaccount:<ns>:<release>-cleanup" cannot watch resource "persistentvolumeclaims" ``` the wait is skipped, and the hook prints `PVC cleanup complete` while the claims are still terminating. Every collection-scoped rule behind a delete in these four hooks now also carries `watch`: `persistentvolumeclaims` in all four; `certificates`, `challenges` and the solver `ingresses` in harbor. The four named Secrets the mariadb hook removes keep `get` and `delete` only: they carry no finalizer, so the delete never reaches the watch, and a `resourceNames` rule does not authorize a collection watch anyway. A real wait needs a bound. kubectl's default is a week, and a claim held by a pod that will not stop, or a stuck detach, would hold the hook until helm-controller's 300s uninstall budget ran out and leave the HelmRelease stuck uninstalling. The PVC deletes now run with `--wait=true --timeout=120s`, so the expiry is reported by the script with the claims still terminating named, and each Job carries `activeDeadlineSeconds: 270` with `terminationGracePeriodSeconds: 10`, the numbers `packages/apps/tenant/templates/cleanup-job.yaml` documents against that budget, as the backstop for a kubectl that never returns. `--wait=false` was not an option: it is exactly the report-complete-while-terminating behaviour this PR removes. The chart unit tests are updated and pass: harbor 21/21, clickhouse 24/24, mariadb 111/111, qdrant 104/104. The behavioural suite `hack/app-cleanup-hooks_test.bats` passes 11/11 against the changed scripts. The second observation in #4085, the harbor hook pod occasionally starting after its ServiceAccount is gone, is a hook-weight question and is not touched here. ### Downstream repositories Hook templates and their tests only: no values, schema, README or defaults change. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(apps): let the harbor, clickhouse, mariadb and qdrant cleanup hooks wait for the deleted objects, with the wait bounded so a stuck claim cannot hang the uninstall ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Enhancements** - Cleanup operations for ClickHouse, Harbor, MariaDB, and Qdrant now wait for persistent volume claim deletion to complete, with a 120-second timeout. - Cleanup jobs are limited to 270 seconds and allow 10 seconds for graceful termination. - Cleanup permissions now support monitoring relevant resources, including persistent volume claims and, where applicable, certificates, challenges, and ingresses. - Named Secret access is limited to retrieval and deletion. - **Tests** - Updated chart validation to verify cleanup timing, deletion behavior, permissions, and restrictions on unrelated resources. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 301a7fe + f886f26 commit bf26ce9

8 files changed

Lines changed: 105 additions & 22 deletions

File tree

‎packages/apps/clickhouse/templates/hooks/cleanup-pvc.yaml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ metadata:
1313
# hook-succeeded so they are reclaimed after a successful uninstall.
1414
"helm.sh/hook-delete-policy": before-hook-creation
1515
spec:
16+
activeDeadlineSeconds: 270
1617
template:
1718
metadata:
1819
labels:
@@ -21,6 +22,7 @@ spec:
2122
spec:
2223
serviceAccountName: "{{ .Release.Name }}-cleanup"
2324
restartPolicy: Never
25+
terminationGracePeriodSeconds: 10
2426
securityContext:
2527
runAsNonRoot: true
2628
runAsUser: 65534
@@ -51,7 +53,7 @@ spec:
5153
- |
5254
set -eu
5355
echo "Deleting orphaned PVCs for {{ .Release.Name }}..."
54-
if ! kubectl delete pvc -n {{ .Release.Namespace }} -l app.kubernetes.io/instance={{ .Release.Name }} --ignore-not-found; then
56+
if ! kubectl delete pvc -n {{ .Release.Namespace }} -l app.kubernetes.io/instance={{ .Release.Name }} --ignore-not-found --wait=true --timeout=120s; then
5557
echo "ERROR: PVC cleanup failed for {{ .Release.Name }}; orphaned PVCs may remain" >&2
5658
exit 1
5759
fi
@@ -87,7 +89,7 @@ metadata:
8789
rules:
8890
- apiGroups: [""]
8991
resources: ["persistentvolumeclaims"]
90-
verbs: ["get", "list", "delete"]
92+
verbs: ["get", "list", "watch", "delete"]
9193
---
9294
apiVersion: rbac.authorization.k8s.io/v1
9395
kind: RoleBinding

‎packages/apps/clickhouse/tests/cleanup_pvc_test.yaml‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ tests:
4545
path: spec.template.spec.containers[0].command[2]
4646
pattern: 'exit 1'
4747

48-
- it: "grants only get/list/delete on persistentvolumeclaims"
48+
- it: "grants get/list/watch/delete on persistentvolumeclaims"
4949
documentSelector:
5050
path: kind
5151
value: Role
@@ -66,8 +66,27 @@ tests:
6666
value:
6767
- get
6868
- list
69+
- watch
6970
- delete
7071

72+
- it: "bounds the cleanup Job so a claim that will not go cannot hang the uninstall"
73+
documentSelector:
74+
path: kind
75+
value: Job
76+
asserts:
77+
- equal:
78+
path: spec.activeDeadlineSeconds
79+
value: 270
80+
- equal:
81+
path: spec.template.spec.terminationGracePeriodSeconds
82+
value: 10
83+
- matchRegex:
84+
path: spec.template.spec.containers[0].command[2]
85+
pattern: 'kubectl delete pvc -n tenant-test -l app\.kubernetes\.io/instance=ch-test --ignore-not-found --wait=true --timeout=120s'
86+
- notMatchRegex:
87+
path: spec.template.spec.containers[0].command[2]
88+
pattern: '--request-timeout='
89+
7190
- it: "binds the cleanup ServiceAccount to the cleanup Role"
7291
documentSelector:
7392
path: kind

‎packages/apps/harbor/templates/hooks/cleanup.yaml‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ spec:
2323
# timeout instead of the hook's own error. Load-bearing only since the script
2424
# gained a non-zero exit; before that the script could not report one.
2525
backoffLimit: 1
26+
activeDeadlineSeconds: 270
2627
template:
2728
metadata:
2829
labels:
@@ -31,6 +32,7 @@ spec:
3132
spec:
3233
serviceAccountName: {{ .Release.Name }}-cleanup
3334
restartPolicy: Never
35+
terminationGracePeriodSeconds: 10
3436
securityContext:
3537
runAsNonRoot: true
3638
runAsUser: 65534
@@ -104,7 +106,7 @@ spec:
104106
# this selector matches only those two kept PVCs.
105107
echo "Deleting orphaned PVCs for {{ .Release.Name }}-system..."
106108
kubectl delete pvc -n {{ .Release.Namespace }} \
107-
-l release={{ .Release.Name }}-system --ignore-not-found \
109+
-l release={{ .Release.Name }}-system --ignore-not-found --wait=true --timeout=120s \
108110
|| fail "failed to delete PVCs matching release={{ .Release.Name }}-system in {{ .Release.Namespace }}; orphaned PVCs may remain"
109111
110112
# Remove the cert-manager Certificate owned by this release's
@@ -201,21 +203,21 @@ metadata:
201203
rules:
202204
- apiGroups: [""]
203205
resources: ["persistentvolumeclaims"]
204-
verbs: ["get", "list", "delete"]
206+
verbs: ["get", "list", "watch", "delete"]
205207
- apiGroups: ["cert-manager.io"]
206208
resources: ["certificates"]
207-
verbs: ["get", "list", "delete"]
209+
verbs: ["get", "list", "watch", "delete"]
208210
# Only Challenges are touched directly (the host-scoped sweep lists and
209211
# deletes them); Orders are cascade-deleted by cert-manager itself when the
210212
# Certificate is removed, so the hook needs no RBAC for them.
211213
- apiGroups: ["acme.cert-manager.io"]
212214
resources: ["challenges"]
213-
verbs: ["get", "list", "delete"]
215+
verbs: ["get", "list", "watch", "delete"]
214216
# We list ACME http01 solver Ingresses (filtered by this release's host)
215217
# and delete the orphaned ones by name.
216218
- apiGroups: ["networking.k8s.io"]
217219
resources: ["ingresses"]
218-
verbs: ["get", "list", "delete"]
220+
verbs: ["get", "list", "watch", "delete"]
219221
---
220222
apiVersion: rbac.authorization.k8s.io/v1
221223
kind: RoleBinding

‎packages/apps/harbor/tests/cleanup_hook_test.yaml‎

Lines changed: 25 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,24 @@ tests:
103103
path: spec.template.spec.containers[0].command[2]
104104
pattern: kubectl delete[^\n]*-l acme.cert-manager.io/http01-solver=true
105105

106+
- it: bounds the cleanup Job so a claim that will not go cannot hang the uninstall
107+
documentSelector:
108+
path: kind
109+
value: Job
110+
asserts:
111+
- equal:
112+
path: spec.activeDeadlineSeconds
113+
value: 270
114+
- equal:
115+
path: spec.template.spec.terminationGracePeriodSeconds
116+
value: 10
117+
- matchRegex:
118+
path: spec.template.spec.containers[0].command[2]
119+
pattern: '-l release=harbor-test-system --ignore-not-found --wait=true --timeout=120s'
120+
- notMatchRegex:
121+
path: spec.template.spec.containers[0].command[2]
122+
pattern: '--request-timeout='
123+
106124
- it: pins the cleanup image by digest and routes it through the images registry
107125
documentSelector:
108126
path: kind
@@ -211,13 +229,13 @@ tests:
211229
content:
212230
apiGroups: [""]
213231
resources: ["persistentvolumeclaims"]
214-
verbs: ["get", "list", "delete"]
232+
verbs: ["get", "list", "watch", "delete"]
215233
- contains:
216234
path: rules
217235
content:
218236
apiGroups: ["cert-manager.io"]
219237
resources: ["certificates"]
220-
verbs: ["get", "list", "delete"]
238+
verbs: ["get", "list", "watch", "delete"]
221239
# Only "challenges" are granted: the host-scoped sweep lists/deletes
222240
# them, while Orders are cascade-deleted by cert-manager itself when the
223241
# Certificate is removed, so the hook needs no RBAC for orders.
@@ -226,35 +244,35 @@ tests:
226244
content:
227245
apiGroups: ["acme.cert-manager.io"]
228246
resources: ["challenges"]
229-
verbs: ["get", "list", "delete"]
247+
verbs: ["get", "list", "watch", "delete"]
230248
- notContains:
231249
path: rules
232250
content:
233251
apiGroups: ["acme.cert-manager.io"]
234252
resources: ["orders", "challenges"]
235-
verbs: ["get", "list", "delete"]
253+
verbs: ["get", "list", "watch", "delete"]
236254
# We list+delete orphaned http01 solver Ingresses by name, so we need
237255
# get/list/delete on networking.k8s.io ingresses.
238256
- contains:
239257
path: rules
240258
content:
241259
apiGroups: ["networking.k8s.io"]
242260
resources: ["ingresses"]
243-
verbs: ["get", "list", "delete"]
261+
verbs: ["get", "list", "watch", "delete"]
244262
# The hook must not grant services/pods: solver Pods/Services cascade
245263
# from the Challenge/Certificate and are never deleted directly.
246264
- notContains:
247265
path: rules
248266
content:
249267
apiGroups: [""]
250268
resources: ["services"]
251-
verbs: ["get", "list", "delete"]
269+
verbs: ["get", "list", "watch", "delete"]
252270
- notContains:
253271
path: rules
254272
content:
255273
apiGroups: [""]
256274
resources: ["pods"]
257-
verbs: ["get", "list", "delete"]
275+
verbs: ["get", "list", "watch", "delete"]
258276

259277
- it: binds the Role to the cleanup ServiceAccount
260278
documentSelector:

‎packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ metadata:
1010
"helm.sh/hook-weight": "10"
1111
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
1212
spec:
13+
activeDeadlineSeconds: 270
1314
template:
1415
metadata:
1516
labels:
@@ -18,6 +19,7 @@ spec:
1819
spec:
1920
serviceAccountName: {{ .Release.Name }}-cleanup
2021
restartPolicy: Never
22+
terminationGracePeriodSeconds: 10
2123
securityContext:
2224
runAsNonRoot: true
2325
runAsUser: 65534
@@ -58,7 +60,7 @@ spec:
5860
# errors behind `|| echo ...`.
5961
cleanup_failed=0
6062
echo "Deleting orphaned PVCs for {{ .Release.Name }}..."
61-
if kubectl delete pvc -n {{ .Release.Namespace }} -l app.kubernetes.io/instance={{ .Release.Name }} --ignore-not-found; then
63+
if kubectl delete pvc -n {{ .Release.Namespace }} -l app.kubernetes.io/instance={{ .Release.Name }} --ignore-not-found --wait=true --timeout=120s; then
6264
echo "PVC cleanup complete."
6365
else
6466
echo "WARNING: PVC cleanup failed for {{ .Release.Name }}; orphaned PVCs may remain" >&2
@@ -113,7 +115,7 @@ rules:
113115
# PVCs are deleted by label selector, which requires list.
114116
- apiGroups: [""]
115117
resources: ["persistentvolumeclaims"]
116-
verbs: ["get", "list", "delete"]
118+
verbs: ["get", "list", "watch", "delete"]
117119
# The Secrets left behind are deleted by name — the first two generated by the
118120
# operator, the last two by cert-manager — so the rule is scoped to those four
119121
# rather than to namespace-wide secret access.

‎packages/apps/mariadb/tests/cleanup_hook_test.yaml‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,25 @@ tests:
5959
path: spec.template.spec.containers[0].command[2]
6060
pattern: 'exit "\$\{cleanup_failed\}"'
6161

62-
- it: "cleanup Role grants get/list/delete on persistentvolumeclaims"
62+
- it: "bounds the cleanup Job so a claim that will not go cannot hang the uninstall"
63+
documentSelector:
64+
path: kind
65+
value: Job
66+
asserts:
67+
- equal:
68+
path: spec.activeDeadlineSeconds
69+
value: 270
70+
- equal:
71+
path: spec.template.spec.terminationGracePeriodSeconds
72+
value: 10
73+
- matchRegex:
74+
path: spec.template.spec.containers[0].command[2]
75+
pattern: 'kubectl delete pvc -n tenant-test -l app\.kubernetes\.io/instance=mariadb-test --ignore-not-found --wait=true --timeout=120s'
76+
- notMatchRegex:
77+
path: spec.template.spec.containers[0].command[2]
78+
pattern: '--request-timeout='
79+
80+
- it: "cleanup Role grants get/list/watch/delete on persistentvolumeclaims"
6381
documentSelector:
6482
path: kind
6583
value: Role
@@ -69,7 +87,7 @@ tests:
6987
content:
7088
apiGroups: [""]
7189
resources: ["persistentvolumeclaims"]
72-
verbs: ["get", "list", "delete"]
90+
verbs: ["get", "list", "watch", "delete"]
7391

7492
- it: "cleanup Role scopes secret access to the named secrets only (least privilege)"
7593
documentSelector:

‎packages/apps/qdrant/templates/hooks/cleanup-pvc.yaml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ spec:
2222
# timeout instead of the hook's own error. Load-bearing only since the script
2323
# gained a non-zero exit; before that the script could not report one.
2424
backoffLimit: 1
25+
activeDeadlineSeconds: 270
2526
template:
2627
metadata:
2728
labels:
@@ -30,6 +31,7 @@ spec:
3031
spec:
3132
serviceAccountName: {{ .Release.Name }}-qdrant-cleanup
3233
restartPolicy: Never
34+
terminationGracePeriodSeconds: 10
3335
securityContext:
3436
runAsNonRoot: true
3537
runAsUser: 65534
@@ -104,7 +106,7 @@ spec:
104106
if [ -z "$pvcs" ]; then
105107
echo "No data PVCs found for release {{ .Release.Name }}; nothing to delete."
106108
else
107-
echo "$pvcs" | xargs kubectl delete -n {{ .Release.Namespace }} --ignore-not-found \
109+
echo "$pvcs" | xargs kubectl delete -n {{ .Release.Namespace }} --ignore-not-found --wait=true --timeout=120s \
108110
|| fail "failed to delete data PVCs of {{ .Release.Name }} in {{ .Release.Namespace }} ($(echo "$pvcs" | tr '\n' ' ')); orphaned PVCs may remain"
109111
fi
110112
else
@@ -147,7 +149,7 @@ metadata:
147149
rules:
148150
- apiGroups: [""]
149151
resources: ["persistentvolumeclaims"]
150-
verbs: ["get", "list", "delete"]
152+
verbs: ["get", "list", "watch", "delete"]
151153
---
152154
apiVersion: rbac.authorization.k8s.io/v1
153155
kind: RoleBinding

‎packages/apps/qdrant/tests/cleanup_pvc_test.yaml‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,26 @@ tests:
6565
path: spec.template.spec.containers[0].command[2]
6666
pattern: 'grep "\^persistentvolumeclaim/qdrant-storage-qdrant-test-"'
6767

68+
- it: "bounds the cleanup Job so a claim that will not go cannot hang the uninstall"
69+
templates:
70+
- templates/hooks/cleanup-pvc.yaml
71+
documentSelector:
72+
path: kind
73+
value: Job
74+
asserts:
75+
- equal:
76+
path: spec.activeDeadlineSeconds
77+
value: 270
78+
- equal:
79+
path: spec.template.spec.terminationGracePeriodSeconds
80+
value: 10
81+
- matchRegex:
82+
path: spec.template.spec.containers[0].command[2]
83+
pattern: 'xargs kubectl delete -n tenant-test --ignore-not-found --wait=true --timeout=120s'
84+
- notMatchRegex:
85+
path: spec.template.spec.containers[0].command[2]
86+
pattern: '--request-timeout='
87+
6888
- it: "creates qdrant-scoped RBAC for the cleanup hook"
6989
templates:
7090
- templates/hooks/cleanup-pvc.yaml
@@ -80,4 +100,4 @@ tests:
80100
content:
81101
apiGroups: [""]
82102
resources: ["persistentvolumeclaims"]
83-
verbs: ["get", "list", "delete"]
103+
verbs: ["get", "list", "watch", "delete"]

0 commit comments

Comments
 (0)