Skip to content

Commit 3d1e111

Browse files
committed
fix(clickhouse)!: clean up orphaned keeper and data PVCs on delete (#3057)
After deleting a ClickHouse app CR, the keeper PVC and the main data/log PVCs were left orphaned in the tenant namespace, consuming storage permanently. The clickhouse-keeper-operator does not delete data PVCs on CR removal (data-safety default) and the CRD exposes no persistentVolumeClaimRetentionPolicy equivalent. Add a post-delete cleanup hook (modeled on the MariaDB hook) that deletes every PVC carrying the release label app.kubernetes.io/instance=<release>. Also stamp that label on the keeper volumeClaimTemplates in chkeeper.yaml: the keeper PVC previously had no release label (the chart labelled only the keeper pod template, not the PVC), so a single selector would have missed the very PVC the issue reports. BREAKING CHANGE: deleting a ClickHouse app now permanently removes all of its PVCs (keeper + data + logs) and the data they hold. Nothing is left behind to recover from. This matches the existing MariaDB cleanup-hook behaviour and the general cozystack model where deleting a managed app reclaims its storage. Signed-off-by: Алексей Артамонов <[email protected]>
1 parent 374cac1 commit 3d1e111

3 files changed

Lines changed: 150 additions & 0 deletions

File tree

‎packages/apps/clickhouse/templates/chkeeper.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,9 @@ spec:
6464

6565
volumeClaimTemplates:
6666
- name: default
67+
metadata:
68+
labels:
69+
app.kubernetes.io/instance: "{{ .Release.Name }}"
6770
spec:
6871
accessModes:
6972
- ReadWriteOnce
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
---
2+
apiVersion: batch/v1
3+
kind: Job
4+
metadata:
5+
name: {{ .Release.Name }}-cleanup
6+
labels:
7+
app.kubernetes.io/instance: {{ .Release.Name }}
8+
annotations:
9+
"helm.sh/hook": post-delete
10+
"helm.sh/hook-weight": "10"
11+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
12+
spec:
13+
template:
14+
metadata:
15+
labels:
16+
app.kubernetes.io/instance: {{ .Release.Name }}
17+
policy.cozystack.io/allow-to-apiserver: "true"
18+
spec:
19+
serviceAccountName: {{ .Release.Name }}-cleanup
20+
restartPolicy: Never
21+
containers:
22+
- name: cleanup
23+
image: docker.io/clastix/kubectl:v1.32
24+
command:
25+
- /bin/sh
26+
- -c
27+
- |
28+
echo "Deleting orphaned PVCs for {{ .Release.Name }}..."
29+
kubectl delete pvc -n {{ .Release.Namespace }} -l app.kubernetes.io/instance={{ .Release.Name }} || true
30+
echo "PVC cleanup complete."
31+
---
32+
apiVersion: v1
33+
kind: ServiceAccount
34+
metadata:
35+
name: {{ .Release.Name }}-cleanup
36+
labels:
37+
app.kubernetes.io/instance: {{ .Release.Name }}
38+
annotations:
39+
"helm.sh/hook": post-delete
40+
helm.sh/hook-weight: "0"
41+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
42+
---
43+
apiVersion: rbac.authorization.k8s.io/v1
44+
kind: Role
45+
metadata:
46+
name: {{ .Release.Name }}-cleanup
47+
labels:
48+
app.kubernetes.io/instance: {{ .Release.Name }}
49+
annotations:
50+
"helm.sh/hook": post-delete
51+
"helm.sh/hook-weight": "5"
52+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
53+
rules:
54+
- apiGroups: [""]
55+
resources: ["persistentvolumeclaims"]
56+
verbs: ["get", "list", "delete"]
57+
---
58+
apiVersion: rbac.authorization.k8s.io/v1
59+
kind: RoleBinding
60+
metadata:
61+
name: {{ .Release.Name }}-cleanup
62+
labels:
63+
app.kubernetes.io/instance: {{ .Release.Name }}
64+
annotations:
65+
"helm.sh/hook": post-delete
66+
helm.sh/hook-weight: "5"
67+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
68+
roleRef:
69+
apiGroup: rbac.authorization.k8s.io
70+
kind: Role
71+
name: {{ .Release.Name }}-cleanup
72+
subjects:
73+
- kind: ServiceAccount
74+
name: {{ .Release.Name }}-cleanup
Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
suite: post-delete PVC cleanup hook
2+
3+
# Verifies the post-delete hook that deletes orphaned PVCs left behind by the
4+
# clickhouse-keeper-operator (and the ClickHouse operator), which do not delete
5+
# data PVCs when their CR is removed. The cleanup Job selects PVCs by the
6+
# release-scoped label app.kubernetes.io/instance, which is set on both the
7+
# keeper volumeClaimTemplates and the main ClickHouse data/log volumeClaimTemplates.
8+
9+
release:
10+
name: ch-test
11+
namespace: tenant-test
12+
13+
set:
14+
_cluster:
15+
cluster-domain: cluster.local
16+
17+
templates:
18+
- templates/hooks/cleanup-pvc.yaml
19+
20+
tests:
21+
- it: "renders a post-delete cleanup Job scoped to the release"
22+
documentSelector:
23+
path: kind
24+
value: Job
25+
asserts:
26+
- equal:
27+
path: metadata.name
28+
value: ch-test-cleanup
29+
- equal:
30+
path: metadata.annotations["helm.sh/hook"]
31+
value: post-delete
32+
- equal:
33+
path: metadata.annotations["helm.sh/hook-delete-policy"]
34+
value: before-hook-creation,hook-succeeded
35+
- equal:
36+
path: spec.template.spec.serviceAccountName
37+
value: ch-test-cleanup
38+
- matchRegex:
39+
path: spec.template.spec.containers[0].command[2]
40+
pattern: "kubectl delete pvc -n tenant-test -l app.kubernetes.io/instance=ch-test"
41+
42+
- it: "grants only get/list/delete on persistentvolumeclaims"
43+
documentSelector:
44+
path: kind
45+
value: Role
46+
asserts:
47+
- equal:
48+
path: rules[0].apiGroups[0]
49+
value: ""
50+
- equal:
51+
path: rules[0].resources[0]
52+
value: persistentvolumeclaims
53+
- equal:
54+
path: rules[0].verbs
55+
value:
56+
- get
57+
- list
58+
- delete
59+
60+
- it: "binds the cleanup ServiceAccount to the cleanup Role"
61+
documentSelector:
62+
path: kind
63+
value: RoleBinding
64+
asserts:
65+
- equal:
66+
path: roleRef.name
67+
value: ch-test-cleanup
68+
- equal:
69+
path: subjects[0].kind
70+
value: ServiceAccount
71+
- equal:
72+
path: subjects[0].name
73+
value: ch-test-cleanup

0 commit comments

Comments
 (0)