Skip to content

Commit a92f060

Browse files
scooby87claude
andcommitted
fix(etcd): clean up orphaned data PVCs and TLS secrets on tenant module delete (#3090)
When the etcd tenant module is disabled (etcd: false), the etcd-operator StatefulSet data PVCs (data-etcd-<ordinal>) are never garbage-collected, and the five etcd-*-tls secrets are left orphaned in the tenant namespace: etcd-ca-tls / etcd-client-tls are pre-install hooks this chart marks with helm.sh/resource-policy: keep, and etcd-peer-ca-tls / etcd-server-tls / etcd-peer-tls are cert-manager-issued secrets with no owner reference back to the chart. None are reclaimed by the Helm uninstall. Add a post-delete cleanup Job (modeled on the seaweedfs/monitoring sibling fixes) that deletes the data PVCs by name (one per replica) and the five TLS secrets by name. check-release-name pins the release to "etcd", so every target name is deterministic and no sibling workload shares the data-etcd-* prefix. The Job runs non-root with a read-only root fs, dropped capabilities and seccompProfile RuntimeDefault; its RBAC is scoped to PVC get/list/delete and the five secrets by resourceNames. Deletes use --ignore-not-found and failures are logged but never block teardown. The stuck-HelmRelease finalizer noted in the issue is the platform-wide sharded-helm-controller side-bug tracked separately; this change addresses the PVC and secret orphans the etcd chart owns. Assisted-By: Claude <[email protected]> Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Signed-off-by: Алексей Артамонов <[email protected]>
1 parent fdbe6b2 commit a92f060

3 files changed

Lines changed: 302 additions & 0 deletions

File tree

‎packages/extra/etcd/Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,5 +6,6 @@ generate:
66
cozyvalues-gen -v values.yaml -s values.schema.json -r README.md
77
../../../hack/update-crd.sh
88

9+
.PHONY: test
910
test:
1011
helm unittest .
Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
---
2+
apiVersion: v1
3+
kind: ServiceAccount
4+
metadata:
5+
name: {{ .Release.Name }}-cleanup
6+
labels:
7+
app.kubernetes.io/instance: {{ .Release.Name }}
8+
annotations:
9+
"helm.sh/hook": post-delete
10+
"helm.sh/hook-weight": "0"
11+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
12+
---
13+
apiVersion: rbac.authorization.k8s.io/v1
14+
kind: Role
15+
metadata:
16+
name: {{ .Release.Name }}-cleanup
17+
labels:
18+
app.kubernetes.io/instance: {{ .Release.Name }}
19+
annotations:
20+
"helm.sh/hook": post-delete
21+
"helm.sh/hook-weight": "5"
22+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
23+
rules:
24+
- apiGroups: [""]
25+
resources: ["persistentvolumeclaims"]
26+
verbs: ["get", "list", "delete"]
27+
# cert-manager TLS secrets carry only the cluster-wide
28+
# controller.cert-manager.io/fao label, and the two pre-install "keep"
29+
# secrets carry none, so we grant delete by explicit resourceNames rather
30+
# than an unsafe label sweep that could hit a sibling app's secrets.
31+
- apiGroups: [""]
32+
resources: ["secrets"]
33+
verbs: ["get", "delete"]
34+
resourceNames:
35+
- etcd-peer-ca-tls
36+
- etcd-ca-tls
37+
- etcd-server-tls
38+
- etcd-peer-tls
39+
- etcd-client-tls
40+
---
41+
apiVersion: rbac.authorization.k8s.io/v1
42+
kind: RoleBinding
43+
metadata:
44+
name: {{ .Release.Name }}-cleanup
45+
labels:
46+
app.kubernetes.io/instance: {{ .Release.Name }}
47+
annotations:
48+
"helm.sh/hook": post-delete
49+
"helm.sh/hook-weight": "5"
50+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
51+
roleRef:
52+
apiGroup: rbac.authorization.k8s.io
53+
kind: Role
54+
name: {{ .Release.Name }}-cleanup
55+
subjects:
56+
- kind: ServiceAccount
57+
name: {{ .Release.Name }}-cleanup
58+
namespace: {{ .Release.Namespace }}
59+
---
60+
apiVersion: batch/v1
61+
kind: Job
62+
metadata:
63+
name: {{ .Release.Name }}-cleanup
64+
labels:
65+
app.kubernetes.io/instance: {{ .Release.Name }}
66+
annotations:
67+
"helm.sh/hook": post-delete
68+
"helm.sh/hook-weight": "10"
69+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
70+
spec:
71+
backoffLimit: 3
72+
activeDeadlineSeconds: 120
73+
template:
74+
metadata:
75+
labels:
76+
app.kubernetes.io/instance: {{ .Release.Name }}
77+
policy.cozystack.io/allow-to-apiserver: "true"
78+
spec:
79+
serviceAccountName: {{ .Release.Name }}-cleanup
80+
restartPolicy: Never
81+
securityContext:
82+
runAsNonRoot: true
83+
runAsUser: 65534
84+
runAsGroup: 65534
85+
fsGroup: 65534
86+
seccompProfile:
87+
type: RuntimeDefault
88+
volumes:
89+
- name: tmp
90+
emptyDir: {}
91+
containers:
92+
- name: cleanup
93+
image: docker.io/clastix/kubectl:v1.32
94+
env:
95+
- name: HOME
96+
value: /tmp
97+
securityContext:
98+
allowPrivilegeEscalation: false
99+
readOnlyRootFilesystem: true
100+
capabilities:
101+
drop:
102+
- ALL
103+
volumeMounts:
104+
- name: tmp
105+
mountPath: /tmp
106+
resources:
107+
requests:
108+
cpu: 10m
109+
memory: 64Mi
110+
limits:
111+
cpu: 200m
112+
memory: 128Mi
113+
command:
114+
- /bin/sh
115+
- -c
116+
- |
117+
# The EtcdCluster is named "etcd" (check-release-name pins the
118+
# release name to "etcd"), so its StatefulSet volumeClaimTemplate
119+
# PVCs are named data-etcd-<ordinal>, one per replica. The
120+
# etcd-operator does not garbage-collect them on EtcdCluster
121+
# deletion, so we delete them here by name (safe: no sibling
122+
# workload shares the data-etcd-* prefix in a tenant namespace).
123+
echo "Deleting orphaned etcd data PVCs for {{ .Release.Name }}..."
124+
kubectl delete pvc -n {{ .Release.Namespace }} {{ range $i := until (int .Values.replicas) }}data-etcd-{{ $i }} {{ end }}--ignore-not-found --wait=false \
125+
|| echo "WARNING: etcd PVC cleanup failed for {{ .Release.Name }}; orphaned PVCs may remain" >&2
126+
# etcd-ca-tls / etcd-client-tls are pre-install "keep" secrets
127+
# this chart owns; etcd-peer-ca-tls / etcd-server-tls /
128+
# etcd-peer-tls are cert-manager-issued secrets with no owner
129+
# reference. None are reclaimed by the uninstall, so delete them
130+
# by name.
131+
echo "Deleting orphaned etcd TLS secrets for {{ .Release.Name }}..."
132+
kubectl delete secret -n {{ .Release.Namespace }} etcd-peer-ca-tls etcd-ca-tls etcd-server-tls etcd-peer-tls etcd-client-tls --ignore-not-found --wait=false \
133+
|| echo "WARNING: etcd secret cleanup failed for {{ .Release.Name }}; orphaned secrets may remain" >&2
134+
echo "etcd cleanup complete."
Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
suite: etcd post-delete cleanup hook removes orphaned PVCs and TLS secrets
2+
templates:
3+
- templates/hooks/cleanup.yaml
4+
5+
release:
6+
name: etcd
7+
namespace: tenant-root
8+
9+
# After the etcd tenant module is disabled (etcd: false), the etcd-operator
10+
# StatefulSet data PVCs (data-etcd-{0,1,2}) are never garbage-collected, and
11+
# the five etcd-*-tls secrets are left orphaned in the tenant namespace:
12+
# - etcd-ca-tls / etcd-client-tls are created by this chart as pre-install
13+
# hooks with helm.sh/resource-policy: keep, so Helm deliberately keeps them.
14+
# - etcd-peer-ca-tls / etcd-server-tls / etcd-peer-tls are populated by
15+
# cert-manager from Certificate resources and carry no owner reference back
16+
# to the chart, so they survive the uninstall.
17+
# The check-release-name.tpl guard pins .Release.Name to "etcd", so every
18+
# target name is deterministic. This post-delete hook renders a Job (plus its
19+
# RBAC) that deletes the data PVCs by name (one per replica) and the five
20+
# secrets by name.
21+
22+
tests:
23+
- it: renders the cleanup ServiceAccount as a post-delete hook
24+
documentSelector:
25+
path: kind
26+
value: ServiceAccount
27+
asserts:
28+
- equal:
29+
path: metadata.name
30+
value: etcd-cleanup
31+
- equal:
32+
path: metadata.annotations["helm.sh/hook"]
33+
value: post-delete
34+
- equal:
35+
path: metadata.annotations["helm.sh/hook-weight"]
36+
value: "0"
37+
- equal:
38+
path: metadata.annotations["helm.sh/hook-delete-policy"]
39+
value: before-hook-creation,hook-succeeded
40+
41+
- it: grants delete on PVCs and only the 5 named TLS secrets
42+
documentSelector:
43+
path: kind
44+
value: Role
45+
asserts:
46+
- equal:
47+
path: metadata.annotations["helm.sh/hook-weight"]
48+
value: "5"
49+
- contains:
50+
path: rules
51+
content:
52+
apiGroups: [""]
53+
resources: ["persistentvolumeclaims"]
54+
verbs: ["get", "list", "delete"]
55+
- contains:
56+
path: rules
57+
content:
58+
apiGroups: [""]
59+
resources: ["secrets"]
60+
verbs: ["get", "delete"]
61+
resourceNames:
62+
- etcd-peer-ca-tls
63+
- etcd-ca-tls
64+
- etcd-server-tls
65+
- etcd-peer-tls
66+
- etcd-client-tls
67+
68+
- it: binds the cleanup Role to the cleanup ServiceAccount
69+
documentSelector:
70+
path: kind
71+
value: RoleBinding
72+
asserts:
73+
- equal:
74+
path: metadata.annotations["helm.sh/hook-weight"]
75+
value: "5"
76+
- equal:
77+
path: roleRef.name
78+
value: etcd-cleanup
79+
- equal:
80+
path: subjects[0].kind
81+
value: ServiceAccount
82+
- equal:
83+
path: subjects[0].name
84+
value: etcd-cleanup
85+
86+
- it: renders a hardened cleanup Job with the right hook metadata
87+
documentSelector:
88+
path: kind
89+
value: Job
90+
asserts:
91+
- equal:
92+
path: metadata.name
93+
value: etcd-cleanup
94+
- equal:
95+
path: metadata.annotations["helm.sh/hook"]
96+
value: post-delete
97+
- equal:
98+
path: metadata.annotations["helm.sh/hook-weight"]
99+
value: "10"
100+
- equal:
101+
path: metadata.annotations["helm.sh/hook-delete-policy"]
102+
value: before-hook-creation,hook-succeeded
103+
- equal:
104+
path: spec.activeDeadlineSeconds
105+
value: 120
106+
- equal:
107+
path: spec.template.metadata.labels["policy.cozystack.io/allow-to-apiserver"]
108+
value: "true"
109+
- equal:
110+
path: spec.template.spec.serviceAccountName
111+
value: etcd-cleanup
112+
- equal:
113+
path: spec.template.spec.containers[0].image
114+
value: docker.io/clastix/kubectl:v1.32
115+
- equal:
116+
path: spec.template.spec.securityContext.runAsNonRoot
117+
value: true
118+
- equal:
119+
path: spec.template.spec.securityContext.runAsUser
120+
value: 65534
121+
- equal:
122+
path: spec.template.spec.securityContext.seccompProfile.type
123+
value: RuntimeDefault
124+
- equal:
125+
path: spec.template.spec.containers[0].securityContext.readOnlyRootFilesystem
126+
value: true
127+
- equal:
128+
path: spec.template.spec.containers[0].securityContext.allowPrivilegeEscalation
129+
value: false
130+
- contains:
131+
path: spec.template.spec.containers[0].securityContext.capabilities.drop
132+
content: ALL
133+
- equal:
134+
path: spec.template.spec.containers[0].resources.requests.cpu
135+
value: 10m
136+
- equal:
137+
path: spec.template.spec.containers[0].resources.requests.memory
138+
value: 64Mi
139+
- equal:
140+
path: spec.template.spec.containers[0].resources.limits.cpu
141+
value: 200m
142+
- equal:
143+
path: spec.template.spec.containers[0].resources.limits.memory
144+
value: 128Mi
145+
146+
- it: deletes one data PVC per replica and the five TLS secrets by name
147+
documentSelector:
148+
path: kind
149+
value: Job
150+
asserts:
151+
- matchRegex:
152+
path: spec.template.spec.containers[0].command[2]
153+
pattern: 'kubectl delete pvc -n tenant-root data-etcd-0 data-etcd-1 data-etcd-2 --ignore-not-found --wait=false'
154+
- matchRegex:
155+
path: spec.template.spec.containers[0].command[2]
156+
pattern: 'kubectl delete secret -n tenant-root etcd-peer-ca-tls etcd-ca-tls etcd-server-tls etcd-peer-tls etcd-client-tls --ignore-not-found --wait=false'
157+
158+
- it: scales the PVC list to the configured replica count
159+
documentSelector:
160+
path: kind
161+
value: Job
162+
set:
163+
replicas: 5
164+
asserts:
165+
- matchRegex:
166+
path: spec.template.spec.containers[0].command[2]
167+
pattern: 'kubectl delete pvc -n tenant-root data-etcd-0 data-etcd-1 data-etcd-2 data-etcd-3 data-etcd-4 --ignore-not-found --wait=false'

0 commit comments

Comments
 (0)