Skip to content

Commit 5dbb352

Browse files
committed
fix(bucket): clean up orphaned cert-manager ACME solver resources on delete (#3060)
cert-manager ACME http01-solver Ingress/Pod/Service objects were left orphaned after a Bucket app CR was deleted (a stuck challenge leaves them without a live owner to cascade-delete). Add a post-delete hook that deletes the release Certificate/Order/Challenge and sweeps leftover solver objects by the acme.cert-manager.io/http01-solver label. RBAC verified against the vendored cert-manager CRDs. The cleanup Job runs non-root with a read-only root fs and dropped capabilities; failures are logged but never block teardown. Signed-off-by: Алексей Артамонов <[email protected]>
1 parent 374cac1 commit 5dbb352

3 files changed

Lines changed: 238 additions & 0 deletions

File tree

‎packages/apps/bucket/Makefile‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
include ../../../hack/package.mk
22

3+
.PHONY: test
4+
test:
5+
helm unittest .
6+
37
generate:
48
cozyvalues-gen -m 'bucket' -v values.yaml -s values.schema.json -r README.md -g ../../../api/apps/v1alpha1/bucket/types.go
59
../../../hack/update-crd.sh
Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
---
2+
apiVersion: batch/v1
3+
kind: Job
4+
metadata:
5+
name: {{ .Release.Name }}-cleanup-acme
6+
labels:
7+
app.kubernetes.io/instance: {{ .Release.Name }}
8+
annotations:
9+
"helm.sh/hook": post-delete
10+
"helm.sh/hook-weight": "10"
11+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
12+
spec:
13+
template:
14+
metadata:
15+
labels:
16+
app.kubernetes.io/instance: {{ .Release.Name }}
17+
policy.cozystack.io/allow-to-apiserver: "true"
18+
spec:
19+
serviceAccountName: {{ .Release.Name }}-cleanup-acme
20+
restartPolicy: Never
21+
securityContext:
22+
runAsNonRoot: true
23+
runAsUser: 65534
24+
runAsGroup: 65534
25+
seccompProfile:
26+
type: RuntimeDefault
27+
containers:
28+
- name: cleanup
29+
image: docker.io/clastix/kubectl:v1.32
30+
securityContext:
31+
allowPrivilegeEscalation: false
32+
readOnlyRootFilesystem: true
33+
capabilities:
34+
drop: ["ALL"]
35+
resources:
36+
requests:
37+
cpu: 10m
38+
memory: 64Mi
39+
limits:
40+
cpu: 200m
41+
memory: 128Mi
42+
env:
43+
- name: HOME
44+
value: /tmp
45+
command:
46+
- /bin/sh
47+
- -c
48+
- |
49+
echo "Cleaning up orphaned cert-manager ACME resources for {{ .Release.Name }}..."
50+
# Delete the per-bucket Certificate/Order/Challenge. cert-manager
51+
# then cascade-removes the cm-acme-http-solver-* solver objects it
52+
# owns. The release's TLS Certificate is named after the secret it
53+
# populates ({{ .Release.Name }}-ui-tls).
54+
kubectl delete certificate {{ .Release.Name }}-ui-tls -n {{ .Release.Namespace }} --ignore-not-found \
55+
|| echo "WARNING: cert-manager cleanup failed for {{ .Release.Name }}; orphaned ACME resources may remain" >&2
56+
kubectl delete certificate,order,challenge -n {{ .Release.Namespace }} -l app.kubernetes.io/instance={{ .Release.Name }} --ignore-not-found \
57+
|| echo "WARNING: cert-manager cleanup failed for {{ .Release.Name }}; orphaned ACME resources may remain" >&2
58+
# NOTE: truly orphaned (owner-less) http01 solver Ingress/Service/Pod
59+
# objects are intentionally NOT swept namespace-wide. The
60+
# acme.cert-manager.io/http01-solver=true label is set on EVERY app's
61+
# solver objects in this tenant namespace, so a blanket sweep would
62+
# delete sibling apps' in-flight ACME challenge solvers. We rely on
63+
# cert-manager cascade-deleting the solvers it owns when the
64+
# release-scoped Certificate/Order/Challenge above are removed.
65+
echo "ACME cleanup complete."
66+
volumeMounts:
67+
- name: tmp
68+
mountPath: /tmp
69+
volumes:
70+
- name: tmp
71+
emptyDir: {}
72+
---
73+
apiVersion: v1
74+
kind: ServiceAccount
75+
metadata:
76+
name: {{ .Release.Name }}-cleanup-acme
77+
labels:
78+
app.kubernetes.io/instance: {{ .Release.Name }}
79+
annotations:
80+
"helm.sh/hook": post-delete
81+
"helm.sh/hook-weight": "0"
82+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
83+
---
84+
apiVersion: rbac.authorization.k8s.io/v1
85+
kind: Role
86+
metadata:
87+
name: {{ .Release.Name }}-cleanup-acme
88+
labels:
89+
app.kubernetes.io/instance: {{ .Release.Name }}
90+
annotations:
91+
"helm.sh/hook": post-delete
92+
"helm.sh/hook-weight": "5"
93+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
94+
rules:
95+
- apiGroups: ["cert-manager.io"]
96+
resources: ["certificates"]
97+
verbs: ["get", "list", "delete"]
98+
- apiGroups: ["acme.cert-manager.io"]
99+
resources: ["orders", "challenges"]
100+
verbs: ["get", "list", "delete"]
101+
---
102+
apiVersion: rbac.authorization.k8s.io/v1
103+
kind: RoleBinding
104+
metadata:
105+
name: {{ .Release.Name }}-cleanup-acme
106+
labels:
107+
app.kubernetes.io/instance: {{ .Release.Name }}
108+
annotations:
109+
"helm.sh/hook": post-delete
110+
"helm.sh/hook-weight": "5"
111+
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
112+
roleRef:
113+
apiGroup: rbac.authorization.k8s.io
114+
kind: Role
115+
name: {{ .Release.Name }}-cleanup-acme
116+
subjects:
117+
- kind: ServiceAccount
118+
name: {{ .Release.Name }}-cleanup-acme
Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
suite: bucket post-delete ACME cleanup hook
2+
templates:
3+
- templates/hooks/cleanup-acme.yaml
4+
5+
release:
6+
name: my-bucket
7+
namespace: tenant-root
8+
9+
tests:
10+
- it: renders a post-delete cleanup Job with the expected name and SA
11+
documentIndex: 0
12+
asserts:
13+
- equal:
14+
path: kind
15+
value: Job
16+
- equal:
17+
path: metadata.name
18+
value: my-bucket-cleanup-acme
19+
- equal:
20+
path: metadata.annotations["helm.sh/hook"]
21+
value: post-delete
22+
- equal:
23+
path: spec.template.spec.serviceAccountName
24+
value: my-bucket-cleanup-acme
25+
- equal:
26+
path: spec.template.spec.restartPolicy
27+
value: Never
28+
- equal:
29+
path: spec.template.metadata.labels["policy.cozystack.io/allow-to-apiserver"]
30+
value: "true"
31+
32+
- it: deletes release-scoped cert-manager resources without a namespace-wide solver sweep
33+
documentIndex: 0
34+
asserts:
35+
- matchRegex:
36+
path: spec.template.spec.containers[0].command[2]
37+
pattern: "kubectl delete certificate my-bucket-ui-tls -n tenant-root"
38+
- matchRegex:
39+
path: spec.template.spec.containers[0].command[2]
40+
pattern: "kubectl delete certificate,order,challenge -n tenant-root -l app.kubernetes.io/instance=my-bucket"
41+
- notMatchRegex:
42+
path: spec.template.spec.containers[0].command[2]
43+
pattern: "kubectl delete ingress,svc,pod"
44+
45+
- it: sets resource requests and limits on the cleanup container
46+
documentIndex: 0
47+
asserts:
48+
- equal:
49+
path: spec.template.spec.containers[0].resources.requests.cpu
50+
value: 10m
51+
- equal:
52+
path: spec.template.spec.containers[0].resources.requests.memory
53+
value: 64Mi
54+
- equal:
55+
path: spec.template.spec.containers[0].resources.limits.cpu
56+
value: 200m
57+
- equal:
58+
path: spec.template.spec.containers[0].resources.limits.memory
59+
value: 128Mi
60+
61+
- it: creates a ServiceAccount for the cleanup job
62+
documentIndex: 1
63+
asserts:
64+
- equal:
65+
path: kind
66+
value: ServiceAccount
67+
- equal:
68+
path: metadata.name
69+
value: my-bucket-cleanup-acme
70+
71+
- it: grants RBAC limited to cert-manager and acme resources
72+
documentIndex: 2
73+
asserts:
74+
- equal:
75+
path: kind
76+
value: Role
77+
- contains:
78+
path: rules
79+
content:
80+
apiGroups: ["cert-manager.io"]
81+
resources: ["certificates"]
82+
verbs: ["get", "list", "delete"]
83+
- contains:
84+
path: rules
85+
content:
86+
apiGroups: ["acme.cert-manager.io"]
87+
resources: ["orders", "challenges"]
88+
verbs: ["get", "list", "delete"]
89+
- notContains:
90+
path: rules
91+
content:
92+
apiGroups: ["networking.k8s.io"]
93+
resources: ["ingresses"]
94+
verbs: ["get", "list", "delete"]
95+
- notContains:
96+
path: rules
97+
content:
98+
apiGroups: [""]
99+
resources: ["services", "pods"]
100+
verbs: ["get", "list", "delete"]
101+
- lengthEqual:
102+
path: rules
103+
count: 2
104+
105+
- it: binds the Role to the cleanup ServiceAccount
106+
documentIndex: 3
107+
asserts:
108+
- equal:
109+
path: kind
110+
value: RoleBinding
111+
- equal:
112+
path: roleRef.name
113+
value: my-bucket-cleanup-acme
114+
- equal:
115+
path: subjects[0].name
116+
value: my-bucket-cleanup-acme

0 commit comments

Comments
 (0)