Skip to content

Commit b223d36

Browse files
committed
fix(kubernetes): clear kamaji datastore-secret finalizer to unblock namespace deletion (#3062)
After deleting a Kubernetes app/tenant, the <release>-datastore-config Secret kept the finalizer finalizer.kamaji.clastix.io/datastore-secret, blocking deletion of the whole namespace. Extend the pre-delete hook to wait for the TenantControlPlane to be removed (so Kamaji can strip the finalizer itself), then, as a backstop, clear the finalizer on the datastore-config Secret if it still lingers. RBAC is scoped via resourceNames to that one Secret (get/patch). The Job runs non-root with a read-only root fs, dropped capabilities and a writable /tmp for the kubectl cache; failures are logged. Signed-off-by: Алексей Артамонов <[email protected]>
1 parent 374cac1 commit b223d36

2 files changed

Lines changed: 125 additions & 3 deletions

File tree

‎packages/apps/kubernetes/templates/delete.yaml‎

Lines changed: 67 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,23 @@ spec:
2121
- key: node-role.kubernetes.io/control-plane
2222
operator: Exists
2323
effect: "NoSchedule"
24+
securityContext:
25+
runAsNonRoot: true
26+
runAsUser: 65534
27+
runAsGroup: 65534
28+
seccompProfile:
29+
type: RuntimeDefault
2430
containers:
2531
- name: kubectl
2632
image: docker.io/clastix/kubectl:v1.32
33+
securityContext:
34+
allowPrivilegeEscalation: false
35+
readOnlyRootFilesystem: true
36+
capabilities:
37+
drop: ["ALL"]
38+
env:
39+
- name: HOME
40+
value: /tmp
2741
command:
2842
- /bin/sh
2943
- -c
@@ -48,9 +62,42 @@ spec:
4862
kubectl -n {{ .Release.Namespace }} delete kamajicontrolplanes.controlplane.cluster.x-k8s.io {{ .Release.Name }} \
4963
--ignore-not-found=true
5064
51-
echo "Step 4: Deleting TenantControlPlane {{ .Release.Name }}"
65+
echo "Step 4: Deleting TenantControlPlane {{ .Release.Name }} and waiting for it to be removed"
66+
# --wait=true blocks until the TenantControlPlane object is fully
67+
# gone. This is required so that Kamaji's TCP finalizer-cleanup
68+
# chain (which strips finalizer.kamaji.clastix.io/datastore-secret
69+
# off the <release>-datastore-config Secret) has a chance to run
70+
# before we fall through to Step 4b. A short --timeout keeps the
71+
# hook from hanging forever if Kamaji is unreachable; Step 4b is
72+
# the backstop for that case.
5273
kubectl -n {{ .Release.Namespace }} delete tenantcontrolplanes.kamaji.clastix.io {{ .Release.Name }} \
53-
--ignore-not-found=true
74+
--ignore-not-found=true --wait=true --timeout=120s \
75+
|| echo "WARNING: TenantControlPlane {{ .Release.Name }} did not delete within timeout; relying on the Step 4b finalizer backstop" >&2
76+
77+
echo "Step 4b: Clearing leftover Kamaji datastore-secret finalizer on {{ .Release.Name }}-datastore-config"
78+
# Kamaji creates a per-tenant datastore connection Secret named
79+
# <tcp-name>-datastore-config (tcp-name == .Release.Name) and adds
80+
# the finalizer "finalizer.kamaji.clastix.io/datastore-secret" to
81+
# it. That finalizer is normally removed by Kamaji as part of the
82+
# TenantControlPlane deletion above. If the TCP was force-deleted,
83+
# or Kamaji did not reconcile the removal in time, the finalizer
84+
# stays on the Secret and blocks deletion of the entire namespace
85+
# (the Secret cannot be garbage-collected, see issue #3062).
86+
#
87+
# By this point the TenantControlPlane is gone (Step 4 waited for
88+
# it), so Kamaji no longer has any legitimate use for this Secret:
89+
# the Secret has an ownerReference to the now-deleted TCP and is
90+
# only awaiting garbage collection. Stripping the finalizer here is
91+
# therefore safe and merely lets the GC reap it. We only patch the
92+
# finalizers field of this one named Secret.
93+
if kubectl -n {{ .Release.Namespace }} get secret {{ .Release.Name }}-datastore-config >/dev/null 2>&1; then
94+
echo " Removing finalizers from secret/{{ .Release.Name }}-datastore-config"
95+
kubectl -n {{ .Release.Namespace }} patch secret {{ .Release.Name }}-datastore-config \
96+
--type=merge -p '{"metadata":{"finalizers":null}}' \
97+
|| echo "WARNING: failed to clear datastore-secret finalizer for {{ .Release.Name }}; the namespace may stay in Terminating (issue #3062)" >&2
98+
else
99+
echo " Secret {{ .Release.Name }}-datastore-config not present; nothing to do"
100+
fi
54101
55102
echo "Step 5: Cleaning up DataVolumes"
56103
kubectl -n {{ .Release.Namespace }} delete datavolumes \
@@ -64,7 +111,12 @@ spec:
64111
--ignore-not-found=true
65112
66113
echo "Cleanup completed successfully"
67-
114+
volumeMounts:
115+
- name: tmp
116+
mountPath: /tmp
117+
volumes:
118+
- name: tmp
119+
emptyDir: {}
68120
---
69121
apiVersion: v1
70122
kind: ServiceAccount
@@ -130,6 +182,18 @@ rules:
130182
- list
131183
- watch
132184
- delete
185+
# Scoped to the single Kamaji datastore connection Secret so Step 4b can
186+
# strip the leftover finalizer.kamaji.clastix.io/datastore-secret finalizer
187+
# (issue #3062). resourceNames limits get/patch to exactly this Secret.
188+
- apiGroups:
189+
- ""
190+
resources:
191+
- secrets
192+
resourceNames:
193+
- {{ .Release.Name }}-datastore-config
194+
verbs:
195+
- get
196+
- patch
133197
---
134198
apiVersion: rbac.authorization.k8s.io/v1
135199
kind: RoleBinding
Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
suite: pre-delete cleanup hook (issue #3062)
2+
3+
# These tests pin the behaviour that prevents the Kamaji
4+
# <release>-datastore-config Secret from blocking namespace deletion:
5+
# the pre-delete Job must wait for the TenantControlPlane to be gone and
6+
# then strip the finalizer.kamaji.clastix.io/datastore-secret finalizer,
7+
# and the cleanup Role must be allowed to patch exactly that one Secret.
8+
9+
templates:
10+
- templates/delete.yaml
11+
12+
release:
13+
name: test-k8s
14+
namespace: tenant-test
15+
16+
values:
17+
- values-ci.yaml
18+
19+
tests:
20+
- it: cleanup Role can get/patch only the datastore-config Secret
21+
documentSelector:
22+
path: kind
23+
value: Role
24+
asserts:
25+
- contains:
26+
path: rules
27+
content:
28+
apiGroups:
29+
- ""
30+
resources:
31+
- secrets
32+
resourceNames:
33+
- test-k8s-datastore-config
34+
verbs:
35+
- get
36+
- patch
37+
38+
- it: pre-delete Job waits for the TenantControlPlane before clearing the Secret
39+
documentSelector:
40+
path: kind
41+
value: Job
42+
asserts:
43+
# TCP deletion must wait so Kamaji can clear the finalizer itself first.
44+
- matchRegex:
45+
path: spec.template.spec.containers[0].command[2]
46+
pattern: 'delete tenantcontrolplanes\.kamaji\.clastix\.io test-k8s[\s\S]*--wait=true'
47+
48+
- it: pre-delete Job strips the leftover datastore-secret finalizer as a backstop
49+
documentSelector:
50+
path: kind
51+
value: Job
52+
asserts:
53+
- matchRegex:
54+
path: spec.template.spec.containers[0].command[2]
55+
pattern: 'patch secret test-k8s-datastore-config'
56+
- matchRegex:
57+
path: spec.template.spec.containers[0].command[2]
58+
pattern: '"finalizers":null'

0 commit comments

Comments
 (0)