Skip to content

Commit 708b3bb

Browse files
committed
fix(kubernetes): clear kamaji datastore-secret finalizer to unblock namespace deletion (#3062)
After deleting a Kubernetes app/tenant, the <release>-datastore-config Secret kept the finalizer finalizer.kamaji.clastix.io/datastore-secret, blocking deletion of the whole namespace. Extend the pre-delete hook to wait for the TenantControlPlane to be removed (so Kamaji can strip the finalizer itself), then, as a backstop, clear the finalizer on the datastore-config Secret if it still lingers. RBAC is scoped via resourceNames to that one Secret (get/patch). The Job runs non-root with a read-only root fs, dropped capabilities and a writable /tmp for the kubectl cache; failures are logged. Signed-off-by: Алексей Артамонов <[email protected]>
1 parent 374cac1 commit 708b3bb

2 files changed

Lines changed: 132 additions & 3 deletions

File tree

‎packages/apps/kubernetes/templates/delete.yaml‎

Lines changed: 74 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,30 @@ spec:
2121
- key: node-role.kubernetes.io/control-plane
2222
operator: Exists
2323
effect: "NoSchedule"
24+
securityContext:
25+
runAsNonRoot: true
26+
runAsUser: 65534
27+
runAsGroup: 65534
28+
seccompProfile:
29+
type: RuntimeDefault
2430
containers:
2531
- name: kubectl
2632
image: docker.io/clastix/kubectl:v1.32
33+
securityContext:
34+
allowPrivilegeEscalation: false
35+
readOnlyRootFilesystem: true
36+
capabilities:
37+
drop: ["ALL"]
38+
env:
39+
- name: HOME
40+
value: /tmp
41+
resources:
42+
requests:
43+
cpu: 10m
44+
memory: 64Mi
45+
limits:
46+
cpu: 200m
47+
memory: 128Mi
2748
command:
2849
- /bin/sh
2950
- -c
@@ -48,9 +69,42 @@ spec:
4869
kubectl -n {{ .Release.Namespace }} delete kamajicontrolplanes.controlplane.cluster.x-k8s.io {{ .Release.Name }} \
4970
--ignore-not-found=true
5071
51-
echo "Step 4: Deleting TenantControlPlane {{ .Release.Name }}"
72+
echo "Step 4: Deleting TenantControlPlane {{ .Release.Name }} and waiting for it to be removed"
73+
# --wait=true blocks until the TenantControlPlane object is fully
74+
# gone. This is required so that Kamaji's TCP finalizer-cleanup
75+
# chain (which strips finalizer.kamaji.clastix.io/datastore-secret
76+
# off the <release>-datastore-config Secret) has a chance to run
77+
# before we fall through to Step 4b. A short --timeout keeps the
78+
# hook from hanging forever if Kamaji is unreachable; Step 4b is
79+
# the backstop for that case.
5280
kubectl -n {{ .Release.Namespace }} delete tenantcontrolplanes.kamaji.clastix.io {{ .Release.Name }} \
53-
--ignore-not-found=true
81+
--ignore-not-found=true --wait=true --timeout=120s \
82+
|| echo "WARNING: TenantControlPlane {{ .Release.Name }} did not delete within timeout; relying on the Step 4b finalizer backstop" >&2
83+
84+
echo "Step 4b: Clearing leftover Kamaji datastore-secret finalizer on {{ .Release.Name }}-datastore-config"
85+
# Kamaji creates a per-tenant datastore connection Secret named
86+
# <tcp-name>-datastore-config (tcp-name == .Release.Name) and adds
87+
# the finalizer "finalizer.kamaji.clastix.io/datastore-secret" to
88+
# it. That finalizer is normally removed by Kamaji as part of the
89+
# TenantControlPlane deletion above. If the TCP was force-deleted,
90+
# or Kamaji did not reconcile the removal in time, the finalizer
91+
# stays on the Secret and blocks deletion of the entire namespace
92+
# (the Secret cannot be garbage-collected, see issue #3062).
93+
#
94+
# By this point the TenantControlPlane is gone (Step 4 waited for
95+
# it), so Kamaji no longer has any legitimate use for this Secret:
96+
# the Secret has an ownerReference to the now-deleted TCP and is
97+
# only awaiting garbage collection. Stripping the finalizer here is
98+
# therefore safe and merely lets the GC reap it. We only patch the
99+
# finalizers field of this one named Secret.
100+
if kubectl -n {{ .Release.Namespace }} get secret {{ .Release.Name }}-datastore-config >/dev/null 2>&1; then
101+
echo " Removing finalizers from secret/{{ .Release.Name }}-datastore-config"
102+
kubectl -n {{ .Release.Namespace }} patch secret {{ .Release.Name }}-datastore-config \
103+
--type=merge -p '{"metadata":{"finalizers":null}}' \
104+
|| echo "WARNING: failed to clear datastore-secret finalizer for {{ .Release.Name }}; the namespace may stay in Terminating (issue #3062)" >&2
105+
else
106+
echo " Secret {{ .Release.Name }}-datastore-config not present; nothing to do"
107+
fi
54108
55109
echo "Step 5: Cleaning up DataVolumes"
56110
kubectl -n {{ .Release.Namespace }} delete datavolumes \
@@ -64,7 +118,12 @@ spec:
64118
--ignore-not-found=true
65119
66120
echo "Cleanup completed successfully"
67-
121+
volumeMounts:
122+
- name: tmp
123+
mountPath: /tmp
124+
volumes:
125+
- name: tmp
126+
emptyDir: {}
68127
---
69128
apiVersion: v1
70129
kind: ServiceAccount
@@ -130,6 +189,18 @@ rules:
130189
- list
131190
- watch
132191
- delete
192+
# Scoped to the single Kamaji datastore connection Secret so Step 4b can
193+
# strip the leftover finalizer.kamaji.clastix.io/datastore-secret finalizer
194+
# (issue #3062). resourceNames limits get/patch to exactly this Secret.
195+
- apiGroups:
196+
- ""
197+
resources:
198+
- secrets
199+
resourceNames:
200+
- {{ .Release.Name }}-datastore-config
201+
verbs:
202+
- get
203+
- patch
133204
---
134205
apiVersion: rbac.authorization.k8s.io/v1
135206
kind: RoleBinding
Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
suite: pre-delete cleanup hook (issue #3062)
2+
3+
# These tests pin the behaviour that prevents the Kamaji
4+
# <release>-datastore-config Secret from blocking namespace deletion:
5+
# the pre-delete Job must wait for the TenantControlPlane to be gone and
6+
# then strip the finalizer.kamaji.clastix.io/datastore-secret finalizer,
7+
# and the cleanup Role must be allowed to patch exactly that one Secret.
8+
9+
templates:
10+
- templates/delete.yaml
11+
12+
release:
13+
name: test-k8s
14+
namespace: tenant-test
15+
16+
values:
17+
- values-ci.yaml
18+
19+
tests:
20+
- it: cleanup Role can get/patch only the datastore-config Secret
21+
documentSelector:
22+
path: kind
23+
value: Role
24+
asserts:
25+
- contains:
26+
path: rules
27+
content:
28+
apiGroups:
29+
- ""
30+
resources:
31+
- secrets
32+
resourceNames:
33+
- test-k8s-datastore-config
34+
verbs:
35+
- get
36+
- patch
37+
38+
- it: pre-delete Job waits for the TenantControlPlane before clearing the Secret
39+
documentSelector:
40+
path: kind
41+
value: Job
42+
asserts:
43+
# TCP deletion must wait so Kamaji can clear the finalizer itself first.
44+
- matchRegex:
45+
path: spec.template.spec.containers[0].command[2]
46+
pattern: 'delete tenantcontrolplanes\.kamaji\.clastix\.io test-k8s[\s\S]*--wait=true'
47+
48+
- it: pre-delete Job strips the leftover datastore-secret finalizer as a backstop
49+
documentSelector:
50+
path: kind
51+
value: Job
52+
asserts:
53+
- matchRegex:
54+
path: spec.template.spec.containers[0].command[2]
55+
pattern: 'patch secret test-k8s-datastore-config'
56+
- matchRegex:
57+
path: spec.template.spec.containers[0].command[2]
58+
pattern: '"finalizers":null'

0 commit comments

Comments
 (0)