Skip to content

Commit 4b9d67f

Browse files
scooby87scooby87
committed
fix(kubernetes): clear kamaji datastore-secret finalizer to unblock namespace deletion (#3062)
After deleting a Kubernetes app/tenant, the <release>-datastore-config Secret kept the finalizer finalizer.kamaji.clastix.io/datastore-secret, blocking deletion of the whole namespace. Extend the pre-delete hook to wait for the TenantControlPlane to be removed (so Kamaji can strip the finalizer itself), then, as a backstop, clear the finalizer on the datastore-config Secret if it still lingers. RBAC is scoped via resourceNames to that one Secret (get/patch). The Job runs non-root with a read-only root fs, dropped capabilities and a writable /tmp for the kubectl cache; failures are logged. Signed-off-by: Алексей Артамонов <[email protected]>
1 parent 374cac1 commit 4b9d67f

2 files changed

Lines changed: 151 additions & 3 deletions

File tree

‎packages/apps/kubernetes/templates/delete.yaml‎

Lines changed: 81 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,30 @@ spec:
2121
- key: node-role.kubernetes.io/control-plane
2222
operator: Exists
2323
effect: "NoSchedule"
24+
securityContext:
25+
runAsNonRoot: true
26+
runAsUser: 65534
27+
runAsGroup: 65534
28+
seccompProfile:
29+
type: RuntimeDefault
2430
containers:
2531
- name: kubectl
2632
image: docker.io/clastix/kubectl:v1.32
33+
securityContext:
34+
allowPrivilegeEscalation: false
35+
readOnlyRootFilesystem: true
36+
capabilities:
37+
drop: ["ALL"]
38+
env:
39+
- name: HOME
40+
value: /tmp
41+
resources:
42+
requests:
43+
cpu: 10m
44+
memory: 64Mi
45+
limits:
46+
cpu: 200m
47+
memory: 128Mi
2748
command:
2849
- /bin/sh
2950
- -c
@@ -48,9 +69,49 @@ spec:
4869
kubectl -n {{ .Release.Namespace }} delete kamajicontrolplanes.controlplane.cluster.x-k8s.io {{ .Release.Name }} \
4970
--ignore-not-found=true
5071
51-
echo "Step 4: Deleting TenantControlPlane {{ .Release.Name }}"
72+
echo "Step 4: Deleting TenantControlPlane {{ .Release.Name }} and waiting for it to be removed"
73+
# --wait=true blocks until the TenantControlPlane object is fully
74+
# gone. This is required so that Kamaji's TCP finalizer-cleanup
75+
# chain (which strips finalizer.kamaji.clastix.io/datastore-secret
76+
# off the <release>-datastore-config Secret) has a chance to run
77+
# before we fall through to Step 4b. A short --timeout keeps the
78+
# hook from hanging forever if Kamaji is unreachable; Step 4b is
79+
# the backstop for that case.
5280
kubectl -n {{ .Release.Namespace }} delete tenantcontrolplanes.kamaji.clastix.io {{ .Release.Name }} \
53-
--ignore-not-found=true
81+
--ignore-not-found=true --wait=true --timeout=120s \
82+
|| echo "WARNING: TenantControlPlane {{ .Release.Name }} did not delete within timeout; relying on the Step 4b finalizer backstop" >&2
83+
84+
echo "Step 4b: Clearing leftover Kamaji datastore-secret finalizer on {{ .Release.Name }}-datastore-config"
85+
# Kamaji creates a per-tenant datastore connection Secret named
86+
# <tcp-name>-datastore-config (tcp-name == .Release.Name) and adds
87+
# the finalizer "finalizer.kamaji.clastix.io/datastore-secret" to
88+
# it. That finalizer is normally removed by Kamaji as part of the
89+
# TenantControlPlane deletion above. If the TCP was force-deleted,
90+
# or Kamaji did not reconcile the removal in time, the finalizer
91+
# stays on the Secret and blocks deletion of the entire namespace
92+
# (the Secret cannot be garbage-collected, see issue #3062).
93+
#
94+
# By this point the TenantControlPlane is gone (Step 4 waited for
95+
# it), so Kamaji no longer has any legitimate use for this Secret:
96+
# the Secret has an ownerReference to the now-deleted TCP and is
97+
# only awaiting garbage collection. Stripping the finalizer here is
98+
# therefore safe and merely lets the GC reap it. We only patch the
99+
# finalizers field of this one named Secret.
100+
# Only clear the finalizer if the TenantControlPlane is actually gone.
101+
# Step 4 may have timed out with the TCP still present; in that case
102+
# Kamaji still legitimately owns the datastore Secret, and stripping
103+
# the finalizer would interfere with its lifecycle management. Gate
104+
# the patch on the TCP being confirmed absent (CodeRabbit #3078).
105+
if kubectl -n {{ .Release.Namespace }} get tenantcontrolplanes.kamaji.clastix.io {{ .Release.Name }} >/dev/null 2>&1; then
106+
echo " TenantControlPlane {{ .Release.Name }} still present (deletion not complete); leaving the datastore-secret finalizer untouched"
107+
elif kubectl -n {{ .Release.Namespace }} get secret {{ .Release.Name }}-datastore-config >/dev/null 2>&1; then
108+
echo " Removing finalizers from secret/{{ .Release.Name }}-datastore-config"
109+
kubectl -n {{ .Release.Namespace }} patch secret {{ .Release.Name }}-datastore-config \
110+
--type=merge -p '{"metadata":{"finalizers":null}}' \
111+
|| echo "WARNING: failed to clear datastore-secret finalizer for {{ .Release.Name }}; the namespace may stay in Terminating (issue #3062)" >&2
112+
else
113+
echo " Secret {{ .Release.Name }}-datastore-config not present; nothing to do"
114+
fi
54115
55116
echo "Step 5: Cleaning up DataVolumes"
56117
kubectl -n {{ .Release.Namespace }} delete datavolumes \
@@ -64,7 +125,12 @@ spec:
64125
--ignore-not-found=true
65126
66127
echo "Cleanup completed successfully"
67-
128+
volumeMounts:
129+
- name: tmp
130+
mountPath: /tmp
131+
volumes:
132+
- name: tmp
133+
emptyDir: {}
68134
---
69135
apiVersion: v1
70136
kind: ServiceAccount
@@ -130,6 +196,18 @@ rules:
130196
- list
131197
- watch
132198
- delete
199+
# Scoped to the single Kamaji datastore connection Secret so Step 4b can
200+
# strip the leftover finalizer.kamaji.clastix.io/datastore-secret finalizer
201+
# (issue #3062). resourceNames limits get/patch to exactly this Secret.
202+
- apiGroups:
203+
- ""
204+
resources:
205+
- secrets
206+
resourceNames:
207+
- {{ .Release.Name }}-datastore-config
208+
verbs:
209+
- get
210+
- patch
133211
---
134212
apiVersion: rbac.authorization.k8s.io/v1
135213
kind: RoleBinding

‎packages/apps/kubernetes/tests/delete_hook_test.yaml‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
suite: pre-delete cleanup hook (issue #3062)
2+
3+
# These tests pin the behaviour that prevents the Kamaji
4+
# <release>-datastore-config Secret from blocking namespace deletion:
5+
# the pre-delete Job must wait for the TenantControlPlane to be gone and
6+
# then strip the finalizer.kamaji.clastix.io/datastore-secret finalizer,
7+
# and the cleanup Role must be allowed to patch exactly that one Secret.
8+
9+
templates:
10+
- templates/delete.yaml
11+
12+
release:
13+
name: test-k8s
14+
namespace: tenant-test
15+
16+
values:
17+
- values-ci.yaml
18+
19+
tests:
20+
- it: cleanup Role can get/patch only the datastore-config Secret
21+
documentSelector:
22+
path: kind
23+
value: Role
24+
asserts:
25+
- contains:
26+
path: rules
27+
content:
28+
apiGroups:
29+
- ""
30+
resources:
31+
- secrets
32+
resourceNames:
33+
- test-k8s-datastore-config
34+
verbs:
35+
- get
36+
- patch
37+
38+
- it: pre-delete Job waits for the TenantControlPlane before clearing the Secret
39+
documentSelector:
40+
path: kind
41+
value: Job
42+
asserts:
43+
# TCP deletion must wait so Kamaji can clear the finalizer itself first.
44+
- matchRegex:
45+
path: spec.template.spec.containers[0].command[2]
46+
pattern: 'delete tenantcontrolplanes\.kamaji\.clastix\.io test-k8s[\s\S]*--wait=true'
47+
48+
- it: pre-delete Job strips the leftover datastore-secret finalizer as a backstop
49+
documentSelector:
50+
path: kind
51+
value: Job
52+
asserts:
53+
- matchRegex:
54+
path: spec.template.spec.containers[0].command[2]
55+
pattern: 'patch secret test-k8s-datastore-config'
56+
- matchRegex:
57+
path: spec.template.spec.containers[0].command[2]
58+
pattern: '"finalizers":null'
59+
60+
- it: pre-delete Job clears the finalizer only after confirming the TenantControlPlane is gone
61+
documentSelector:
62+
path: kind
63+
value: Job
64+
asserts:
65+
# The finalizer patch must be gated on the TCP being absent (#3078): if the
66+
# Step 4 delete timed out with the TCP still alive, we must not strip the
67+
# finalizer Kamaji still legitimately owns.
68+
- matchRegex:
69+
path: spec.template.spec.containers[0].command[2]
70+
pattern: 'get tenantcontrolplanes\.kamaji\.clastix\.io test-k8s[\s\S]*still present[\s\S]*patch secret test-k8s-datastore-config'

0 commit comments

Comments
 (0)