Skip to content

Commit 959a9b1

Browse files
authored
fix(kubernetes): clear kamaji datastore-secret finalizer to unblock namespace deletion (cozystack#3062) (cozystack#3078)
## What this PR does After deleting a `Kubernetes` app CR / tenant, the secret `<release>-datastore-config` kept the finalizer `finalizer.kamaji.clastix.io/datastore-secret`, which was never removed — **blocking deletion of the entire namespace** (observed stuck in `Terminating` for 115 minutes). Kamaji adds this finalizer to the datastore connection Secret and normally strips it as part of TenantControlPlane (TCP) deletion; if the TCP is force-deleted or Kamaji does not reconcile in time, the finalizer hangs and the Secret can never be garbage-collected, wedging the namespace. This extends the existing `pre-delete` cleanup hook (`packages/apps/kubernetes/templates/delete.yaml`): - **Step 4** now deletes the TenantControlPlane with `--wait=true --timeout=120s` so Kamaji's own finalizer-cleanup chain has a chance to run before we fall through. - **Step 4b (backstop)** patches `metadata.finalizers: null` on `<release>-datastore-config` only if it still exists. By this point the TCP is gone, the Secret is owner-ref'd to the deleted TCP and only awaiting GC, so clearing the finalizer merely lets GC reap it — it cannot strip a finalizer while a live TCP still needs the datastore. The Secret holds only connection config, not tenant data (which lives in the external etcd datastore), so no data is lost. - **RBAC** adds a `secrets` rule scoped via `resourceNames` to exactly `<release>-datastore-config`, verbs `get`/`patch` only. The Job is hardened: non-root (uid 65534), read-only root fs, all caps dropped, `seccompProfile: RuntimeDefault`, writable `/tmp` for kubectl cache. Failures at both steps are logged (the Step 4b warning explicitly mentions the namespace may stay Terminating) but the hook still completes so it cannot itself block teardown. This is the most impactful of the create→delete leak sweep findings: without it, tenant teardown hangs and the create→delete cycle requires manual intervention. Fixes cozystack#3062. ### Release note ```release-note fix(kubernetes): clear the leftover Kamaji `finalizer.kamaji.clastix.io/datastore-secret` finalizer on the datastore-config Secret during tenant teardown, so deleting a Kubernetes app / tenant no longer leaves the namespace stuck in Terminating. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Extended the pre-delete cleanup workflow to more reliably remove `TenantControlPlane` and handle slow deletions with a bounded wait. * **Security** * Hardened the cleanup job by running as non-root with restricted security settings, dropped capabilities, and a safe writable temp location. * **Bug Fixes** * Added a conditional backstop to remove leftover `finalizers` from the relevant datastore Secret so namespace termination isn’t blocked. * **Tests** * Added a YAML test suite to validate the wait-and-patch behavior and ensure the patch runs only after the `TenantControlPlane` is gone. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 080ceac + 4b9d67f commit 959a9b1

2 files changed

Lines changed: 151 additions & 3 deletions

File tree

‎packages/apps/kubernetes/templates/delete.yaml‎

Lines changed: 81 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,30 @@ spec:
2121
- key: node-role.kubernetes.io/control-plane
2222
operator: Exists
2323
effect: "NoSchedule"
24+
securityContext:
25+
runAsNonRoot: true
26+
runAsUser: 65534
27+
runAsGroup: 65534
28+
seccompProfile:
29+
type: RuntimeDefault
2430
containers:
2531
- name: kubectl
2632
image: docker.io/clastix/kubectl:v1.32
33+
securityContext:
34+
allowPrivilegeEscalation: false
35+
readOnlyRootFilesystem: true
36+
capabilities:
37+
drop: ["ALL"]
38+
env:
39+
- name: HOME
40+
value: /tmp
41+
resources:
42+
requests:
43+
cpu: 10m
44+
memory: 64Mi
45+
limits:
46+
cpu: 200m
47+
memory: 128Mi
2748
command:
2849
- /bin/sh
2950
- -c
@@ -48,9 +69,49 @@ spec:
4869
kubectl -n {{ .Release.Namespace }} delete kamajicontrolplanes.controlplane.cluster.x-k8s.io {{ .Release.Name }} \
4970
--ignore-not-found=true
5071
51-
echo "Step 4: Deleting TenantControlPlane {{ .Release.Name }}"
72+
echo "Step 4: Deleting TenantControlPlane {{ .Release.Name }} and waiting for it to be removed"
73+
# --wait=true blocks until the TenantControlPlane object is fully
74+
# gone. This is required so that Kamaji's TCP finalizer-cleanup
75+
# chain (which strips finalizer.kamaji.clastix.io/datastore-secret
76+
# off the <release>-datastore-config Secret) has a chance to run
77+
# before we fall through to Step 4b. A short --timeout keeps the
78+
# hook from hanging forever if Kamaji is unreachable; Step 4b is
79+
# the backstop for that case.
5280
kubectl -n {{ .Release.Namespace }} delete tenantcontrolplanes.kamaji.clastix.io {{ .Release.Name }} \
53-
--ignore-not-found=true
81+
--ignore-not-found=true --wait=true --timeout=120s \
82+
|| echo "WARNING: TenantControlPlane {{ .Release.Name }} did not delete within timeout; relying on the Step 4b finalizer backstop" >&2
83+
84+
echo "Step 4b: Clearing leftover Kamaji datastore-secret finalizer on {{ .Release.Name }}-datastore-config"
85+
# Kamaji creates a per-tenant datastore connection Secret named
86+
# <tcp-name>-datastore-config (tcp-name == .Release.Name) and adds
87+
# the finalizer "finalizer.kamaji.clastix.io/datastore-secret" to
88+
# it. That finalizer is normally removed by Kamaji as part of the
89+
# TenantControlPlane deletion above. If the TCP was force-deleted,
90+
# or Kamaji did not reconcile the removal in time, the finalizer
91+
# stays on the Secret and blocks deletion of the entire namespace
92+
# (the Secret cannot be garbage-collected, see issue #3062).
93+
#
94+
# By this point the TenantControlPlane is gone (Step 4 waited for
95+
# it), so Kamaji no longer has any legitimate use for this Secret:
96+
# the Secret has an ownerReference to the now-deleted TCP and is
97+
# only awaiting garbage collection. Stripping the finalizer here is
98+
# therefore safe and merely lets the GC reap it. We only patch the
99+
# finalizers field of this one named Secret.
100+
# Only clear the finalizer if the TenantControlPlane is actually gone.
101+
# Step 4 may have timed out with the TCP still present; in that case
102+
# Kamaji still legitimately owns the datastore Secret, and stripping
103+
# the finalizer would interfere with its lifecycle management. Gate
104+
# the patch on the TCP being confirmed absent (CodeRabbit #3078).
105+
if kubectl -n {{ .Release.Namespace }} get tenantcontrolplanes.kamaji.clastix.io {{ .Release.Name }} >/dev/null 2>&1; then
106+
echo " TenantControlPlane {{ .Release.Name }} still present (deletion not complete); leaving the datastore-secret finalizer untouched"
107+
elif kubectl -n {{ .Release.Namespace }} get secret {{ .Release.Name }}-datastore-config >/dev/null 2>&1; then
108+
echo " Removing finalizers from secret/{{ .Release.Name }}-datastore-config"
109+
kubectl -n {{ .Release.Namespace }} patch secret {{ .Release.Name }}-datastore-config \
110+
--type=merge -p '{"metadata":{"finalizers":null}}' \
111+
|| echo "WARNING: failed to clear datastore-secret finalizer for {{ .Release.Name }}; the namespace may stay in Terminating (issue #3062)" >&2
112+
else
113+
echo " Secret {{ .Release.Name }}-datastore-config not present; nothing to do"
114+
fi
54115
55116
echo "Step 5: Cleaning up DataVolumes"
56117
kubectl -n {{ .Release.Namespace }} delete datavolumes \
@@ -64,7 +125,12 @@ spec:
64125
--ignore-not-found=true
65126
66127
echo "Cleanup completed successfully"
67-
128+
volumeMounts:
129+
- name: tmp
130+
mountPath: /tmp
131+
volumes:
132+
- name: tmp
133+
emptyDir: {}
68134
---
69135
apiVersion: v1
70136
kind: ServiceAccount
@@ -130,6 +196,18 @@ rules:
130196
- list
131197
- watch
132198
- delete
199+
# Scoped to the single Kamaji datastore connection Secret so Step 4b can
200+
# strip the leftover finalizer.kamaji.clastix.io/datastore-secret finalizer
201+
# (issue #3062). resourceNames limits get/patch to exactly this Secret.
202+
- apiGroups:
203+
- ""
204+
resources:
205+
- secrets
206+
resourceNames:
207+
- {{ .Release.Name }}-datastore-config
208+
verbs:
209+
- get
210+
- patch
133211
---
134212
apiVersion: rbac.authorization.k8s.io/v1
135213
kind: RoleBinding
Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
suite: pre-delete cleanup hook (issue #3062)
2+
3+
# These tests pin the behaviour that prevents the Kamaji
4+
# <release>-datastore-config Secret from blocking namespace deletion:
5+
# the pre-delete Job must wait for the TenantControlPlane to be gone and
6+
# then strip the finalizer.kamaji.clastix.io/datastore-secret finalizer,
7+
# and the cleanup Role must be allowed to patch exactly that one Secret.
8+
9+
templates:
10+
- templates/delete.yaml
11+
12+
release:
13+
name: test-k8s
14+
namespace: tenant-test
15+
16+
values:
17+
- values-ci.yaml
18+
19+
tests:
20+
- it: cleanup Role can get/patch only the datastore-config Secret
21+
documentSelector:
22+
path: kind
23+
value: Role
24+
asserts:
25+
- contains:
26+
path: rules
27+
content:
28+
apiGroups:
29+
- ""
30+
resources:
31+
- secrets
32+
resourceNames:
33+
- test-k8s-datastore-config
34+
verbs:
35+
- get
36+
- patch
37+
38+
- it: pre-delete Job waits for the TenantControlPlane before clearing the Secret
39+
documentSelector:
40+
path: kind
41+
value: Job
42+
asserts:
43+
# TCP deletion must wait so Kamaji can clear the finalizer itself first.
44+
- matchRegex:
45+
path: spec.template.spec.containers[0].command[2]
46+
pattern: 'delete tenantcontrolplanes\.kamaji\.clastix\.io test-k8s[\s\S]*--wait=true'
47+
48+
- it: pre-delete Job strips the leftover datastore-secret finalizer as a backstop
49+
documentSelector:
50+
path: kind
51+
value: Job
52+
asserts:
53+
- matchRegex:
54+
path: spec.template.spec.containers[0].command[2]
55+
pattern: 'patch secret test-k8s-datastore-config'
56+
- matchRegex:
57+
path: spec.template.spec.containers[0].command[2]
58+
pattern: '"finalizers":null'
59+
60+
- it: pre-delete Job clears the finalizer only after confirming the TenantControlPlane is gone
61+
documentSelector:
62+
path: kind
63+
value: Job
64+
asserts:
65+
# The finalizer patch must be gated on the TCP being absent (#3078): if the
66+
# Step 4 delete timed out with the TCP still alive, we must not strip the
67+
# finalizer Kamaji still legitimately owns.
68+
- matchRegex:
69+
path: spec.template.spec.containers[0].command[2]
70+
pattern: 'get tenantcontrolplanes\.kamaji\.clastix\.io test-k8s[\s\S]*still present[\s\S]*patch secret test-k8s-datastore-config'

0 commit comments

Comments
 (0)