kubernetes: datastore-config secret with kamaji finalizer blocks namespace deletion #3062

Describe the bug

After deleting a Kubernetes app CR / tenant, the secret <release>-datastore-config retains the finalizer finalizer.kamaji.clastix.io/datastore-secret, which is never removed — blocking deletion of the entire namespace (observed stuck in Terminating for 115 minutes).

Environment

• Cozystack version: v1.4.0-rc.2
• Provider: on-prem (bare-metal)

To Reproduce

1. Create a tenant, deploy Kubernetes/test.
2. Delete the Kubernetes CR / tenant.
3. The namespace tenant-<x> stays in Terminating indefinitely.
4. kubectl -n <ns> get secret <release>-datastore-config -o jsonpath='{.metadata.finalizers}' → ["finalizer.kamaji.clastix.io/datastore-secret"].

Expected behaviour
On app/tenant deletion, Kamaji removes its finalizer from the datastore secret, and the namespace deletes cleanly.

Actual behaviour
The TenantControlPlane is deleted before Kamaji clears the finalizer, so the secret hangs forever and the namespace cannot be deleted.

Additional context

• Root cause: Kamaji sets finalizer.kamaji.clastix.io/datastore-secret on the <release>-datastore-config secret. On deletion the TenantControlPlane is removed before Kamaji reconciles the finalizer off the secret, leaving it stuck.
• Severity note: This is the most impactful of the leaks — it blocks namespace teardown entirely and breaks the create→delete cycle (re-runs require manual intervention).
• Proposed fix (Kamaji/cozystack integration):
  • Ensure the finalizer is removed in a pre-delete step before the TenantControlPlane is deleted, or
  • Enforce deletion order: TenantControlPlane first (Kamaji clears finalizer), then namespace.
• Workaround:

  kubectl -n <ns> patch secret <release>-datastore-config \
    --type=json -p='[{"op":"remove","path":"/metadata/finalizers"}]'

Found during a create→delete leak sweep of all 22 apps on Cozystack v1.4.0-rc.2.