Tenant deletion wedges namespace in Terminating: pinned kamaji 26.3.5-edge predates upstream DataStore-finalizer fix (clastix/kamaji#1115/#1122) #3018

Summary

Deleting a Tenant whose etcd DataStore has no referencing TenantControlPlane can leave the tenant namespace stuck in Terminating indefinitely. This is a known, already-fixed upstream Kamaji bug (clastix/kamaji#1115, fixed by clastix/kamaji#1122) that is not present in the kamaji version Cozystack currently pins (26.3.5-edge). Filing as a tracking/acknowledgment issue — the remedy is a kamaji bump (relates to #2885).

Version evidence

• Reproduced deterministically on Cozystack v1.4.3 and v1.4.4.
• Pinned kamaji: packages/system/kamaji/images/kamaji/Dockerfile → ARG VERSION=26.3.5-edge (upstream commit dated 2026-03-22).
• Upstream fix fix(ds): allow deletion for unused datastores clastix/kamaji#1122 merged 2026-04-14, so it is absent in 26.3.5-edge (and 26.3.6-edge) and present from 26.4.4-edge onward (verified in 26.4.5-edge, 26.5.2-edge, 26.6.4-edge).

Symptoms

• kubectl get ns tenant-<name> stuck Terminating.
• The child etcd Flux HelmRelease frozen at Helm status=uninstalling (release secret sh.helm.release.v1.etcd.v1 never removed).
• Blocker: cluster-scoped Kamaji DataStore tenant-<name> stuck with finalizer kamaji.clastix.io/TenantControlPlane; kubectl get tenantcontrolplane -A shows 0 referencing it.
• kamaji logs DataStore can be safely deleted and then goes silent — the finalizer is never removed.

Root cause (upstream, confirmed in code)

In kamaji controllers/datastore_controller.go at 26.3.x-edge, DataStore deletion drives an AllowedDeletion status condition that only transitions False → True. When a DataStore has 0 referencing TenantControlPlanes at deletion time, that condition is never initialized (IsStatusConditionFalse returns false for an absent condition), so the reconcile falls through to logging DataStore can be safely deleted, and the deferred finalizer-removal — gated on IsStatusConditionTrue — never runs. No requeue follows, so it deadlocks (clastix/kamaji#1115).

Cascade: stuck DataStore → helm uninstall <etcd> --wait hangs → the etcd release freezes at uninstalling → the parent tenant uninstall times out (UninstallFailed) → the namespace is wedged in Terminating.

The 0-TCP condition is reached whenever a tenant is deleted before any Kubernetes / TenantControlPlane is created in it (quick create→delete), or when the TCP is removed before DataStore deletion is triggered.

Upstream fix

clastix/kamaji#1122 adds the missing 0-TCP path:

- if meta.IsStatusConditionFalse(ds.Status.Conditions, ...AllowedDeletionType) {
+ if meta.IsStatusConditionFalse(ds.Status.Conditions, ...AllowedDeletionType) || len(tcpList.Items) == 0 {

Reproduction

1. Create a Tenant with etcd: true.
2. Delete it before any TenantControlPlane (Kubernetes app) exists in it (e.g. within ~30s of creation).
3. The tenant namespace hangs in Terminating; the DataStore keeps kamaji.clastix.io/TenantControlPlane with 0 referencing TCPs.

Suggested fix

Bump pinned kamaji to a tag that includes clastix/kamaji#1122 — 26.4.4-edge or later (latest is 26.6.4-edge) — in packages/system/kamaji/images/kamaji/Dockerfile. This dovetails with #2885 (kamaji bump for CVEs); this issue adds a functional/availability reason and a hard minimum version.

Related but distinct: #2473 (tenant-controlplane HelmRelease deletion hang via a pre-delete hook on child HR finalizers) — different mechanism, same "tenant won't delete" family.

Workaround

With 0 referencing TenantControlPlanes:

kubectl patch datastore tenant-<name> --type=merge -p '{"metadata":{"finalizers":null}}'

The DataStore then deletes and the blocked etcd uninstall + namespace finalization complete.