Tenant namespace deleted directly is stuck Terminating on Kamaji finalizers #5002

What we see

On a v1.6.5 cluster the namespace of a tenant with a Kubernetes app, the Etcd module and a Postgres app was deleted directly through the API instead of through the Tenant app. The Tenant app and its HelmRelease in the parent namespace are still there. The namespace has been Terminating for more than 7 hours.

The namespace controller removed things in this order.

1. Pods first. The tenant etcd pod was gone 14 seconds after the namespace got its deletionTimestamp, and pods cannot start again in a terminating namespace. This matches OrderedNamespaceDeletion, on by default since Kubernetes 1.33, so the etcd pods are gone before the TenantControlPlane is even marked for deletion.
2. About 3 minutes later it removed the rest within 7 seconds, tenantcontrolplanes (deletecollection), etcdclusters, the etcd HelmRelease (its DataStore got a deletionTimestamp in the same second), secrets (deletecollection, this removed etcd-ca-tls, which has no finalizer), then kamajicontrolplanes.

What is left now.

TenantControlPlane <cluster>             finalizer.kamaji.clastix.io
Secret <cluster>-datastore-config        finalizer.kamaji.clastix.io/datastore-secret
DataStore <tenant-namespace> (cluster)   kamaji.clastix.io/TenantControlPlane
  kamaji.clastix.io/DataStoreAllowedDeletion=False  reason=DataStoreStillUsed  usedBy=<tenant-namespace>/<cluster>

Kamaji logs this on every retry.

unable to create connection config object: Secret "etcd-ca-tls" not found

Why it cannot finish

Kamaji in v1.6.5 is upstream 26.3.6-edge plus the two patches in packages/system/kamaji/images/kamaji/patches. The TenantControlPlane reconcile builds the DataStore connection before it reaches the deletion branch (tenantcontrolplane_controller.go L157-L191). The DataStore rendered by the etcd chart takes its CA and client certificate from etcd-ca-tls and etcd-client-tls in the tenant namespace (datastore.yaml L10-L32), so with etcd-ca-tls gone every reconcile stops there. Even with the Secret back, Setup.Delete drops the tenant's role, prefix and user in etcd before it removes the finalizer (datastore_setup.go L145-L181), and the etcd pods are already gone. The datastore-config finalizer is cleared in the same deletion pass and the DataStore waits for the TenantControlPlane, so nothing moves.

Kamaji guards <cluster>-datastore-config with a finalizer for the namespace deletion case (clastix/kamaji#376). Here the DataStore's own TLS Secrets and the etcd pods sit in the same namespace as the TenantControlPlane and nothing guards them. The deletion path is the same in 26.9.4-edge, which main pins, and on Kamaji master (L167-L201).

Why the Tenant app path does not hit it

The tenant pre-delete hook deletes the application HelmReleases first and the tenant modules such as etcd after them (cleanup-job.yaml L84-L92, same order on main). The Kubernetes app's pre-delete hook deletes the TenantControlPlane and waits for it while etcd and its Secrets are still up, then clears a leftover datastore-config finalizer once the TenantControlPlane is gone (delete.yaml L92-L138, #3078). That is the order suggested in clastix/kamaji#491. We have not tried this path on v1.6.5. On another cluster still on v1.6.4, where these hooks are the same, a tenant with Kubernetes, a VM and Postgres deleted through the Tenant app was gone from the dashboard after about 15 minutes and fully removed later.

Deleting the Tenant app now does not look like a way out either. Its pre-delete hook creates a Role and RoleBinding inside the tenant namespace (cleanup-job.yaml L12-L42), and a terminating namespace does not accept new objects. We read this from the chart and have not tried it.

Recovery we plan

Not run yet.

1. Remove the finalizer from the TenantControlPlane.
2. Remove the finalizer from <cluster>-datastore-config.
3. Wait for the namespace to go. The DataStore should then release itself. Kamaji reconciles it on the TenantControlPlane delete event and usedBy becomes empty.
4. Delete the Tenant app with spec.uninstall.disableHooks: true on its HelmRelease.

The keys Kamaji would clean up live in an etcd that is being deleted with the namespace, so skipping that cleanup loses nothing here.

Ideas

Would either of these make sense?

1. A guard for child tenant namespaces. cozystack-no-delete-guardrail from feat(platform): add deletion-protection guardrail via ValidatingAdmissionPolicy #2650 covers tenant-root only. A ValidatingAdmissionPolicy or a webhook that denies DELETE of a namespace labeled tenant.cozystack.io/tenant-root unless the request comes from the helm-controller running the Tenant uninstall would send people to the Tenant app instead of leaving a namespace that cannot finish.
2. On the Kamaji side, when the DataStore itself has a deletionTimestamp or its connection Secret is gone, the TenantControlPlane deletion could stop retrying the datastore cleanup after a bounded number of attempts and drop its finalizers, with an event or a condition saying so. In Kamaji broken after namespace removal clastix/kamaji#491 the answer was to order deletions on the platform side, and the Tenant app does that, so this would only cover the case when the order is broken. We are happy to raise it in clastix/kamaji if it sounds reasonable.

Related issues with a different cause are #3018 (DataStore with no control planes), #3062 with #3078 (datastore-config finalizer left after the TenantControlPlane is gone) and clastix/kamaji#1337 (finalizer removal errors). #1660 used the same manual remedy and was closed in February as already fixed. The draft #4817 notes the same Kamaji dependency for clusters with their own etcd.