[kubernetes] HelmRelease stuck on deletion when tenant controlplane is unavailable (pre-delete hook hangs on child HR finalizers) #2473

Summary

When a tenant Kubernetes cluster's controlplane is broken (e.g. etcd quorum lost, Kamaji failures, admin-kubeconfig secret missing), the parent kubernetes HelmRelease in the management cluster gets stuck on deletion indefinitely. The root cause is the pre-delete hook in packages/apps/kubernetes/templates/delete.yaml, which deletes child HelmReleases with --wait=true. Those child HRs (cilium, coredns, cert-manager, csi, …) reference the tenant apiserver via spec.kubeConfig.secretRef. Flux cannot run their uninstall against a dead apiserver, finalizers never clear, the cleanup Job blocks forever, the pre-delete hook never succeeds, and Helm refuses to remove the parent release.

Reproduction

1. Create a tenant Kubernetes cluster via the kubernetes app.
2. Break the controlplane so the tenant apiserver is unreachable. Any of these reproduces:
   • Delete/corrupt the <name>-admin-kubeconfig Secret.
   • Lose etcd quorum on the TenantControlPlane (e.g. the backing DataVolumes are gone).
   • Make the Kamaji TenantControlPlane pods unable to start.
3. Delete the tenant HelmRelease: kubectl -n tenant-foo delete hr kubernetes.

Observed: the HR sits in Terminating forever. The <release>-pre-cleanup Job is Active with a running pod. Its log is stuck at:

Step 2: Deleting HelmReleases with label cozystack.io/target-cluster-name=<release>

The child HRs remain, each with a finalizers.fluxcd.io finalizer that Flux cannot release because its uninstall reconcile keeps failing to talk to the tenant apiserver.

Root cause

packages/apps/kubernetes/templates/delete.yaml Step 2:

kubectl -n {{ .Release.Namespace }} delete helmreleases.helm.toolkit.fluxcd.io \
  -l "cozystack.io/target-cluster-name={{ .Release.Name }}" \
  --ignore-not-found=true --wait=true

--wait=true blocks on finalizer removal. Child HRs such as packages/apps/kubernetes/templates/helmreleases/cilium.yaml point Flux at the tenant:

kubeConfig:
  secretRef:
    name: {{ .Release.Name }}-admin-kubeconfig
    key: super-admin.svc

So Flux's uninstall requires the tenant apiserver to be reachable. If it is not, the finalizer is never removed and the Job hangs until timeout, after which Helm records the pre-delete hook as failed and the parent HR stays wedged.

Impact

• Operators cannot delete broken tenant clusters without manually editing finalizers.
• Reinstall/recreate flows (flux reconcile, GitOps drift correction) block on the stuck parent HR.
• Can cascade: namespace deletion blocks on the HR; other resources in the tenant namespace cannot be cleaned up.

Workaround

Manually strip finalizers from the stuck child HRs so Flux gives up, then re-run the delete:

NS=tenant-foo
NAME=<release>
for hr in $(kubectl -n $NS get hr -l cozystack.io/target-cluster-name=$NAME -o name); do
  kubectl -n $NS patch $hr --type=merge -p '{"metadata":{"finalizers":[]}}'
done
# Then also clear finalizers on the cleanup Job / parent HR if needed.

Afterwards the pre-cleanup Job finishes Step 2 and the parent HR deletes.

Suggested fixes

A few options, not mutually exclusive:

1. Don't --wait in the cleanup Job. Drop --wait=true from Step 2 and instead patch the child HRs to remove their finalizers before deletion (mirrors what operators do manually). Flux will stop reconciling them once the parent is gone and garbage-collected resources are fine to leak since the tenant cluster itself is going away.

2. Detect unreachable controlplane and skip Flux uninstall. Before Step 2, probe the tenant apiserver (e.g. kubectl --kubeconfig=… version --request-timeout=5s). If it is unreachable, patch out finalizers instead of waiting.

3. Bound the wait. At minimum add --timeout to kubectl delete so the Job fails fast and Helm surfaces a real error rather than wedging forever.

Option 1 is the most robust — once the user has asked to delete the tenant cluster, in-tenant resources are going to disappear with the VMs anyway, so finalizer bookkeeping has no purpose.

Environment

• Cozystack v1.x
• packages/apps/kubernetes (any variant)