Skip to content

Commit 02b87b6

Browse files
myasnikovdaniilclaude
myasnikovdaniil
and
claude
committed
fix(apps/kubernetes): bound the pre-delete hook's tenant HelmRelease wait (#3271)
The pre-delete hook (delete.yaml) deleted the tenant's addon HelmReleases with an unbounded `kubectl delete … --wait=true`. When the tenant has no working nodes (LINSTOR/ZFS exhaustion, worker VMs that never joined, or a nodeGroup at minReplicas: 0), those uninstalls cannot converge — in-tenant cilium stays InstallFailed because cilium-operator has nothing to schedule on — so the hook hung indefinitely, the parent Helm uninstall failed on the pre-cleanup Job, and the Kubernetes CR was stuck deleting. Bound the wait with --timeout=120s; on expiry, force-clear the Flux finalizer from any child HelmRelease still present so it can be garbage-collected — safe because the whole tenant (control plane torn down in the steps below) is being destroyed, so there is nothing left to orphan. A healthy tenant's children uninstall well within the timeout and never reach the force path. Also unblocks the render-side kubernetes-oidc-{system,customconfig} Chainsaw e2e suites, which deterministically hit this teardown hang under the Chainsaw cleanup budget. Assisted-By: Claude <[email protected]> Co-Authored-By: Claude Opus 4.8 <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent ff0d0d7 commit 02b87b6

1 file changed

Lines changed: 27 additions & 3 deletions

File tree

‎packages/apps/kubernetes/templates/delete.yaml‎

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -61,9 +61,33 @@ spec:
6161
done
6262
6363
echo "Step 2: Deleting HelmReleases with label cozystack.io/target-cluster-name={{ .Release.Name }}"
64-
kubectl -n {{ .Release.Namespace }} delete helmreleases.helm.toolkit.fluxcd.io \
65-
-l "cozystack.io/target-cluster-name={{ .Release.Name }}" \
66-
--ignore-not-found=true --wait=true
64+
# --wait=true blocks until the child HelmReleases (in-tenant cilium,
65+
# coredns, csi, ingress-nginx, …) are fully gone. When the tenant
66+
# never became a working cluster — no Ready nodes: storage/LINSTOR
67+
# exhaustion, worker VMs that never joined, or a nodeGroup at
68+
# minReplicas: 0 — those uninstalls cannot converge (e.g. in-tenant
69+
# cilium stays InstallFailed because cilium-operator has nothing to
70+
# schedule on), and an UNBOUNDED wait hangs this hook forever: the
71+
# parent Helm uninstall then fails on the pre-cleanup Job and the
72+
# Kubernetes CR is stuck deleting (see #3271). Bound the wait; if it
73+
# expires, force-remove the Flux finalizer from any child HR still
74+
# present so it can be garbage-collected. This is safe here because
75+
# the whole tenant — its control plane included (torn down in Steps
76+
# 3/4 below) — is being destroyed, so there is nothing left to
77+
# orphan; a healthy tenant's children uninstall well within the
78+
# timeout and never reach the force path.
79+
if ! kubectl -n {{ .Release.Namespace }} delete helmreleases.helm.toolkit.fluxcd.io \
80+
-l "cozystack.io/target-cluster-name={{ .Release.Name }}" \
81+
--ignore-not-found=true --wait=true --timeout=120s; then
82+
echo "WARNING: child HelmReleases for {{ .Release.Name }} did not uninstall within timeout (tenant likely has no working nodes); force-clearing their finalizers so teardown can proceed" >&2
83+
for hr in $(kubectl -n {{ .Release.Namespace }} get helmreleases.helm.toolkit.fluxcd.io -l "cozystack.io/target-cluster-name={{ .Release.Name }}" -o name 2>/dev/null || true); do
84+
[ -n "$hr" ] || continue
85+
echo " Force-clearing finalizers on $hr"
86+
kubectl -n {{ .Release.Namespace }} patch "$hr" \
87+
--type=merge -p '{"metadata":{"finalizers":null}}' \
88+
|| echo "WARNING: failed to clear finalizers on $hr" >&2
89+
done
90+
fi
6791
6892
echo "Step 3: Deleting KamajiControlPlane {{ .Release.Name }}"
6993
kubectl -n {{ .Release.Namespace }} delete kamajicontrolplanes.controlplane.cluster.x-k8s.io {{ .Release.Name }} \

0 commit comments

Comments
 (0)