Repository navigation
Server-side apply on an application drops its HelmRelease finalizer, so deleting it orphans the release #4482
Description
Server-side apply on any apps.cozystack.io application strips the Flux finalizer from its HelmRelease, and deleting the application afterwards leaves the Helm release and everything it created behind. helm-controller does not put the finalizer back on its own, so this is not a race: after one apply, every later delete of that application orphans its release.
Reproduced on a fresh cluster with a Tenant, which makes the leftover easy to see:
kubectl apply --filename tenant-qa.yaml # Tenant qa in tenant-root
kubectl --namespace tenant-root get hr tenant-qa -o jsonpath='{.metadata.finalizers}'
# ["finalizers.fluxcd.io"]
kubectl apply --server-side --field-manager=probe --force-conflicts --filename tenant-qa.yaml
kubectl --namespace tenant-root get hr tenant-qa -o jsonpath='{.metadata.finalizers}'
# (empty, and still empty minutes later)
kubectl --namespace tenant-root delete tenants.apps.cozystack.io qa
kubectl --namespace tenant-root get secret -l owner=helm,name=tenant-qa # release still "deployed"
kubectl get namespace tenant-qa # still ActiveA merge patch through the same API keeps the finalizer; only apply loses it. The same thing happens through the in-process apiserver with a fake client on both the current main and the Kubernetes 1.37 branch (#4473), so it is not new.
REST.Update rebuilds the HelmRelease from the Application and copies back only the shard label and spec.suspend from the live object (rest.go#L619). metadata.finalizers is not among them, so whatever the incoming Application carries wins, and an applied Application carries none. Keeping the live HelmRelease's finalizers the same way would fix it. A test that applies an application whose HelmRelease has a finalizer and expects the finalizer to survive would pin it.
To clean up an orphan: helm --namespace tenant-root uninstall <release>.
Activity
2mind commentedon Nov 4, 2026
We hit the same failure on our internal platform (a fork of the cozystack app packages: postgres/redis/clickhouse; platform components themselves are vanilla upstream images: cozystack-controller v1.1.3, helm-controller v1.4.3 via flux-tenants).
Our reproduction adds a data point on the trigger: the Application (apps.cozystack.io PaxPostgres/PaxRedis/PaxClickHouse) is written by kcp's syncagent using server-side apply (it projects our catalog CR into the tenant namespace), so the finalizer loss is not a one-off CLI apply — every projected object loses the finalizer before helm-controller ever sees it, and every delete then orphans the release. Observed identically on all three kinds: after deleting the application,
helm liststill shows the releasedeployedand the pods keep running (plus a leftover Completed init Job from helm hooks); we currently clean up manually withhelm uninstall.Two extra details:
spec.uninstall.deletionPropagationcannot be worked around from the outside: cozystack-api reverts a manual patch of the generated HelmRelease on its next reconcile.pkg/registry/apps/application/rest.godoes not set anyUninstallpolicy, so nothing restores GC safety today even on a fresh cluster without SSA clients — the guard lives only in the finalizer SSA keeps dropping.Happy to provide more repro details if useful.