Deleting a tenant with a Kafka app hangs the namespace in Terminating (KafkaTopic strimzi.io/topic-operator finalizer) #3793

Activity

1. github-actions

   added

   triage/needs-triageIndicates an issue needs triage by a maintainerIndicates an issue needs triage by a maintainer

   on Aug 13, 2026
2. dosubot commented on Aug 13, 2026

   

   dosubotBot

   on Aug 13, 2026 – with DosuBot

   More actions

   Good write-up. This is the same class of "operator torn down before its finalizer-owning CRs" bug that has already been fixed for several other apps in the repo, and the mechanism is well understood.

   Root cause

   The Kafka chart's topics.yaml creates KafkaTopic CRs with no finalizers field [1] — Strimzi's entity-operator adds the strimzi.io/topic-operator finalizer at runtime when it takes ownership. When the Kafka app (and the entity-operator pod) is torn down first, nothing remains to drain that finalizer, so the KafkaTopic objects block namespace GC indefinitely. This was already observed as a side-effect in PR #2934, which noted that a CrashLoopBackOff entity-operator leaves KafkaTopics stuck with the same finalizer.

   No pre-delete hook exists for Kafka today

   The packages/apps/kafka/templates directory contains only kafka.yaml, topics.yaml, and monitoring/config templates — there is no delete.yaml or any pre-delete hook [2]. The tenant-level cleanup-job.yaml deletes HelmReleases in the namespace with a 90s bounded wait, but that only removes the Kafka HelmRelease — it has no authority over KafkaTopic CRs or their Strimzi finalizers.

   Existing patterns to follow

   The repo already has two merged fixes for the identical problem class:

   1. PR #3078 — the packages/apps/kubernetes chart's delete.yaml deletes the TenantControlPlane with --wait=true --timeout=120s to let Kamaji drain its own finalizer, then as a backstop patches metadata.finalizers: null on the datastore-config secret if it still exists.
   2. PR #3598 — the tenant pre-delete hook with bounded waits, activeDeadlineSeconds: 270, backoffLimit: 0, and restartPolicy: Never, all sized to fit inside helm-controller's 300s uninstall budget.

   Suggested fix direction

   Add a Helm pre-delete hook Job to the kafka chart (mirroring the kubernetes app's delete.yaml) that:

   1. Deletes all KafkaTopic CRs in the namespace with --wait=true --timeout=<N>s — while the entity-operator is still alive to process the deletion and drain its finalizer.
   2. As a backstop after the timeout, patches metadata.finalizers: null on any surviving KafkaTopic CRs — safe because the Kafka CR and its operator are being torn down in the same release.
   3. Carries activeDeadlineSeconds set below the HelmRelease uninstall budget, backoffLimit: 0, and restartPolicy: Never, following the same arithmetic used in the tenant and kubernetes hooks.

   Your manual workaround (patching finalizers: null) is correct in the interim.

   To reply, just mention @dosu.

   ---

   Docs are dead. Just use Dosu.

   
3. yankawai commented on Aug 21, 2026

   

   europrinter (yankawai)

   on Aug 21, 2026

   Contributor

   More actions

   Reproduced on Cozystack with Strimzi 0.45. Helm uninstall removes the topic operator together with the KafkaTopics, so a topic still carrying the strimzi.io/topic-operator finalizer blocks deletion, and the next install with the same release name times out on the leftover topic.

   Opened #3938 with a pre-delete hook that deletes only release-labelled KafkaTopics while the operator is still running, without forcing finalizer removal. Render and unit tests pass, and the kafka e2e suite gains an absence assert for the release topics after deletion.
4. IvanHunters commented on Sep 15, 2026

   

   IvanHunters

   on Sep 15, 2026

   CollaboratorAuthor

   More actions

   Still reproduces on main at b222cb658, and there is a second entry point worth recording: deleting the app hits it, not only deleting the tenant.

   From the cozyreport of an E2E (in-tree) run, kafka suite, step verify-pvc-reclaim-on-delete:

   16:35:32  DELETE  RUN    apps.cozystack.io/v1alpha1/Kafka @ tenant-test/test
   16:40:32  DELETE  ERROR  context deadline exceeded
   

   Five minutes, the operation's whole budget, on deleting the Kafka CR itself. No tenant is being removed here.

   What the snapshot holds at that moment:

   KafkaTopic  kafka-test-test-orders   deletionTimestamp=2026-09-14T16:38:16Z  finalizers=[strimzi.io/topic-operator]
   KafkaTopic  kafka-test-test-results  deletionTimestamp=2026-09-14T16:38:16Z  finalizers=[strimzi.io/topic-operator]
   HelmRelease kafka-test               deletionTimestamp set                   finalizers=[finalizers.fluxcd.io]
   

   and not one Kafka pod left in the namespace — the entity operator that owns that finalizer is already gone. So the ordering is the same one #3938 fixes, reached from the app side: the release tears down the topic operator, the topics keep a finalizer only it can clear, and the HelmRelease never finalizes behind them.

   Two things this adds.

   The e2e suite already catches it. hack/e2e-chainsaw/kafka/chainsaw-test.yaml fails on this in ordinary runs, so the defect is not waiting on someone to reproduce it by hand — it is one of the failures authors of unrelated PRs are currently working around. #3938 is open since 9 Sep and the hook is not on main, so every run that reaches that step spends five minutes proving the same thing.

   And the app-deletion path may want its own assertion. #3938 adds an absence assert for the release topics after deletion, which covers this if the suite deletes the app; worth confirming that is the path the new assert exercises, since the tenant path and the app path tear things down in a different order.

   👍React with 👍1Reacted by europrinter
5. IvanHunters

   added 2 commits that reference this issue on Sep 15, 2026

   fix(kafka): delete release topics before the topic operator is removed (

   ...

   a19fd02

   fix(kafka): keep the pre-delete hook's credentials as long as its Job (…

   ...

   b323d5d
6. Aleksei Sviridkin (lexfrei)

   added

   kind/bugCategorizes issue or PR as related to a bugCategorizes issue or PR as related to a bug

   area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)

   triage/not-reproducibleIndicates an issue can not be reproduced as describedIndicates an issue can not be reproduced as described

   and removed

   triage/needs-triageIndicates an issue needs triage by a maintainerIndicates an issue needs triage by a maintainer

   on Sep 23, 2026
7. lexfrei commented on Sep 23, 2026

   

   Aleksei Sviridkin (lexfrei)

   on Sep 23, 2026

   Contributor

   More actions

   Fixed by #3938, which deletes the release's KafkaTopics in a pre-delete hook while the topic operator is still running, and #4280 for the hook credentials. The release-1.6 backport is #4421.
8. Aleksei Sviridkin (lexfrei)

   closed this as completedon Sep 23, 2026
9. myasnikovdaniil

   added a commit that references this issue on Sep 25, 2026

   [Backport release-1.6] fix(kafka): delete release topics before the t…

   ...

   ece050c