Repository navigation
Description
A deleted TenantControlPlane can stay on finalizer.kamaji.clastix.io long after its deletion, because every reconcile after a stall reaches the DataStore with an already-expired context. I hit this in Cozystack e2e on kamaji 26.3.6-edge: 30 minutes after the delete the TCP was still there, with both apiserver pods running.
What the log shows
The manager runs with the defaults: one TCP worker and --controller-reconcile-timeout=30s. Timeline for one TCP (UTC), main controller only:
- 16:29:29: last "has been reconciled".
- 16:35:52: spec update. 16:36:59: deletionTimestamp set. The main controller logs nothing for this TCP until 16:50:06. The only line for it in that window, "soot manager completed its process" at 16:36:59, comes from the soot controller.
- 16:50:06: a reconcile ends with "unable to create the DataStore user". Its stack goes through the create/update branch (
resources.Handle→Setup.CreateOrUpdate), so it read the object before the deletion. That reconcile ran for at least 13 minutes. - 16:55:13 and 17:01:56: deletion reconciles log "marked for deletion" and fail about 70µs later:
2026-09-24T16:55:13Z INFO marked for deletion, performing clean-up {"controller": "tenantcontrolplane", "controllerGroup": "kamaji.clastix.io", "controllerKind": "TenantControlPlane", "TenantControlPlane": {"name":"kubernetes-test-latest-version","namespace":"tenant-test"}, "namespace": "tenant-test", "name": "kubernetes-test-latest-version", "reconcileID": "78163948-9c1c-43ef-9e72-fa7a7f216caf"}
{"level":"warn","ts":"2026-09-24T16:55:13.730999Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x13e4f627a3c0/etcd.tenant-root.svc:2379","peer":"Peer{Addr: <nil>, LocalAddr: <nil>, AuthInfo: <nil>}","method":"/etcdserverpb.Auth/RoleGet","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
The TCP ends with generation: 5, status.observedGeneration: 3 and the finalizer still set. The kamaji pod holds its lease and idles at about 1.4m CPU. The etcd DataStore had leader elections between 16:31:39 and 16:33:04 and a stable leader after that.
One thing in the code that fits the expired-context part
GetResources (controllers/resources.go:76 at master) calls utilities.AreGatewayResourcesAvailable on every reconcile. That ends in discoveryClient.ServerGroups() / ServerResourcesForGroupVersion() (internal/utilities/gateway_discovery.go:37 and :55), which take no context, so the call is bounded only by client-go's discovery default of 32s per request, more than the whole 30s reconcile budget. When discovery is slow, the rest of the reconcile runs on a dead context, and the first uncached call, the etcd user or role lookup, fails at once, as above. The same availability is already computed once at startup in SetupWithManager to decide whether Gateway types are watched, so the per-reconcile check does not buy anything. I will send a PR that reuses the startup result.
What I cannot explain
The discovery calls are bounded, so they do not explain a single reconcile running for 13 minutes. I did not find another call on that path that ignores the reconcile context, and without a goroutine dump I cannot say which one it was. The deletion reconciles also fail instantly although the deletion path does not run discovery; the only live call before the DataStore there is the APIReader.Get at controllers/tenantcontrolplane_controller.go:99, which does honour the context. Ideas on where else to look are welcome.
#1225 (first finding) is about the same visible result, a TCP stuck in Terminating, but there the finalizer error is swallowed and nothing requeues. Here the reconciles do requeue and fail each time.
Activity
fix(kamaji): resolve Gateway API availability once at startup (#4501)
prometherion commentedon Oct 11, 2026
It sounds related to #1337: any thoughts on this, @lexfrei?
lexfrei commentedon Oct 11, 2026
@prometherion Same symptom, different path. #1337 helps when removing the finalizer fails and the error was swallowed, so nothing requeued. Here the deletion reconciles do requeue, and they fail earlier: the etcd RoleGet runs on a context that is already expired, before the finalizer step is reached. So #1337 doesn't change what I saw. Shame, a leet PR fixing it would have been a nice story. The expired context is what #1327 goes after. The 13-minute reconcile is still unexplained on my side.