Repository navigation
cluster-api: capi-operator stops reconciling after v1.7 upgrade and blocks kubernetes-rd until restarted #4926
Description
What is wrong?
After upgrade from v1.6.4 to v1.7.0-rc.1 HelmRelease capi-providers-core failed with timeout waiting for: [CoreProvider/cozy-cluster-api/cluster-api status: 'InProgress']. CoreProvider had Ready=False with MinimumReplicasUnavailable while all deployments in cozy-cluster-api were 1/1. Because of the dependency kubernetes-rd, kubernetes-nodes-rd and computeplane-rd stayed False for more than 30 minutes.
About 25 minutes after upgrade start capi pods were restarting (7 to 13 restarts each) with leader election lost, requests to apiserver timed out:
Failed to update lock: Put "https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/cozy-cluster-api/leases/controller-leader-election-capi?timeout=5s": context deadline exceeded
"Problem running manager" err="leader election lost"
capi-operator had no new log lines after its last such error. kubectl -n cozy-cluster-api rollout restart deploy/capi-operator-cluster-api-operator fixed it, CoreProvider became Ready=True and dependent releases converged in a few minutes.
Seen once, cause is unknown. Not sure if it is related to the cilium update in the same upgrade.
Environment
Cozystack v1.6.4 -> v1.7.0-rc.1, isp-full variant, 3 nodes with 8 vCPU and 32 GiB each.
How it should work?
Providers reconcile after upgrade without manual restart of capi-operator.
Activity
myasnikovdaniil commentedon Oct 14, 2026
Correction to what I wrote: restarting capi-operator did not fix this. CoreProvider became Ready about 12 minutes before my manual restart, the convergence I saw after the restart was a later upgrade to a PR build (it restarted cilium, kube-ovn and the CAPI managers too), so I mixed two upgrade waves in the report.
Cause is upstream bug in capi-operator v0.19.0 that we ship, its health check controller ignores Deployment create events (kubernetes-sigs/cluster-api-operator#1031). After operator start the CoreProvider condition is not refreshed until the first resync, in the operator log it was about 20 minutes after start and then every 10 minutes, and every operator restart resets that. Fixed in kubernetes-sigs/cluster-api-operator#1032, first released in v0.27.0, I'll bump operator in a separate PR.
What caused the apiserver timeouts about 25 minutes after upgrade start is still unknown, no metrics or logs were stored on that cluster for that time.
fix(cluster-api): resync capi-operator every minute (#4939)
myasnikovdaniil commentedon Oct 29, 2026
Mitigated by #4939: capi-operator now runs with
--sync-period=1m, so a provider leftReady=Falseacross an operator restart recovers in about two minutes instead of about twenty. It is in v1.7.0-rc.2 and later through #5031 (packages/system/capi-operator/values.yaml:9) and on release-1.6 through #5081. Proper fix needs the operator bump, that is #5037, so I'm closing this one.