Repository navigation
Description
What steps did you take and what happened:
[A clear and concise description on how to REPRODUCE the bug.]
- Deploy older CAPI Operator with CAPZ, CAPG, CAPA, and several other providers of older version.
- Perform upgrade of CAPZ among other providers to a later version together with CAPI operator itself.
- Monitor the
Readycondition on providers. - CAPZ provider keeps staying unready for more then 30m with:
status:
conditions:
- lastTransitionTime: "2026-03-30T21:57:00Z"
message: ""
observedGeneration: 2
reason: MinimumReplicasUnavailable
status: "False"
type: Readybut the deployment actually has enough replicas and reports MinimumReplicasAvailable condition True on both azureserviceoperator-controller-manager and capz-controller-manager deployments.
Patching any of these deployments with an arbitrary annotation or waiting for 30m leads to Ready condition to become finally True
What did you expect to happen:
- Health check to pickup current provider deployment health at all times.
- For multiple deployment providers, like CAPZ, the condition should be aggregated from all provider deployments, not a random one.
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
It seems that current health check mechanism has an uncovered case with providerDeploymentPredicates reacts only on UpdateEvent.
In case of simultaneous upgrade of multiple providers together with CAPI Operator pod, with CAPZ included, the CAPZ deployment may reach ready state before the new CAPI operator pod starts watching events after becoming a leader.
As a result, the initial cache population gets the current state of CAPZ deployment in a Create event (as an initial state during watch initialization on empty cache), and this is ignored by the controller by the predicates logic.
After some time, when the controller sync period re-syncs the cache, an Update event is triggered with no deployment changes, which finally gets to the health check controller, and updates condition to a long set Ready True condition.
While the original scenario is a race condition, and is hard to reproduce, I was able to reproduce it manually with these steps:
kubectl patch infrastructureprovider azure \
-n capz-system \
--subresource=status \
--type='merge' \
-p '{"status":{"conditions":[]}}'
to patch the provider status condition. CAPZ deployments are ready.
- Restart the CAPI Operator pod.
- See no health check logs for any provider, and conditions staying unchanged.
- Even deleting hash annotation on the provider to force reconciliation doesn't set the
Readycondition.
Environment:
- Cluster-api-operator version: any
- Cluster-api version: v1.11.2
- Minikube/KIND version:
- Kubernetes version: (use
kubectl version): 1.35 - OS (e.g. from
/etc/os-release):
/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api-operator/labels?q=area for the list of labels]
Activity
k8s-ci-robot commentedon Apr 3, 2026
This issue is currently awaiting triage.
If CAPI Operator contributors determines this is a relevant issue, they will accept it by applying the
triage/acceptedlabel and provide further guidance.The
triage/acceptedlabel can be added by org members by writing/triage acceptedin a comment.Details
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
fix(cluster-api): resync capi-operator every minute (#4939)
fix(cluster-api): resync capi-operator every minute