Skip to content

Operator health check reconciler occasionally leaves deployment condition unchecked #1031

Closed

Description

@Danil-Grigorev

What steps did you take and what happened:
[A clear and concise description on how to REPRODUCE the bug.]

  • Deploy older CAPI Operator with CAPZ, CAPG, CAPA, and several other providers of older version.
  • Perform upgrade of CAPZ among other providers to a later version together with CAPI operator itself.
  • Monitor the Ready condition on providers.
  • CAPZ provider keeps staying unready for more then 30m with:
status:                                                                                                                                                                                                                                                                                                                      
  conditions:                                                                                                                                                                                                                                                                                                                
  - lastTransitionTime: "2026-03-30T21:57:00Z"                                                                                                                                                                                                                                                                               
    message: ""                                                                                                                                                                                                                                                                                                              
    observedGeneration: 2                                                                                                                                                                                                                                                                                                    
    reason: MinimumReplicasUnavailable                                                                                                                                                                                                                                                                                       
    status: "False"                                                                                                                                                                                                                                                                                                          
    type: Ready

but the deployment actually has enough replicas and reports MinimumReplicasAvailable condition True on both azureserviceoperator-controller-manager and capz-controller-manager deployments.

Patching any of these deployments with an arbitrary annotation or waiting for 30m leads to Ready condition to become finally True

What did you expect to happen:

  • Health check to pickup current provider deployment health at all times.
  • For multiple deployment providers, like CAPZ, the condition should be aggregated from all provider deployments, not a random one.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

It seems that current health check mechanism has an uncovered case with providerDeploymentPredicates reacts only on UpdateEvent.

In case of simultaneous upgrade of multiple providers together with CAPI Operator pod, with CAPZ included, the CAPZ deployment may reach ready state before the new CAPI operator pod starts watching events after becoming a leader.

As a result, the initial cache population gets the current state of CAPZ deployment in a Create event (as an initial state during watch initialization on empty cache), and this is ignored by the controller by the predicates logic.

After some time, when the controller sync period re-syncs the cache, an Update event is triggered with no deployment changes, which finally gets to the health check controller, and updates condition to a long set Ready True condition.

While the original scenario is a race condition, and is hard to reproduce, I was able to reproduce it manually with these steps:

kubectl patch infrastructureprovider azure \
  -n capz-system \
  --subresource=status \
  --type='merge' \
  -p '{"status":{"conditions":[]}}'

to patch the provider status condition. CAPZ deployments are ready.

  • Restart the CAPI Operator pod.
  • See no health check logs for any provider, and conditions staying unchanged.
  • Even deleting hash annotation on the provider to force reconciliation doesn't set the Ready condition.

Environment:

  • Cluster-api-operator version: any
  • Cluster-api version: v1.11.2
  • Minikube/KIND version:
  • Kubernetes version: (use kubectl version): 1.35
  • OS (e.g. from /etc/os-release):

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api-operator/labels?q=area for the list of labels]

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug.
    needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.
    on Apr 3, 2026
  2. k8s-ci-robot commented on Apr 3, 2026

    @k8s-ci-robot
    Contributor

    This issue is currently awaiting triage.

    If CAPI Operator contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

    The triage/accepted label can be added by org members by writing /triage accepted in a comment.

    Details

    Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions