Skip to content

RollingUpdate strategy does not seem to work with leader election #419

Closed

Description

@rouke-broersma

RollingUpdate strategy does not seem to work with the default enabled leader election. The old instance keeps the lease claimed, and the new instance has a failing readiness probe until it can claim the lease. This was discovered while testing the helm chart pull request, but I believe the same should be happening with the kustomize install since it also uses rolling update strategy by default.

This is what I observe when triggering a rollout restart operation:

image

The readiness probe is unhealthy until the new pod gets the lease:

image

{"level":"info","ts":"2025-06-24T15:26:01.370038527Z","msg":"starting manager"}
{"level":"info","ts":"2025-06-24T15:26:01.37156252Z","msg":"starting server","name":"health probe","addr":"[::]:8081"}
{"level":"info","ts":"2025-06-24T15:26:01.372609226Z","msg":"attempting to acquire leader lease cloudnative-postgres/822e3f5c.cnpg.io..."}

Because the readiness probe of the new pod doesn't become ready until it can get the lease, there aren't sufficient ready pods and kubernetes doesn't send the shutdown signal to the old leader. Because the old leader doesn't get a shutdown signal, it has no reason to drop the lease claim.

This seems like a bug in plugin-barman-cloud leader election, but means that rolling update currently doesn't properly work as a default. Leader election does work in the cloudnative-pg operator, so something in the leader election works differently in the barman cloud plugin

Originally posted by @rouke-broersma in cloudnative-pg/charts#598 (comment)

Activity

  1. jstewart612 commented on Aug 16, 2025

    @jstewart612

    I just hit this exact issue. I'm looking for alternative options for the health check but not finding any.

  2. added a commit that references this issue on Oct 27, 2025
    49f1096
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions