Skip to content

[Bug]: Cluster can become stuck after operator upgrade to version >= 1.27.1 #10348

New issue
New issue
Closed
#10342
Closed
[Bug]: Cluster can become stuck after operator upgrade to version >= 1.27.1#10348
#10342
Assignees
gbartolini
Labels
bug 🐛Something isn't workingtriagePending triage

Description

@mateusoliveira43
mateusoliveira43
opened on Mar 20, 2026

Is there an existing issue already for this bug?

  • I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

  • I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

  • I have read the troubleshooting guide and I think this is a new bug.

Contact Details

No response

Version

1.27 (latest patch)

What version of Kubernetes are you using?

1.35

What is your Kubernetes environment?

Self-managed: kind (evaluation)

How did you install the operator?

Helm

What happened?

It was noticed that after upgrading CNPG operator from 1.27.0 to 1.28.1, all its Clusters are stuck in Waiting for the instances to become active after operator upgrade state and only one Pod from each Cluster was updated

Steps to reproduce

git clone [email protected]:cloudnative-pg/cloudnative-pg.git
cd cloudnative-pg
hack/setup-cluster.sh create load
helm repo add cnpg https://cloudnative-pg.github.io/charts
helm upgrade --install cnpg \
  --namespace cnpg-system \
  --create-namespace \
  --version v0.26.0 \
  --set additionalArgs={"--log-level=debug"} \
  cnpg/cloudnative-pg
kubectl get deployments.apps -n cnpg-system cnpg-cloudnative-pg -o jsonpath='{.spec.template.spec.containers.*.image}{"\n"}'
kubectl rollout status deployment/cnpg-cloudnative-pg -n cnpg-system
kubectl apply -f upgrade-stall-repro.yaml
kubectl get cluster,pod -n default
# wait become healthy
helm upgrade --install cnpg \
  --namespace cnpg-system \
  --create-namespace \
  --version v0.27.1 \
  --set additionalArgs={"--log-level=debug"} \
  cnpg/cloudnative-pg
kubectl get deployments.apps -n cnpg-system cnpg-cloudnative-pg -o jsonpath='{.spec.template.spec.containers.*.image}{"\n"}'
kubectl rollout status deployment/cnpg-cloudnative-pg -n cnpg-system
kubectl get cluster,pod -n default
# confirm only one pod from each cluster was updated and Clusters are stuck in Waiting for the instances to become active state

Added Cluster manifest used for reproduction in issue, and also logs seen in the operator after upgrade

As @leonardoce noticed, this is due to change in the algorithm we use to calculate the PostgreSQL configuration hash (which changed in 1.27.1 https://github.com/cloudnative-pg/cloudnative-pg/blob/v1.27.1/pkg/postgres/configuration.go#L857-L883 ). So issue happens from any upgrade of < 1.27.0 to >= 1.27.1 (it does not happen from upgrade from 1.27.1 to 1.28.1, for example)

Cluster resource

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-a
spec:
  instances: 3

  postgresql:
    synchronous:
      dataDurability: required
      maxStandbyNamesFromCluster: 2
      method: any
      number: 3
      standbyNamesPre:
        - cluster-b
        - cluster-c

  storage:
    size: 1Gi

  externalClusters:
    - name: cluster-b
      connectionParameters:
        host: cluster-b-rw
        user: streaming_replica
        dbname: postgres
        sslmode: verify-full
      sslKey:
        name: cluster-b-replication
        key: tls.key
      sslCert:
        name: cluster-b-replication
        key: tls.crt
      sslRootCert:
        name: cluster-b-ca
        key: ca.crt
    - name: cluster-c
      connectionParameters:
        host: cluster-c-rw
        user: streaming_replica
        dbname: postgres
        sslmode: verify-full
      sslKey:
        name: cluster-c-replication
        key: tls.key
      sslCert:
        name: cluster-c-replication
        key: tls.crt
      sslRootCert:
        name: cluster-c-ca
        key: ca.crt
---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-b
spec:
  instances: 3

  bootstrap:
    pg_basebackup:
      source: cluster-a

  replica:
    primary: cluster-a
    source: cluster-a

  postgresql:
    synchronous:
      dataDurability: required
      maxStandbyNamesFromCluster: 2
      method: any
      number: 3
      standbyNamesPre:
        - cluster-a
        - cluster-c

  storage:
    size: 1Gi

  externalClusters:
    - name: cluster-a
      connectionParameters:
        host: cluster-a-rw
        user: streaming_replica
        dbname: postgres
        sslmode: verify-full
      sslKey:
        name: cluster-a-replication
        key: tls.key
      sslCert:
        name: cluster-a-replication
        key: tls.crt
      sslRootCert:
        name: cluster-a-ca
        key: ca.crt
---
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-c
spec:
  instances: 3

  bootstrap:
    pg_basebackup:
      source: cluster-a

  replica:
    primary: cluster-a
    source: cluster-a

  postgresql:
    synchronous:
      dataDurability: required
      maxStandbyNamesFromCluster: 2
      method: any
      number: 3
      standbyNamesPre:
        - cluster-a
        - cluster-b

  storage:
    size: 1Gi

  externalClusters:
    - name: cluster-a
      connectionParameters:
        host: cluster-a-rw
        user: streaming_replica
        dbname: postgres
        sslmode: verify-full
      sslKey:
        name: cluster-a-replication
        key: tls.key
      sslCert:
        name: cluster-a-replication
        key: tls.crt
      sslRootCert:
        name: cluster-a-ca
        key: ca.crt

Relevant log output

{"level":"debug","ts":"2026-03-11T00:38:58.720647141Z","msg":"Reconciliation loop start","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-a","namespace":"default"},"namespace":"default","name":"cluster-a","reconcileID":"c5da0a5b-e8f6-4017-bbeb-c20ef131545b","caller":"internal/controller/cluster_controller.go:155"}
{"level":"debug","ts":"2026-03-11T00:38:58.764993088Z","msg":"Waiting for all Pods to have the same PostgreSQL configuration","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-a","namespace":"default"},"namespace":"default","name":"cluster-a","reconcileID":"c5da0a5b-e8f6-4017-bbeb-c20ef131545b","caller":"internal/controller/cluster_controller.go:1079","configurationReport":[{"podName":"cluster-a-1","configHash":"dd37cc6e218f476ca5e5ca26cd58a2956de9520f3837618dd1bcdba3eac68a1e"},{"podName":"cluster-a-2","configHash":"6421ec10cf32231aca618678016692c4f7a6eed4963fbb92dd6089c824c0abd8"},{"podName":"cluster-a-3","configHash":"dd37cc6e218f476ca5e5ca26cd58a2956de9520f3837618dd1bcdba3eac68a1e"}]}
{"level":"debug","ts":"2026-03-11T00:38:58.765013295Z","msg":"Reconciliation loop end","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-a","namespace":"default"},"namespace":"default","name":"cluster-a","reconcileID":"c5da0a5b-e8f6-4017-bbeb-c20ef131545b","caller":"internal/controller/cluster_controller.go:157"}

Code of Conduct

  • I agree to follow this project's Code of Conduct

Activity

  1. mateusoliveira43
    assigned
    gbartolini
    on Mar 20, 2026
  2. mateusoliveira43
    added
    triagePending triage
    bug
    on Mar 20, 2026

  3. dosubot commented on Mar 20, 2026

    @dosubot
    dosubotBot
    on Mar 20, 2026 – with DosuBot

    Thanks for the detailed bug report! This issue is already being tracked and a fix is in progress.

    Root Cause: PR #8868 (released in v1.27.1) changed the configuration hash calculation algorithm to exclude cnpg.* internal parameters. During upgrades from < 1.27.1 to >= 1.27.1, pods running different operator versions compute different hashes for identical configurations, creating a deadlock where the uniformity check blocks indefinitely.

    Fix in Progress: PR #10342 addresses this by introducing an isConfigNonUniformityFromUpgrade() function that detects when hash differences are caused by operator version mismatches and skips the wait-for-uniformity step to allow the rolling update to proceed.

    Workarounds (from Issue #8371):

    1. Temporarily remove synchronous replication settings before upgrading, then re-add after completion
    2. Manually delete old pods to force recreation with the new operator version

    You may want to follow PR #10342 for the fix timeline.

    To reply, just mention @dosu.

    Share context across your team and agents. Try Dosu.

    Leave Feedback Learn about cloudnative-pg with Dosu Add Dosu to your team

  4. mnencia
    closed this as completedin #10342on Mar 22, 2026
  5. mnencia
    added a commit that references this issue on Mar 22, 2026

    fix(upgrade): break config uniformity deadlock during operator upgrad…

    324709c
  6. cnpg-bot
    added a commit that references this issue on Mar 22, 2026

    fix(upgrade): break config uniformity deadlock during operator upgrad…

    aae47e4
  7. mnencia
    added
    bug 🐛Something isn't working
    and removed
    bug
    on Jun 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bug 🐛Something isn't workingtriagePending triage

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions