Skip to content

[Bug]: invalid PATCH operation with storage and resource resize #9786

New issue
New issue
Closed
Defect
#10427
Closed
[Bug]: invalid PATCH operation with storage and resource resize#9786
Defect
#10427
Assignees
mnencia
Labels
no-stale

Description

@mateusoliveira43
mateusoliveira43
opened on Jan 21, 2026

Is there an existing issue already for this bug?

  • I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

  • I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

  • I have read the troubleshooting guide and I think this is a new bug.

Contact Details

No response

Version

trunk (main)

What version of Kubernetes are you using?

other (unsupported)

What is your Kubernetes environment?

Self-managed: kind (evaluation)

How did you install the operator?

Other

What happened?

If both storage and resources (CPU and Memory) are changed in a PATCH, Cluster becomes stuck in Waiting for the instances to become active phase. Operator should not allow this operation

Reproduction steps

Install CNPG and create Cluster

git clone [email protected]:cloudnative-pg/cloudnative-pg.git
cd cloudnative-pg
./hack/setup-cluster.sh create load deploy

kubectl apply -f c.yaml

wait Cluster become healthy

$ kubectl get cluster,pod,pvc,pv
NAME                                         AGE   INSTANCES   READY   STATUS                     PRIMARY
cluster.postgresql.cnpg.io/cluster-example   55s   2           2       Cluster in healthy state   cluster-example-1

NAME                       READY   STATUS    RESTARTS   AGE
pod/cluster-example-1      1/1     Running   0          49s
pod/cluster-example-2      1/1     Running   0          17s
pod/csi-hostpathplugin-0   8/8     Running   0          3h14m

NAME                                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/cluster-example-1   Bound    pvc-21cafeaf-27ea-48e5-93b1-e7f5925c78c5   1Gi        RWO            csi-hostpath-sc   <unset>                 54s
persistentvolumeclaim/cluster-example-2   Bound    pvc-fde91ea4-f345-44a0-b615-34ac9abce5a2   1Gi        RWO            csi-hostpath-sc   <unset>                 32s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS      VOLUMEATTRIBUTESCLASS   REASON   AGE
persistentvolume/pvc-21cafeaf-27ea-48e5-93b1-e7f5925c78c5   1Gi        RWO            Delete           Bound    default/cluster-example-1   csi-hostpath-sc   <unset>                          54s
persistentvolume/pvc-fd

Update Cluster

kubectl patch cluster cluster-example --type=merge --patch '{"spec": {"storage": {"size": "2Gi"}, "resources": {"limits": {"cpu": "2", "memory": "2Gi"},"requests": {"cpu": "2", "memory": "2Gi"}}}}'
$ kubectl get cluster,pod,pvc,pv
NAME                                         AGE   INSTANCES   READY   STATUS                                       PRIMARY
cluster.postgresql.cnpg.io/cluster-example   71s   2           1       Waiting for the instances to become active   cluster-example-1

NAME                       READY   STATUS    RESTARTS   AGE
pod/cluster-example-1      1/1     Running   0          65s
pod/csi-hostpathplugin-0   8/8     Running   0          3h15m

NAME                                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/cluster-example-1   Bound    pvc-21cafeaf-27ea-48e5-93b1-e7f5925c78c5   1Gi        RWO            csi-hostpath-sc   <unset>                 70s
persistentvolumeclaim/cluster-example-2   Bound    pvc-fde91ea4-f345-44a0-b615-34ac9abce5a2   1Gi        RWO            csi-hostpath-sc   <unset>                 48s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS      VOLUMEATTRIBUTESCLASS   REASON   AGE
persistentvolume/pvc-21cafeaf-27ea-48e5-93b1-e7f5925c78c5   2Gi        RWO            Delete           Bound    default/cluster-example-1   csi-hostpath-sc   <unset>                          70s
persistentvolume/pvc-fde91ea4-f345-44a0-b615-34ac9abce5a2   2Gi        RWO            Delete           Bound    default/cluster-example-2   csi-hostpath-sc   <unset>                          48s

Cluster resource

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-example
spec:
  logLevel: debug

  instances: 2

  storage:
    resizeInUseVolumes: true
    size: 1Gi
    storageClass: csi-hostpath-sc

  resources:
    limits:
      cpu: "1"
      memory: 1Gi
    requests:
      cpu: "1"
      memory: 1Gi

Relevant log output

the last logs from CNPG operator are

{"level":"debug","ts":"2026-01-21T16:48:01.110386729Z","msg":"Reconciliation loop start","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-example","namespace":"default"},"namespace":"default","name":"cluster-example","reconcileID":"52cd011c-9e87-423d-8949-8412f3ee9818","caller":"internal/controller/cluster_controller.go:155"}
{"level":"debug","ts":"2026-01-21T16:48:01.160273006Z","msg":"Instance pod not active. Retrying...","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-example","namespace":"default"},"namespace":"default","name":"cluster-example","reconcileID":"52cd011c-9e87-423d-8949-8412f3ee9818","inactiveInstances":["cluster-example-2"],"caller":"internal/controller/cluster_controller.go:785"}
{"level":"debug","ts":"2026-01-21T16:48:01.16032984Z","msg":"Reconciliation loop end","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-example","namespace":"default"},"namespace":"default","name":"cluster-example","reconcileID":"52cd011c-9e87-423d-8949-8412f3ee9818","caller":"internal/controller/cluster_controller.go:157"}
{"level":"debug","ts":"2026-01-21T16:48:01.635213088Z","msg":"Reconciliation loop start","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-example","namespace":"default"},"namespace":"default","name":"cluster-example","reconcileID":"bbba603b-391d-4bfd-b62b-419d1aa641c0","caller":"internal/controller/cluster_controller.go:155"}
{"level":"debug","ts":"2026-01-21T16:48:01.664772948Z","msg":"Waiting for Pods to be ready","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-example","namespace":"default"},"namespace":"default","name":"cluster-example","reconcileID":"bbba603b-391d-4bfd-b62b-419d1aa641c0","caller":"internal/controller/cluster_controller.go:1010","podStatus":{"healthy":["cluster-example-1"]}}
{"level":"debug","ts":"2026-01-21T16:48:01.664793781Z","msg":"Reconciliation loop end","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cluster-example","namespace":"default"},"namespace":"default","name":"cluster-example","reconcileID":"bbba603b-391d-4bfd-b62b-419d1aa641c0","caller":"internal/controller/cluster_controller.go:157"}

Code of Conduct

  • I agree to follow this project's Code of Conduct

Activity

  1. mateusoliveira43
    assigned
    gbartolini
    on Jan 21, 2026
  2. mateusoliveira43
    added
    triagePending triage
    on Jan 21, 2026

  3. mateusoliveira43 commented on Jan 21, 2026

    @mateusoliveira43
    mateusoliveira43
    on Jan 21, 2026
    ContributorAuthor

    adding also full CNPG operator log and Cluster Pod log (the one that remained running)

    pod.log

    cnpg.log

  4. dosubot commented on Jan 21, 2026

    @dosubot
    dosubotBot
    on Jan 21, 2026 – with DosuBot

    Hi @mateusoliveira43! I'm Dosu and I’m helping the cloudnative-pg team.

    Thanks for sharing the full operator and pod logs. I can't access external attachments directly, but based on the code and similar reports, here's what's happening:

    CloudNativePG currently allows simultaneous changes to storage and resources in a single PATCH, but doesn't sequence or block them as a special case. When both are changed together, the operator tries to reconcile, but if pods or PVCs aren't ready (for example, if a PVC resize is waiting for a pod restart, and the pod can't start due to resource changes), the cluster gets stuck in the "Waiting for the instances to become active" phase. This is a known deadlock scenario—neither the pod nor the PVC can progress, and the operator just keeps waiting for readiness, sometimes indefinitely (example).

    There is no explicit validation in the webhook or controller to prevent this, so the best workaround is to make storage and resource changes in separate steps: resize storage first, wait for the cluster to become healthy, then update resources (or vice versa). If the cluster is already stuck, check for pods or PVCs that are not progressing, and try deleting the affected pod and PVC to force recreation and recovery (community advice). In some cases, making a small data change on the primary can also trigger the cluster to become healthy (example).

    This is a gap in the current operator logic, and improving validation or sequencing for combined changes would help prevent these deadlocks in the future.

    To reply, just mention @dosu.

    Share context across your team and agents. Try Dosu.

    Leave Feedback Learn about cloudnative-pg with Dosu Add Dosu to your team

  5. github-actions commented on Mar 23, 2026

    @github-actions
    github-actionsBot
    on Mar 23, 2026 – with GitHub Actions
    Contributor

    This issue is stale because it has been open for 60 days with no activity.

  6. github-actions
    added
    Stale
    on Mar 23, 2026

  7. github-actions commented on Apr 6, 2026

    @github-actions
    github-actionsBot
    on Apr 6, 2026 – with GitHub Actions
    Contributor

    This issue was closed because it has been inactive for 14 days since being marked as stale.

  8. github-actions
    closed this as not plannedon Apr 6, 2026
  9. mnencia
    reopened this on Apr 9, 2026
  10. mnencia
    added
    no-stale
    and removed
    Stale
    on Apr 9, 2026
  11. mnencia
    assigned
    mnencia
    and unassigned
    gbartolini
    on Apr 9, 2026
  12. mnencia
    removed
    triagePending triage
    on Apr 9, 2026
  13. mnencia
    added the
    Defect
    issue type on Apr 9, 2026
  14. armru
    added a commit that references this issue on Apr 14, 2026

    fix(pvc-resize): avoid deadlock on simultaneous storage and resource …

    1c9d44e
  15. leonardoce
    closed this as completedin 9147871on Apr 16, 2026
  16. cnpg-bot
    added 2 commits that reference this issue on Apr 16, 2026

    fix(pvc-resize): avoid deadlock on simultaneous storage and resource …

    f58902a

    fix(pvc-resize): avoid deadlock on simultaneous storage and resource …

    881706a
  17. leonardoce
    added a commit that references this issue on Apr 16, 2026

    fix(pvc-resize): avoid deadlock on simultaneous storage and resource …

    57e95c0
  18. pull
    added a commit that references this issue on Apr 16, 2026

    fix(pvc-resize): avoid deadlock on simultaneous storage and resource …

    9147871
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

no-stale

Type

Defect

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions