[Bug]: pod creation stuck during pvc resize

Is there an existing issue already for this bug?

I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

I have read the troubleshooting guide and I think this is a new bug.

Contact Details

[email protected]

Version

1.26 (latest patch)

What version of Kubernetes are you using?

1.32

What is your Kubernetes environment?

Other

How did you install the operator?

Helm

What happened?

We rolled out 2 changes simultaneously: an update to the Pod spec and an increase in PVC size. I'm not sure what happened initially as this was an automatic rollout. The Cluster ended up stuck 'waiting for instances to become active', while no new Pod resource for a replica was created. The PVC for the replica had the storage increased but the filesystem resize had not happened yet, with status stating it was waiting for a Pod restart so the filesystem resize could be completed. Result: the cnpg operator waited for the PVC to finish resizing before starting a new Pod, while the PVC resize was waiting for the operator to start a new Pod. This was stuck for about 16 hours. Impact was negligible in this case as it concerned a replica pod.

I sort of verified this was the problem by starting a Pod with the PVC in question mounted, which allowed the filesystem resize to complete, and a few seconds later the replica pod was created.

The fix would be for the operator to check if the PVC is waiting for a pod to start, instead of waiting until a resize is completely finished before starting the Pod.

Cluster resource

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  creationTimestamp: "2025-05-02T12:31:06Z"
  generation: 8
  name: pg17-defectdojo
  namespace: it-apps
  resourceVersion: "2287187937"
  uid: 64fd9dbe-63c6-41c9-9bb9-4b967960661d
spec:
  affinity:
    podAntiAffinityType: preferred
  backup:
    barmanObjectStore:
      data:
        compression: bzip2
      destinationPath: s3://barman-6efe6e22-f2e4-4fce-91bb-6c174704d354
      endpointURL: http://rook-ceph-rgw-ceph-obj-hdd.rook-ceph.svc
      s3Credentials:
        accessKeyId:
          key: AWS_ACCESS_KEY_ID
          name: barman
        secretAccessKey:
          key: AWS_SECRET_ACCESS_KEY
          name: barman
      wal:
        compression: bzip2
    retentionPolicy: 3d
    target: prefer-standby
  bootstrap:
    initdb:
      database: app
      encoding: UTF8
      localeCType: C
      localeCollate: C
      owner: app
  enablePDB: true
  enableSuperuserAccess: false
  failoverDelay: 0
  imageName: ghcr.io/cloudnative-pg/postgresql:17.4-bookworm@sha256:7d9959d92111fa4192f5db986ea3e74270c4cc609f947d212ae9383b5f6008c1
  inheritedMetadata:
    labels:
      app.kubernetes.io/instance: defectdojo
  instances: 2
  logLevel: info
  maxSyncReplicas: 0
  minSyncReplicas: 0
  monitoring:
    customQueriesConfigMap:
    - key: queries
      name: cnpg-default-monitoring
    disableDefaultQueries: false
    enablePodMonitor: false
  postgresGID: 26
  postgresUID: 26
  postgresql:
    parameters:
      archive_mode: "on"
      archive_timeout: 5min
      dynamic_shared_memory_type: posix
      full_page_writes: "on"
      log_destination: csvlog
      log_directory: /controller/log
      log_filename: postgres
      log_rotation_age: "0"
      log_rotation_size: "0"
      log_truncate_on_rotation: "false"
      logging_collector: "on"
      max_parallel_workers: "32"
      max_replication_slots: "32"
      max_worker_processes: "32"
      shared_memory_type: mmap
      shared_preload_libraries: ""
      ssl_max_protocol_version: TLSv1.3
      ssl_min_protocol_version: TLSv1.3
      wal_keep_size: 128MB
      wal_level: logical
      wal_log_hints: "on"
      wal_receiver_timeout: 5s
      wal_sender_timeout: 5s
    syncReplicaElectionConstraint:
      enabled: false
  primaryUpdateMethod: switchover
  primaryUpdateStrategy: unsupervised
  replicationSlots:
    highAvailability:
      enabled: true
      slotPrefix: _cnpg_
    synchronizeReplicas:
      enabled: true
    updateInterval: 30
  resources:
    limits:
      cpu: "4"
      memory: 4Gi
    requests:
      cpu: 500m
      memory: 500Mi
  smartShutdownTimeout: 180
  startDelay: 3600
  stopDelay: 1800
  storage:
    resizeInUseVolumes: true
    size: 4Gi
    storageClass: ceph-block-hdd
  switchoverDelay: 3600
status:
  availableArchitectures:
  - goArch: amd64
    hash: 024c5162e1dee336b21121d781397b347d04939a857d9d8a47664d1796e8b7c9
  - goArch: arm64
    hash: 63f3a3fbe19b8fe1c26a0e516ecb965c3412e390729a02706825399f585d9298
  certificates:
    clientCASecret: pg17-defectdojo-ca
    expirations:
      pg17-defectdojo-ca: 2025-07-31 12:26:06 +0000 UTC
      pg17-defectdojo-replication: 2025-07-31 12:26:06 +0000 UTC
      pg17-defectdojo-server: 2025-07-31 12:26:06 +0000 UTC
    replicationTLSSecret: pg17-defectdojo-replication
    serverAltDNSNames:
    - pg17-defectdojo-rw
    - pg17-defectdojo-rw.it-apps
    - pg17-defectdojo-rw.it-apps.svc
    - pg17-defectdojo-rw.it-apps.svc.cluster.local
    - pg17-defectdojo-r
    - pg17-defectdojo-r.it-apps
    - pg17-defectdojo-r.it-apps.svc
    - pg17-defectdojo-r.it-apps.svc.cluster.local
    - pg17-defectdojo-ro
    - pg17-defectdojo-ro.it-apps
    - pg17-defectdojo-ro.it-apps.svc
    - pg17-defectdojo-ro.it-apps.svc.cluster.local
    serverCASecret: pg17-defectdojo-ca
    serverTLSSecret: pg17-defectdojo-server
  cloudNativePGCommitHash: 1535f3c17
  cloudNativePGOperatorHash: 024c5162e1dee336b21121d781397b347d04939a857d9d8a47664d1796e8b7c9
  conditions:
  - lastTransitionTime: "2025-07-09T07:21:27Z"
    message: Cluster is Ready
    reason: ClusterIsReady
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-07-09T07:20:46Z"
    message: Continuous archiving is working
    reason: ContinuousArchivingSuccess
    status: "True"
    type: ContinuousArchiving
  - lastTransitionTime: "2025-07-08T15:07:54Z"
    message: Backup was successful
    reason: LastBackupSucceeded
    status: "True"
    type: LastBackupSucceeded
  configMapResourceVersion:
    metrics:
      cnpg-default-monitoring: "2227200574"
  currentPrimary: pg17-defectdojo-2
  currentPrimaryTimestamp: "2025-07-09T07:20:45.381075Z"
  firstRecoverabilityPoint: "2025-07-05T15:07:45Z"
  firstRecoverabilityPointByMethod:
    barmanObjectStore: "2025-07-05T15:07:45Z"
  healthyPVC:
  - pg17-defectdojo-1
  - pg17-defectdojo-2
  image: ghcr.io/cloudnative-pg/postgresql:17.4-bookworm@sha256:7d9959d92111fa4192f5db986ea3e74270c4cc609f947d212ae9383b5f6008c1
  instanceNames:
  - pg17-defectdojo-1
  - pg17-defectdojo-2
  instances: 2
  instancesReportedState:
    pg17-defectdojo-1:
      ip: 10.241.16.210
      isPrimary: false
      timeLineID: 6
    pg17-defectdojo-2:
      ip: 10.241.6.248
      isPrimary: true
      timeLineID: 6
  instancesStatus:
    healthy:
    - pg17-defectdojo-1
    - pg17-defectdojo-2
  lastSuccessfulBackup: "2025-07-08T15:07:48Z"
  lastSuccessfulBackupByMethod:
    barmanObjectStore: "2025-07-08T15:07:48Z"
  latestGeneratedNode: 2
  managedRolesStatus: {}
  onlineUpdateEnabled: true
  pgDataImageInfo:
    image: ghcr.io/cloudnative-pg/postgresql:17.4-bookworm@sha256:7d9959d92111fa4192f5db986ea3e74270c4cc609f947d212ae9383b5f6008c1
    majorVersion: 17
  phase: Cluster in healthy state
  poolerIntegrations:
    pgBouncerIntegration: {}
  pvcCount: 2
  readService: pg17-defectdojo-r
  readyInstances: 2
  secretsResourceVersion:
    applicationSecretVersion: "2227200853"
    clientCaSecretVersion: "2123430247"
    replicationSecretVersion: "2123430257"
    serverCaSecretVersion: "2123430247"
    serverSecretVersion: "2123430256"
  switchReplicaClusterStatus: {}
  targetPrimary: pg17-defectdojo-2
  targetPrimaryTimestamp: "2025-07-09T07:20:39.963657Z"
  timelineID: 6
  topology:
    instances:
      pg17-defectdojo-1: {}
      pg17-defectdojo-2: {}
    nodesUsed: 2
    successfullyExtracted: true
  writeService: pg17-defectdojo-rw

Relevant log output

no log output in the operator related to this that I could find.

Code of Conduct

I agree to follow this project's Code of Conduct

Edit: I've just spotted #7296 which I think explains my issue. Ignore my noise :)

FWIW I've just experienced the same thing, but without any other configuration changes - just a PVC resize.

Scenario:

Azure
Resize in place enabled
Running 1.24 (upgrading is on the list of things to do, at the same time as moving off Azure)

Ended up creating a temporary pod also, this immediately kicked everything back into life.

+1 just encountered the same issue. Solved it by destroying the pod and creating a new instance + PVC, did not think about mounting to a temporary pod to finish the resize

This issue is stale because it has been open for 60 days with no activity.

Not stale

This issue is stale because it has been open for 60 days with no activity.

not stale

This issue is stale because it has been open for 60 days with no activity.

Navigation Menu

Repository navigation

[Bug]: pod creation stuck during pvc resize #7997

Is there an existing issue already for this bug?

I have read the troubleshooting guide

I am running a supported version of CloudNativePG

Contact Details

Version

What version of Kubernetes are you using?

What is your Kubernetes environment?

How did you install the operator?

What happened?

Cluster resource

Relevant log output

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: pod creation stuck during pvc resize #7997

Description

Is there an existing issue already for this bug?

I have read the troubleshooting guide

I am running a supported version of CloudNativePG

Contact Details

Version

What version of Kubernetes are you using?

What is your Kubernetes environment?

How did you install the operator?

What happened?

Cluster resource

Relevant log output

Code of Conduct

Activity

theangryangel commented on Jul 20, 2025

max-ae commented on Aug 8, 2025

github-actions commented on Oct 11, 2025

bh-tt commented on Oct 13, 2025

github-actions commented on Dec 17, 2025

bh-tt commented on Dec 17, 2025

github-actions commented on Feb 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions