[Bug]: PVC resize on Azure is not properly detected

Is there an existing issue already for this bug?

I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

I have read the troubleshooting guide and I think this is a new bug.

Contact Details

[email protected]

Version

1.26.0-rc1

What version of Kubernetes are you using?

1.31

What is your Kubernetes environment?

Cloud: Azure AKS

How did you install the operator?

YAML manifest

What happened?

Observed a weird scenario during PVC resize on Azure where the Resizing condition is not removed when the new state FileSystemResizePending is appearing:

status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 4Gi
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2025-04-08T15:53:24Z"
    status: "True"
    type: Resizing
  - lastProbeTime: null
    lastTransitionTime: "2025-04-08T15:53:27Z"
    message: Waiting for user to (re-)start a pod to finish file system resize of
      volume on node.
    status: "True"
    type: FileSystemResizePending
  phase: Bound

As example, on Google:

status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2025-04-08T13:10:09Z"
    message: Waiting for user to (re-)start a pod to finish file system resize of
      volume on node.
    status: "True"
    type: FileSystemResizePending
  phase: Bound

This cause a new pod is not getting created because of the PVC still having the Resizing condition set to true, and this only happens on Azure (not on AWS nor on GCP) of the 3 major CSPs

Cluster resource

apiVersion: postgresql.k8s.enterprisedb.io/v1
kind: Cluster
metadata:
  annotations:
    container.apparmor.security.beta.kubernetes.io/initdb: runtime/default
    container.apparmor.security.beta.kubernetes.io/join: runtime/default
    container.apparmor.security.beta.kubernetes.io/postgres: runtime/default
    disktuner.biganimal.enterprisedb.com/enabled: "true"
    k8s.enterprisedb.io/livepatch-status: '{"ews-enable":"not-applicable","pgcollation-enabled-1":"complete"}'
    k8s.enterprisedb.io/managedRoleRevoke: disabled
    k8s.enterprisedb.io/reconcilePodSpec: disabled
    kubectl.kubernetes.io/restartedAt: "2025-04-08T15:53:24Z"
    prometheus.io/path: /metrics
    prometheus.io/port: "9187"
    prometheus.io/scheme: http
    prometheus.io/scrape: "true"
    starlight.enterprisedb.com/dataPlanePostgresClusterGeneration: "2"
    starlight.enterprisedb.com/dataPlanePostgresOperatorVersion: v1.295.4
  creationTimestamp: "2025-04-08T15:45:33Z"
  generation: 2
  name: p-4t8gvjq5e8
  namespace: default
  ownerReferences:
  - apiVersion: starlight.enterprisedb.com/v1beta1
    controller: true
    kind: DataPlanePostgresCluster
    name: p-4t8gvjq5e8
    uid: df4c1e24-622e-49bb-8254-fa63ae65503e
  resourceVersion: "92807807"
  uid: cd3b36c9-c449-4e8e-80a4-34efdd16e96b
spec:
  affinity:
    additionalPodAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: k8s.enterprisedb.io/cluster
            operator: Exists
        namespaceSelector: {}
        topologyKey: kubernetes.io/hostname
      - labelSelector:
          matchLabels:
            k8s.enterprisedb.io/cluster: p-4t8gvjq5e8
            k8s.enterprisedb.io/podRole: instance
        namespaceSelector: {}
        topologyKey: topology.kubernetes.io/zone
    enablePodAntiAffinity: false
    nodeSelector:
      upm.enterprisedb.io/instance-type: Standard_D4s_v3
      upm.enterprisedb.io/node-capability: ultra_disk
      upm.enterprisedb.io/node-type: postgresql
      upm.enterprisedb.io/rotating-number: "0"
    tolerations:
    - effect: NoSchedule
      key: ReservedForUltraDisk
      operator: Equal
      value: "true"
    - effect: NoSchedule
      key: ReservedForPostgres
      operator: Equal
      value: "true"
  backup:
    barmanObjectStore:
      azureCredentials:
        inheritFromAzureAD: true
      data:
        compression: snappy
        jobs: 1
      destinationPath: https://pgvg508km8.blob.core.windows.net/p-4t8gvjq5e8/
      historyTags:
        backupRetentionPolicy: keep
      tags:
        backupRetentionPolicy: expire
      wal:
        compression: snappy
        maxParallel: 2
    target: primary
  bootstrap:
    initdb:
      dataChecksums: false
      database: edb_admin
      encoding: UTF8
      localeCType: C
      localeCollate: C
      owner: edb_admin
      postInitApplicationSQL:
      - CREATE EXTENSION IF NOT EXISTS edb_wait_states WITH SCHEMA pg_catalog
      postInitSQL:
      - REVOKE ALL ON DATABASE postgres FROM PUBLIC
      - CREATE USER streaming_replica REPLICATION
      - GRANT ALL ON DATABASE postgres TO streaming_replica
      postInitTemplateSQL:
      - GRANT EXECUTE ON FUNCTION pg_create_restore_point(text) TO pg_ba_admin
      secret:
        name: p-4t8gvjq5e8-pwd
      walSegmentSize: 16
  certificates:
    serverCASecret: p-4t8gvjq5e8-tls
    serverTLSSecret: p-4t8gvjq5e8-tls
  description: test-matteo-FileSystemResizePending
  enablePDB: true
  enableSuperuserAccess: false
  failoverDelay: 0
  imageName: upmdev.azurecr.io/enterprisedb/starlight:16.8.0-4-debian-ba@sha256:2ef54e43ceafb2249df8859ce280a5639005cc858fd85deaf240d72bff63f524
  imagePullPolicy: Always
  imagePullSecrets:
  - name: edbcloud-cred
  inheritedMetadata:
    labels:
      azure.workload.identity/use: "true"
      k8s.enterprisedb.io/beacon: enabled
      k8s.enterprisedb.io/beacon-project-id: prj_vcdbreleasOAqZwM
      k8s.enterprisedb.io/beacon-resource-id: p-4t8gvjq5e8
      k8s.enterprisedb.io/beacon-schema-collection: enabled
  instances: 1
  logLevel: info
  managed:
    roles:
    - connectionLimit: -1
      createdb: true
      createrole: true
      ensure: present
      inRoles:
      - pg_monitor
      - pg_signal_backend
      - pg_ba_admin
      inherit: true
      login: true
      name: edb_admin
      passwordSecret:
        name: p-4t8gvjq5e8-pwd
  maxSyncReplicas: 0
  minSyncReplicas: 0
  monitoring:
    customQueriesSecret:
    - key: queries
      name: postgresql-operator-default-monitoring
    disableDefaultQueries: false
    enablePodMonitor: false
  postgresGID: 26
  postgresUID: 26
  postgresql:
    enableAlterSystem: true
    parameters:
      archive_mode: "on"
      archive_timeout: 5min
      autovacuum_max_workers: "5"
      autovacuum_vacuum_cost_limit: "3000"
      ba_extensions: amcheck,auth_delay,autocluster,autoinc,bloom,btree_gin,btree_gist,citext,cube,dblink,dbms_scheduler,earthdistance,edb_dbo,edb_job_scheduler,edb_pg_tuner,edb_wait_states,edbspl,fuzzystrmatch,hstore,hstore_plperl,index_advisor,insert_username,intagg,intarray,isn,jsonb_plperl,lo,ltree,moddatetime,mongo_fdw,mysql_fdw,old_snapshot,pageinspect,pg_buffercache,pg_cron,pg_failover_slots,pg_freespacemap,pg_prewarm,pg_stat_statements,pg_surgery,pg_trgm,pg_visibility,pgaudit,pgcrypto,pgrowlocks,pgstattuple,plperl,pltcl,postgres_fdw,query_advisor,refdata,refint,seg,spl_check,sql_profiler,sslinfo,tablefunc,tcn,tsm_system_rows,tsm_system_time,unaccent,uuid-ossp,vector,xml2,postgis,postgis_topology,fuzzystrmatch,address_standardizer,address_standardizer_data_us,postgis_tiger_geocoder,postgis_sfcgal,bluefin,pg_squeeze,pgaa
      checkpoint_completion_target: "0.9"
      checkpoint_timeout: "900"
      cpu_tuple_cost: "0.03"
      cron.database_name: edb_admin
      cron.use_background_workers: "on"
      dynamic_shared_memory_type: posix
      effective_cache_size: "1572864"
      full_page_writes: "on"
      log_destination: csvlog
      log_directory: /controller/log
      log_filename: postgres
      log_rotation_age: "0"
      log_rotation_size: "0"
      log_truncate_on_rotation: "false"
      logging_collector: "on"
      maintenance_work_mem: "377487"
      max_parallel_workers: "32"
      max_replication_slots: "32"
      max_worker_processes: "64"
      pg_stat_statements.max: "10000"
      pg_stat_statements.track: all
      pgaudit.log: role, ddl
      pgaudit.log_relation: "on"
      random_page_cost: "1.1"
      shared_buffers: "524288"
      shared_memory_type: mmap
      shared_preload_libraries: ""
      ssl_max_protocol_version: TLSv1.3
      ssl_min_protocol_version: TLSv1.3
      superuser_reserved_connections: "10"
      tcp_keepalives_idle: "120"
      tcp_keepalives_interval: "30"
      wal_buffers: "8192"
      wal_compression: "on"
      wal_keep_size: 512MB
      wal_level: logical
      wal_log_hints: "on"
      wal_receiver_timeout: "3600000"
      wal_sender_timeout: "3600000"
    pg_hba:
    - hostssl all streaming_replica all cert
    shared_preload_libraries:
    - auth_delay
    - auto_explain
    - passwordcheck
    - pgaudit
    - pg_prewarm
    - pg_stat_statements
    - sql-profiler
    - pg_failover_slots
    - autocluster
    - refdata
    - query_advisor
    - edb_wait_states
    - pg_cron
    - pg_squeeze
    syncReplicaElectionConstraint:
      enabled: false
  primaryUpdateMethod: switchover
  primaryUpdateStrategy: unsupervised
  replicationSlots:
    highAvailability:
      enabled: true
      slotPrefix: _cnp_
    synchronizeReplicas:
      enabled: true
    updateInterval: 30
  resources: {}
  serviceAccountTemplate:
    metadata:
      annotations:
        azure.workload.identity/client-id: 2c076722-60dc-478e-a2da-688c1b7c083e
        azure.workload.identity/tenant-id: fb0c7e10-e8dc-41ff-8203-2faa9e0b3d0f
      labels:
        azure.workload.identity/use: "true"
  smartShutdownTimeout: 180
  startDelay: 3600
  stopDelay: 1800
  storage:
    resizeInUseVolumes: true
    size: 8Gi
    storageClass: edb-azure-ultra-v1
  switchoverDelay: 3600
status:
  availableArchitectures:
  - goArch: amd64
    hash: f897ccb11df38e52fb693e12a50068028eeb794bac7af33ec404069f8517fee2
  - goArch: arm64
    hash: be65a164987a3ed22f286403f22d60090a44df63db5bc34ad6939bb33c50c151
  - goArch: ppc64le
    hash: 969acaf7b0955d5c44f0e769fa47d37a6c85f06a2065e846d330b00396c38657
  - goArch: s390x
    hash: 99b17506799f4835f0d679eee69ad973282ad0d68abf50dd957f028198c39c2e
  certificates:
    clientCASecret: p-4t8gvjq5e8-ca
    expirations:
      p-4t8gvjq5e8-ca: 2025-07-07 15:40:33 +0000 UTC
      p-4t8gvjq5e8-replication: 2025-07-07 15:40:33 +0000 UTC
      p-4t8gvjq5e8-tls: 2025-07-07 14:46:09 +0000 UTC
    replicationTLSSecret: p-4t8gvjq5e8-replication
    serverAltDNSNames:
    - p-4t8gvjq5e8-rw
    - p-4t8gvjq5e8-rw.default
    - p-4t8gvjq5e8-rw.default.svc
    - p-4t8gvjq5e8-rw.default.svc.cluster.local
    - p-4t8gvjq5e8-r
    - p-4t8gvjq5e8-r.default
    - p-4t8gvjq5e8-r.default.svc
    - p-4t8gvjq5e8-r.default.svc.cluster.local
    - p-4t8gvjq5e8-ro
    - p-4t8gvjq5e8-ro.default
    - p-4t8gvjq5e8-ro.default.svc
    - p-4t8gvjq5e8-ro.default.svc.cluster.local
    serverCASecret: p-4t8gvjq5e8-tls
    serverTLSSecret: p-4t8gvjq5e8-tls
  cloudNativePostgresqlCommitHash: 60d870f
  cloudNativePostgresqlOperatorHash: f897ccb11df38e52fb693e12a50068028eeb794bac7af33ec404069f8517fee2
  conditions:
  - lastTransitionTime: "2025-04-08T15:53:24Z"
    message: Cluster Is Not Ready
    reason: ClusterIsNotReady
    status: "False"
    type: Ready
  - lastTransitionTime: "2025-04-08T15:50:32Z"
    message: Continuous archiving is working
    reason: ContinuousArchivingSuccess
    status: "True"
    type: ContinuousArchiving
  - lastTransitionTime: "2025-04-08T15:50:53Z"
    message: velero addon is disabled
    reason: Disabled
    status: "False"
    type: k8s.enterprisedb.io/velero
  - lastTransitionTime: "2025-04-08T15:50:53Z"
    message: external-backup-adapter addon is disabled
    reason: Disabled
    status: "False"
    type: k8s.enterprisedb.io/externalBackupAdapter
  - lastTransitionTime: "2025-04-08T15:50:54Z"
    message: external-backup-adapter-cluster addon is disabled
    reason: Disabled
    status: "False"
    type: k8s.enterprisedb.io/externalBackupAdapterCluster
  - lastTransitionTime: "2025-04-08T15:50:54Z"
    message: kasten addon is disabled
    reason: Disabled
    status: "False"
    type: k8s.enterprisedb.io/kasten
  - lastTransitionTime: "2025-04-08T15:51:37Z"
    message: Backup was successful
    reason: LastBackupSucceeded
    status: "True"
    type: LastBackupSucceeded
  configMapResourceVersion: {}
  currentPrimary: p-4t8gvjq5e8-1
  currentPrimaryTimestamp: "2025-04-08T15:50:28.769253Z"
  firstRecoverabilityPoint: "2025-04-08T15:51:32Z"
  firstRecoverabilityPointByMethod:
    barmanObjectStore: "2025-04-08T15:51:32Z"
  image: upmdev.azurecr.io/enterprisedb/starlight:16.8.0-4-debian-ba@sha256:2ef54e43ceafb2249df8859ce280a5639005cc858fd85deaf240d72bff63f524
  instanceNames:
  - p-4t8gvjq5e8-1
  instances: 1
  lastSuccessfulBackup: "2025-04-08T15:51:32Z"
  lastSuccessfulBackupByMethod:
    barmanObjectStore: "2025-04-08T15:51:32Z"
  latestGeneratedNode: 1
  licenseStatus:
    licenseStatus: Not enforced
    repositoryAccess: false
    valid: false
  managedRolesStatus:
    byStatus:
      reconciled:
      - edb_admin
      reserved:
      - postgres
      - streaming_replica
    passwordStatus:
      edb_admin:
        resourceVersion: "92801777"
        transactionID: 745
  onlineUpdateEnabled: true
  phase: Waiting for the instances to become active
  phaseReason: Some instances are not yet active. Please wait.
  poolerIntegrations:
    pgBouncerIntegration: {}
  pvcCount: 1
  readService: p-4t8gvjq5e8-r
  resizingPVC:
  - p-4t8gvjq5e8-1
  secretsResourceVersion:
    applicationSecretVersion: "92801777"
    clientCaSecretVersion: "92803336"
    managedRoleSecretVersion:
      p-4t8gvjq5e8-pwd: "92801777"
    metrics:
      postgresql-operator-default-monitoring: "92803365"
    replicationSecretVersion: "92803337"
    serverCaSecretVersion: "92802834"
    serverSecretVersion: "92802834"
  switchReplicaClusterStatus: {}
  targetPrimary: p-4t8gvjq5e8-1
  targetPrimaryTimestamp: "2025-04-08T15:45:34.627856Z"
  timelineID: 1
  topology:
    successfullyExtracted: true
  writeService: p-4t8gvjq5e8-rw

Relevant log output

postgresql-operator-controller-manager-598bbd9d9-gptnr manager {"level":"info","ts":"2025-04-08T13:10:03.22570749Z","msg":"Pod rollout required","controller":"cluster","controllerGroup":"postgresql.k8s.enterprisedb.io","controllerKind":"Cluster","Cluster":{"name":"p-4t8gvn0gzx","namespace":"default"},"namespace":"default","name":"p-4t8gvn0gzx","reconcileID":"2cf4cf34-5f0c-4c1c-988f-ca1e854562c3","podName":"p-4t8gvn0gzx-1","reason":"cluster has been explicitly restarted via annotation"}

postgresql-operator-controller-manager-598bbd9d9-gptnr manager {"level":"info","ts":"2025-04-08T13:10:03.26457534Z","msg":"Cluster has become unhealthy","controller":"cluster","controllerGroup":"postgresql.k8s.enterprisedb.io","controllerKind":"Cluster","Cluster":{"name":"p-4t8gvn0gzx","namespace":"default"},"namespace":"default","name":"p-4t8gvn0gzx","reconcileID":"2cf4cf34-5f0c-4c1c-988f-ca1e854562c3"}

postgresql-operator-controller-manager-598bbd9d9-gptnr manager {"level":"info","ts":"2025-04-08T13:10:03.267563289Z","msg":"Recreating instance pod","controller":"cluster","controllerGroup":"postgresql.k8s.enterprisedb.io","controllerKind":"Cluster","Cluster":{"name":"p-4t8gvn0gzx","namespace":"default"},"namespace":"default","name":"p-4t8gvn0gzx","reconcileID":"2cf4cf34-5f0c-4c1c-988f-ca1e854562c3","pod":"p-4t8gvn0gzx-1","to":"upmdev.azurecr.io/enterprisedb/starlight:16.8.0-4-debian-ba@sha256:2ef54e43ceafb2249df8859ce280a5639005cc858fd85deaf240d72bff63f524","reason":"cluster has been explicitly restarted via annotation"}

Code of Conduct

I agree to follow this project's Code of Conduct

@mbiagetti

Ciao @mbiagetti! Sono Dosu, un bot qui per aiutarti a risolvere bug, rispondere a domande e diventare un collaboratore. Sono qui per supportarti mentre aspetti un mantenitore umano. Come posso aiutarti oggi?

I found a similar discussion regarding a persistent volume claim (PVC) resize error during cluster creation, which might be related to your issue. The discussion is titled "persistentvolumeclaims resize error during cluster creation" and is currently unsolved ^[1]. You might find some insights or additional context there that could help address the problem you're experiencing with PVC resize on Azure.

To continue talking to Dosu, mention @dosu.

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

#7297 (comment)

When triggering two operations (VM +disk update) at the same time, the operator asks the pod restart for VM upgrade to complete. Azure preserves both PVC conditions at the same time, while AWS & GCP removes the Resizing one when it adds the FileSystemResizePending. This will force the operator to wait and not recreate the pod on the new VM.

diff --git a/pkg/reconciler/persistentvolumeclaim/resources.go b/pkg/reconciler/persistentvolumeclaim/resources.go
index 1ca30256f..bcaaef34a 100644
--- a/pkg/reconciler/persistentvolumeclaim/resources.go
+++ b/pkg/reconciler/persistentvolumeclaim/resources.go
@@ -65,15 +65,21 @@ external:
        return true
 }

-// isResizing returns true if PersistentVolumeClaimResizing condition is present
+// isResizing returns true if PersistentVolumeClaimResizing condition is present and not PersistentVolumeClaimFileSystemResizePending
 func isResizing(pvc corev1.PersistentVolumeClaim) bool {
+       isResizing := false
+       isFileSystemResizePending := false
+
        for _, condition := range pvc.Status.Conditions {
                if condition.Type == corev1.PersistentVolumeClaimResizing {
-                       return true
+                       isResizing = true
+               }
+               if condition.Type == corev1.PersistentVolumeClaimFileSystemResizePending {
+                       isFileSystemResizePending = true
                }
        }

-       return false
+       return (isResizing && !isFileSystemResizePending)
 }

@leonardoce

The PersistentVolumeClaimFileSystemResizePending condition is managed by the kubelet, meaning it cannot be resolved if no pod is running. In my opinion, we should open a bug report in the azure-csi-driver repository or in the external-resizer, depending on where the issue lies. I'm curious to hear what @leonardoce thinks about this.

@mnencia

@mnencia @leonardoce I've asked here also, kubernetes-sigs/azuredisk-csi-driver#3027

Also this seems interesting
https://github.com/kubernetes-csi/external-resizer/blob/6d07f88640dff903e21e9342a099816b0aa8c306/pkg/controller/controller.go#L533

I just took a look at the ControllerExpandVolume func implementation of different cloud provider with Azure, implementation is similar, the issue could be in external-resizer version, AKS are using external resizer v1.13.2 from AKS 1.29. Not sure whether this is the real issue since Resizing means resize is not complete, there is FileSystemResizePending job to do.

@andyzhangx

Thanks @andyzhangx

Not sure whether this is the real issue since Resizing means resize is not complete, there is FileSystemResizePending job to do.

Yes, I'm wondering if the resizer should works as GCP and AWS (removing the condition) or CNPG should be able to handle this scenarios too

that's expected behavior (quotes from external-resizer owner): kubernetes-sigs/azuredisk-csi-driver#3027 (comment)

only AKS "breaks" because AKS is using latest v1.13.2 external-resizer, that's expected behavior

This issue is stale because it has been open for 60 days with no activity.

This issue was closed because it has been inactive for 14 days since being marked as stale.

Hello, we're still seeing the cnpg storage issue. Currently, we have to manually attach the PVC to a dummy pod to trigger the CSI expansion. Is there a plan for the CNPG operator to automate this process so we can increase cluster storage sizes more seamlessly?

@mbiagetti

@mbiagetti having the same issue as @josgutie - we're also attaching the PVCs to dummy pods as a fix, although this is less than ideal in daily operations.

It all worked until we upgraded K8s to 1.34.2 and thereby to Azure CSI driver 1.33.7 which introduced an issue related to volume expansion (see kubernetes-sigs/azuredisk-csi-driver#3478). However, this issue has been fixed with CSI driver 1.33.8 which is already deployed in our cluster.

Right now, the NodeResizeError condition is gone on resizing, but the FileSystemResizePending still remains with Waiting for user to (re-)start a pod to finish file system resize of volume on node

Navigation Menu

Repository navigation

[Bug]: PVC resize on Azure is not properly detected #7324

Is there an existing issue already for this bug?

I have read the troubleshooting guide

I am running a supported version of CloudNativePG

Contact Details

Version

What version of Kubernetes are you using?

What is your Kubernetes environment?

How did you install the operator?

What happened?

Cluster resource

Relevant log output

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: PVC resize on Azure is not properly detected #7324

Description

Is there an existing issue already for this bug?

I have read the troubleshooting guide

I am running a supported version of CloudNativePG

Contact Details

Version

What version of Kubernetes are you using?

What is your Kubernetes environment?

How did you install the operator?

What happened?

Cluster resource

Relevant log output

Code of Conduct

Activity

dosubot commented on Apr 9, 2025

nfaust-edb commented on Apr 9, 2025

mnencia commented on Apr 9, 2025

mbiagetti commented on Apr 15, 2025

mbiagetti commented on Apr 15, 2025

andyzhangx commented on Apr 15, 2025

mbiagetti commented on Apr 15, 2025

andyzhangx commented on Apr 26, 2025

andyzhangx commented on Apr 26, 2025

github-actions commented on Jul 26, 2025

github-actions commented on Aug 10, 2025

josgutie commented on Feb 2, 2026

arenk commented on Feb 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions