Repository navigation
Description
[Bug]: PVC with FileSystemResizePending condition causes pod creation deadlock
Summary
Fixed by #9981
When a PVC has the FileSystemResizePending condition (volume resized at storage layer, filesystem resize pending), CloudNativePG incorrectly classifies it as "resizing" even when no pod is attached. This prevents pod creation, causing a deadlock where the filesystem resize can never complete.
What happened?
After a PVC resize operation completes at the storage layer, Kubernetes sets the FileSystemResizePending condition on the PVC. This condition indicates that the volume has been resized but the filesystem resize is pending - it requires a pod to mount the volume to complete the resize.
Current behavior: CloudNativePG's classifyPVC() function returns resizing for such PVCs, which prevents the operator from creating a new pod. Without a pod to mount the volume, the filesystem resize cannot complete.
Expected behavior: When a PVC has FileSystemResizePending and no pod is attached, it should be classified as dangling to trigger pod recreation. This allows the pod to mount the volume and complete the filesystem resize.
Deadlock sequence
- User or dynamic storage triggers a PVC resize
- Storage layer completes the volume resize
- Kubernetes adds
FileSystemResizePendingcondition to PVC - Pod is deleted or crashes
- CloudNativePG sees PVC is "resizing" and does NOT create a new pod
- Filesystem resize never completes because no pod mounts the volume
- Cluster is stuck - manual intervention required
Reproduction
This can be reproduced on any Kubernetes cluster with a CSI driver that supports volume expansion:
- Create a CloudNativePG cluster with a small PVC (e.g., 1Gi)
- Resize the PVC to a larger size (e.g., 2Gi)
- Delete the pod while the resize is in progress (after storage resize completes but before filesystem resize)
- Observe that the pod is never recreated
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: test-resize
spec:
instances: 1
storage:
size: 1Gi
storageClass: standard # Must support volume expansionEnvironment
- CloudNativePG Version: All versions (trunk/main)
- Kubernetes Version: Any with CSI volume expansion support
- Environment: Any (observed on AKS, likely affects all)
- Installation Method: Any
Relevant log output
The cluster will show the PVC stuck in resizing state with no pod:
$ kubectl get pvc -l cnpg.io/cluster=test-resize
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-resize-1 Bound pv-xxx 2Gi RWO standard 10m
$ kubectl get pvc test-resize-1 -o jsonpath='{.status.conditions[*].type}'
Resizing FileSystemResizePending
$ kubectl get pods -l cnpg.io/cluster=test-resize
No resources found.
Root cause
In pkg/reconciler/persistentvolumeclaim/status.go, the classifyPVC() function checks isResizing() before checking if a pod exists:
// Current (buggy) logic:
if isResizing(pvc) {
return resizing // Returns resizing even if no pod exists
}
if hasPod(pvc, podList) {
return healthy
}The fix is to check pod existence first, and only classify as resizing if either:
- A pod exists (filesystem resize can complete), OR
- No
FileSystemResizePendingcondition (storage resize still in progress)
Proposed fix
Cluster resource
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: test-resize
spec:
instances: 1
storage:
size: 1Gi
storageClass: standard # Must support volume expansionRelevant log output
$ kubectl get pvc -l cnpg.io/cluster=test-resize
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-resize-1 Bound pv-xxx 2Gi RWO standard 10m
$ kubectl get pvc test-resize-1 -o jsonpath='{.status.conditions[*].type}'
Resizing FileSystemResizePending
$ kubectl get pods -l cnpg.io/cluster=test-resize
No resources found.Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
fix(pvc): prevent deadlock on filesystem resize without pod (#9981)
fix(pvc): prevent deadlock on filesystem resize without pod (#9981)
fix(pvc): prevent deadlock on filesystem resize without pod (#9981)