Repository navigation
[Bug]: Non-ready pods are not cleared via alpha.cnpg.io/unrecoverable=true annotation #9947
Description
Is there an existing issue already for this bug?
- I have searched for an existing issue, and could not find anything. I believe this is a new bug.
I have read the troubleshooting guide
- I have read the troubleshooting guide and I think this is a new bug.
I am running a supported version of CloudNativePG
- I have read the troubleshooting guide and I think this is a new bug.
Contact Details
No response
Version
1.28 (latest patch)
What version of Kubernetes are you using?
1.34
What is your Kubernetes environment?
Other
How did you install the operator?
Helm
What happened?
I have a postgres cluster with 2 nodes.
Both nodes containing postgres pods were shut down. After nodes booted back up, one pod works normally, but another one is permanently not ready. The working pod is a primary, the failed one is marked as replica.
I believe that this is a known issue. I have seen such state previously, and fixed it by manually deleting the PVC and the failed pod.
But in the CNPG 1.28 release notes I noticed a mention of alpha.cnpg.io/unrecoverable=true annotation and tried to use it.
However, the pod is not deleted, and CNPG operator logs don't show any attempts to do so, or errors.
Logs contain only repeated "Cannot extract Pod status" message, nothing else.
Cluster resource
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
annotations:
creationTimestamp: "2025-04-02T10:40:40Z"
generation: 8
name: postgres
namespace: cm-acme-dns
resourceVersion: "1513722147"
uid: 83deae02-da95-4055-801d-3939eabe8c0e
spec:
affinity:
enablePodAntiAffinity: true
podAntiAffinityType: required
topologyKey: kubernetes.io/hostname
bootstrap:
initdb:
database: app
encoding: UTF8
localeCType: C
localeCollate: C
owner: app
enablePDB: true
enableSuperuserAccess: false
failoverDelay: 0
imageName: ghcr.io/cloudnative-pg/postgresql:17.4
instances: 2
logLevel: info
managed:
services:
disabledDefaultServices:
- ro
- r
maxSyncReplicas: 0
minSyncReplicas: 0
monitoring:
customQueriesConfigMap:
- key: queries
name: cnpg-default-monitoring
disableDefaultQueries: false
enablePodMonitor: false
plugins:
- enabled: true
isWALArchiver: true
name: barman-cloud.cloudnative-pg.io
parameters:
barmanObjectName: s3-acme-dns-backup
postgresGID: 26
postgresUID: 26
postgresql:
parameters:
archive_mode: "on"
archive_timeout: 5min
dynamic_shared_memory_type: posix
full_page_writes: "on"
hot_standby_feedback: "on"
log_destination: csvlog
log_directory: /controller/log
log_filename: postgres
log_rotation_age: "0"
log_rotation_size: "0"
log_truncate_on_rotation: "false"
logging_collector: "on"
max_parallel_workers: "32"
max_replication_slots: "32"
max_slot_wal_keep_size: 1GB
max_worker_processes: "32"
shared_memory_type: mmap
shared_preload_libraries: ""
ssl_max_protocol_version: TLSv1.3
ssl_min_protocol_version: TLSv1.3
wal_keep_size: 1GB
wal_level: logical
wal_log_hints: "on"
wal_receiver_timeout: 5s
wal_sender_timeout: 5s
syncReplicaElectionConstraint:
enabled: false
primaryUpdateMethod: restart
primaryUpdateStrategy: unsupervised
probes:
liveness:
isolationCheck:
connectionTimeout: 1000
enabled: true
requestTimeout: 1000
replicationSlots:
highAvailability:
enabled: true
slotPrefix: _cnpg_
synchronizeReplicas:
enabled: true
updateInterval: 30
resources:
limits:
memory: 150Mi
requests:
memory: 50Mi
smartShutdownTimeout: 180
startDelay: 3600
stopDelay: 1800
storage:
resizeInUseVolumes: true
size: 2Gi
storageClass: tulip-nvmeof
switchoverDelay: 3600Relevant log output
{"level":"info","ts":"2026-02-10T06:26:38.675641965Z","logger":"setup","msg":"Starting CloudNativePG Operator","version":"1.28.1","build":{"Version":"1.28.1","Commit":"1ecc48340","Date":"2026-02-05"}}
...
{"level":"info","ts":"2026-02-10T06:26:42.955437713Z","msg":"Cannot extract Pod status","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres","namespace":"cm-acme-dns"},"namespace":"cm-acme-dns","name":"postgres","reconcileID":"2e923631-d495-4188-84e6-f389616f654c","podName":"postgres-3","error":"error status code: 500, body: failed to connect to `user=postgres database=postgres`: /controller/run/.s.PGSQL.5432 (/controller/run): dial error: dial unix /controller/run/.s.PGSQL.5432: connect: no such file or directory\n"}Code of Conduct
- I agree to follow this project's Code of Conduct
Reacted by dosubot and nat
Activity
d-uzlov commentedon Feb 10, 2026
kubectl -n cm-acme-dns describe pod postgres-3 (click me)
dosubot commentedon Feb 10, 2026
The operator is supposed to delete pods annotated with
alpha.cnpg.io/unrecoverable=true(as long as they're not the current or target primary), but in practice, pods like yours can get stuck and not be deleted automatically. This happens when the operator can't extract the pod's status—often because the main Postgres container is failing its startup probe and restarting, while a sidecar (like barman-cloud) is still running. In this state, the operator logs repeated "Cannot extract Pod status" messages and doesn't proceed with deletion, even though the annotation is present and valid (operator code, user reports).This is a known lifecycle handling gap. The recommended workaround is to manually delete the stuck pod and its PVC. The operator will then recreate the pod and resync the replica as expected. This is a safe and common recovery step in current CNPG releases (docs, discussion). Future releases may improve this behavior, but for now, manual intervention is sometimes necessary.
To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
fix: allow unrecoverable annotation to work on non-ready pods
fix: allow unrecoverable annotation to work on non-ready pods
fix: allow unrecoverable annotation to work on non-ready pods (#9968)
fix: allow unrecoverable annotation to work on non-ready pods (#9968)