Recover an unhealthy/not ready cluster #9712

rizlas · 2026-01-14T08:56:03Z

rizlas
Jan 14, 2026

Hi everyone,

we’re running into a pretty frustrating situation with a CloudNativePG cluster and we’re hoping someone can point us in the right direction.

Here’s what happened.

Backups and WAL archiving were configured to S3. At some point, S3 became unreachable and the WAL disk filled up completely. As a result, the CNPG cluster entered an error state.

Following the documentation, we extended the disk. After that, the cluster came back up, WAL archiving started working again, and backups were running successfully. However, the cluster never returned to a healthy state.

At that point, we decided to try upgrading the cluster by enabling the barman-cloud.cloudnative-pg.io plugin. We deployed it and immediately started seeing communication errors toward Barman. Those errors were expected, since the cluster never created the dedicated container needed to run the plugin.

We then tried the usual reload and restart commands as described in the documentation, mainly to “poke” the operator and force a reconciliation, hoping it would restart the cluster and create the containers required by the plugin but nothing happened.

no output in the terminal,
no errors,
no logs.

It almost feels like once the cluster enters an unhealthy state, any action taken on it is simply ignored. Even looking at the operator logs (with log level set to trace), we couldn’t find any meaningful or specific error that could help us understand what’s wrong or how to fix it.

Out of desperation, we tried restarting a single instance. The result was even worse: the instance disappeared and never came back.

So at this point we’re basically stuck.

Our questions are pretty straightforward:

How can we recover this cluster and bring it back to a healthy state?
Where should we really be looking for errors or logs that can help with proper troubleshooting in situations like this?

An options is of course to recreate the entire cluster from a backup, but we want to learn how to fix this issue because this is something we are not encountering for the first time. Any help, guidance, or pointers would be very much appreciated.

Thanks!

We’re left with a cluster that looks like this:

    kubectl cnpg -n acme-proxy status acme-proxy
    Cluster Summary
    Name                 acme-proxy/acme-proxy
    System ID:           7514076314268319775
    PostgreSQL Image:    ghcr.io/cloudnative-pg/postgresql:16.4
    Primary instance:    acme-proxy-2
    Primary start time:  2025-10-15 14:13:32 +0000 UTC (uptime 2157h41m28s)
    Status:              Waiting for the instances to become active Some instances are not yet active. Please wait.
    Instances:           3
    Ready instances:     2
    Size:                648M
    Current Write LSN:   28/A400C040 (Timeline: 4 - WAL File: 0000000400000028000000A4)
     
    Continuous Backup status
    First Point of Recoverability:  2026-01-13T11:48:25Z
    Working WAL archiving:          OK
    WALs waiting to be archived:    0
    Last Archived WAL:              0000000400000028000000A3   @   2026-01-13T11:47:44.212205Z
    Last Failed WAL:                000000040000002800000099   @   2026-01-13T11:46:36.149775Z
     
    Streaming Replication status
    Replication Slots Enabled
    Name          Sent LSN     Write LSN    Flush LSN    Replay LSN   Write Lag  Flush Lag  Replay Lag  State      Sync State  Sync Priority  Replication Slot
    ----          --------     ---------    ---------    ----------   ---------  ---------  ----------  -----      ----------  -------------  ----------------
    acme-proxy-3  28/A400C040  28/A400C040  28/A400C040  28/A400C040  00:00:00   00:00:00   00:00:00    streaming  async       0              active
     
    Instances status
    Name          Current LSN  Replication role  Status  QoS         Manager Version  Node
    ----          -----------  ----------------  ------  ---         ---------------  ----
    acme-proxy-2  28/A400C040  Primary           OK      BestEffort  1.27.0           k8s-ms-13
    acme-proxy-3  28/A400C040  Standby (async)   OK      BestEffort  1.27.0           k8s-ms-04

    status:
      availableArchitectures:
      - goArch: amd64
        hash: 433bae32a2e3e459a2a181d67ed2b0dfcf67f359480b9646bbdc8ad8e2dd4ef1
      - goArch: arm64
        hash: 981f8452ecf423a37802c14018b7a73bfd25c1c2e82ec19de9c85cd54912adb7
      certificates:
        clientCASecret: acme-proxy-ca
        expirations:
          acme-proxy-ca: 2026-02-20 23:55:17 +0000 UTC
          acme-proxy-replication: 2026-02-20 23:55:17 +0000 UTC
          acme-proxy-server: 2026-02-20 23:55:17 +0000 UTC
        replicationTLSSecret: acme-proxy-replication
        serverAltDNSNames:
        - acme-proxy-rw
        - acme-proxy-rw.acme-proxy
        - acme-proxy-rw.acme-proxy.svc
        - acme-proxy-rw.acme-proxy.svc.cluster.local
        - acme-proxy-r
        - acme-proxy-r.acme-proxy
        - acme-proxy-r.acme-proxy.svc
        - acme-proxy-r.acme-proxy.svc.cluster.local
        - acme-proxy-ro
        - acme-proxy-ro.acme-proxy
        - acme-proxy-ro.acme-proxy.svc
        - acme-proxy-ro.acme-proxy.svc.cluster.local
        serverCASecret: acme-proxy-ca
        serverTLSSecret: acme-proxy-server
      cloudNativePGCommitHash: 8b442dcc3
      cloudNativePGOperatorHash: 433bae32a2e3e459a2a181d67ed2b0dfcf67f359480b9646bbdc8ad8e2dd4ef1
      conditions:
      - lastTransitionTime: "2026-01-13T05:07:23Z"
        message: Cluster Is Not Ready
        reason: ClusterIsNotReady
        status: "False"
        type: Ready
      - lastTransitionTime: "2026-01-13T11:47:36Z"
        message: Continuous archiving is working
        reason: ContinuousArchivingSuccess
        status: "True"
        type: ContinuousArchiving
      - lastTransitionTime: "2026-01-13T11:48:28Z"
        message: Backup was successful
        reason: LastBackupSucceeded
        status: "True"
        type: LastBackupSucceeded
      - lastTransitionTime: "2025-09-02T09:50:21Z"
        message: A single, unique system ID was found across reporting instances.
        reason: Unique
        status: "True"
        type: ConsistentSystemID
      configMapResourceVersion:
        metrics:
          cnpg-default-monitoring: "485478649"
      currentPrimary: acme-proxy-2
      currentPrimaryTimestamp: "2025-10-15T14:13:31.723070Z"
      firstRecoverabilityPoint: "2026-01-13T11:48:25Z"
      firstRecoverabilityPointByMethod:
        barmanObjectStore: "2026-01-13T11:48:25Z"
      image: ghcr.io/cloudnative-pg/postgresql:16.4
      instanceNames:
      - acme-proxy-1
      - acme-proxy-2
      - acme-proxy-3
      instances: 3
      instancesReportedState:
        acme-proxy-2:
          ip: 10.42.13.206
          isPrimary: true
          timeLineID: 4
        acme-proxy-3:
          ip: 10.42.15.19
          isPrimary: false
          timeLineID: 4
      instancesStatus:
        healthy:
        - acme-proxy-2
        - acme-proxy-3
      lastFailedBackup: "2026-01-13T11:48:15Z"
      lastSuccessfulBackup: "2026-01-13T11:48:25Z"
      lastSuccessfulBackupByMethod:
        barmanObjectStore: "2026-01-13T11:48:25Z"
      latestGeneratedNode: 3
      managedRolesStatus: {}
      pgDataImageInfo:
        image: ghcr.io/cloudnative-pg/postgresql:16.4
        majorVersion: 16
      phase: Waiting for the instances to become active
      phaseReason: Some instances are not yet active. Please wait.
      poolerIntegrations:
        pgBouncerIntegration: {}
      pvcCount: 3
      readService: acme-proxy-r
      readyInstances: 2
      secretsResourceVersion:
        applicationSecretVersion: "402015898"
        clientCaSecretVersion: "569340715"
        replicationSecretVersion: "569340717"
        serverCaSecretVersion: "569340715"
        serverSecretVersion: "569340716"
        superuserSecretVersion: "485479043"
      switchReplicaClusterStatus: {}
      systemID: "7514076314268319775"
      targetPrimary: acme-proxy-2
      targetPrimaryTimestamp: "2025-10-15T14:13:27.176667Z"
      timelineID: 4
      topology:
        instances:
          acme-proxy-2: {}
          acme-proxy-3: {}
        nodesUsed: 2
        successfullyExtracted: true
      unusablePVC:
      - acme-proxy-1
      - acme-proxy-2
      - acme-proxy-3
      writeService: acme-proxy-rw

@rizlas · 2026-01-14T09:02:46Z

dosubot[bot]
Bot Jan 14, 2026

Hi @rizlas! I'm Dosu and I’m helping the cloudnative-pg team.

Your cluster is stuck because all three PVCs are marked as "unusable," which means the operator sees incomplete or broken storage for each instance and cannot proceed with reconciliation or pod creation. This often happens after WAL disk exhaustion, S3 outages, or failed restarts, and simply expanding the disk is not always enough to recover the cluster [PVC status logic, troubleshooting docs].

What you can do:

Check pod and PVC status:
Run kubectl get pods -n acme-proxy and kubectl get pvc -n acme-proxy to see which pods and PVCs are missing, stuck, or in error. For the missing instance (acme-proxy-1), use kubectl describe pod acme-proxy-1 -n acme-proxy and kubectl describe pvc acme-proxy-1 -n acme-proxy to look for events or errors.
Review logs:
Use kubectl logs -n cnpg-system deployment/cnpg-controller-manager --all-containers=true for operator logs, and kubectl logs -n acme-proxy acme-proxy-2 (or -3) for PostgreSQL pod logs. Filter for errors with jq as described in the docs.
Take an emergency logical backup:
Before any disruptive action, run kubectl exec acme-proxy-2 -c postgres -- pg_dump -Fc -d <your_db> > backup.dump to safeguard your data [emergency backup].
Recover stuck replicas:
If a replica pod/PVC is unusable or missing, delete the PVC and pod for that instance (e.g., acme-proxy-1) with:
```
kubectl delete pod acme-proxy-1 -n acme-proxy
kubectl delete pvc acme-proxy-1 -n acme-proxy
kubectl delete pvc acme-proxy-1-wal -n acme-proxy
```
The operator should recreate the pod and PVC, resyncing the replica from the primary. Do not delete the PVC for the primary unless you are prepared to restore from backup. [replica recovery, community advice]
Plugin/container issues:
In v1.27, if plugin containers (like barman-cloud) are OOMKilled or fail, the plugin connection pool can get stuck in a "closed" state, blocking reconciliation. There is no user-facing fix for this in 1.27; deleting stuck pods may help, but if the plugin remains unhealthy, the cluster will stay blocked [plugin bug].
If recovery fails:
If the cluster remains stuck after PVC/pod deletion, or if PVCs for the primary are missing, the only safe recovery is to restore from a backup [unrecoverable state].

Where to look for errors:

Operator logs (kubectl logs -n cnpg-system ...)
Pod logs (kubectl logs -n acme-proxy ...)
Pod/PVC events (kubectl describe pod ..., kubectl describe pvc ...)
Cluster status (kubectl cnpg status ..., kubectl cnpg report --logs ...)

References:

Let me know if you need more specific commands or help interpreting your pod/PVC status!

To reply, just mention @dosu.

Docs are dead. Just use Dosu.

0 replies

rizlas · 2026-01-14T09:10:22Z

rizlas
Jan 14, 2026
Author

Just and addition for you (and the AI). We have another cluster that encounter the same problem. It should be replica 3 but currently only two pod are running. Same as before, every command issued is useless.

Current status:

status:
  availableArchitectures:
  - goArch: amd64
    hash: 433bae32a2e3e459a2a181d67ed2b0dfcf67f359480b9646bbdc8ad8e2dd4ef1
  - goArch: arm64
    hash: 981f8452ecf423a37802c14018b7a73bfd25c1c2e82ec19de9c85cd54912adb7
  certificates:
    clientCASecret: cnpg-keycloak-prod-ca
    expirations:
      cnpg-keycloak-prod-ca: 2026-03-16 00:04:11 +0000 UTC
      cnpg-keycloak-prod-replication: 2026-03-16 00:04:11 +0000 UTC
      cnpg-keycloak-prod-server: 2026-03-16 00:04:11 +0000 UTC
    replicationTLSSecret: cnpg-keycloak-prod-replication
    serverAltDNSNames:
    - cnpg-keycloak-prod-rw
    - cnpg-keycloak-prod-rw.satosa
    - cnpg-keycloak-prod-rw.satosa.svc
    - cnpg-keycloak-prod-rw.satosa.svc.cluster.local
    - cnpg-keycloak-prod-r
    - cnpg-keycloak-prod-r.satosa
    - cnpg-keycloak-prod-r.satosa.svc
    - cnpg-keycloak-prod-r.satosa.svc.cluster.local
    - cnpg-keycloak-prod-ro
    - cnpg-keycloak-prod-ro.satosa
    - cnpg-keycloak-prod-ro.satosa.svc
    - cnpg-keycloak-prod-ro.satosa.svc.cluster.local
    serverCASecret: cnpg-keycloak-prod-ca
    serverTLSSecret: cnpg-keycloak-prod-server
  cloudNativePGCommitHash: 8b442dcc3
  cloudNativePGOperatorHash: 433bae32a2e3e459a2a181d67ed2b0dfcf67f359480b9646bbdc8ad8e2dd4ef1
  conditions:
  - lastTransitionTime: "2025-09-23T14:36:48Z"
    message: A single, unique system ID was found across reporting instances.
    reason: Unique
    status: "True"
    type: ConsistentSystemID
  - lastTransitionTime: "2026-01-13T09:51:56Z"
    message: Cluster Is Not Ready
    reason: ClusterIsNotReady
    status: "False"
    type: Ready
  - lastTransitionTime: "2026-01-13T11:15:02Z"
    message: Continuous archiving is working
    reason: ContinuousArchivingSuccess
    status: "True"
    type: ContinuousArchiving
  - lastTransitionTime: "2026-01-14T00:00:00Z"
    message: cannot proceed with the backup as the cluster has no plugin configured
    reason: LastBackupFailed
    status: "False"
    type: LastBackupSucceeded
  configMapResourceVersion:
    metrics:
      cnpg-default-monitoring: "506191379"
  currentPrimary: cnpg-keycloak-prod-1
  currentPrimaryTimestamp: "2025-10-15T14:14:28.992066Z"
  firstRecoverabilityPoint: "2026-01-13T11:15:33Z"
  firstRecoverabilityPointByMethod:
    barmanObjectStore: "2026-01-13T11:15:33Z"
  image: ghcr.io/cloudnative-pg/postgresql:17
  instancesReportedState:
    cnpg-keycloak-prod-1:
      ip: 10.42.13.133
      isPrimary: true
      timeLineID: 3
    cnpg-keycloak-prod-3:
      ip: 10.42.7.73
      isPrimary: false
      timeLineID: 3
  instancesStatus:
    healthy:
    - cnpg-keycloak-prod-1
    - cnpg-keycloak-prod-3
  lastFailedBackup: "2026-01-14T00:00:00Z"
  lastSuccessfulBackup: "2026-01-13T11:15:33Z"
  lastSuccessfulBackupByMethod:
    barmanObjectStore: "2026-01-13T11:15:33Z"
  latestGeneratedNode: 3
  managedRolesStatus: {}
  pgDataImageInfo:
    image: ghcr.io/cloudnative-pg/postgresql:17
    majorVersion: 17
  phase: Waiting for the instances to become active
  phaseReason: Some instances are not yet active. Please wait.
  poolerIntegrations:
    pgBouncerIntegration: {}
  readService: cnpg-keycloak-prod-r
  readyInstances: 2
  secretsResourceVersion:
    applicationSecretVersion: "494044426"
    clientCaSecretVersion: "594320769"
    replicationSecretVersion: "594320771"
    serverCaSecretVersion: "594320769"
    serverSecretVersion: "594320770"
    superuserSecretVersion: "494044463"
  switchReplicaClusterStatus: {}
  systemID: "7553293546182463515"
  targetPrimary: cnpg-keycloak-prod-1
  targetPrimaryTimestamp: "2025-10-15T14:14:25.347627Z"
  timelineID: 3
  topology:
    instances:
      cnpg-keycloak-prod-1: {}
      cnpg-keycloak-prod-3: {}
    nodesUsed: 2
    successfullyExtracted: true
  writeService: cnpg-keycloak-prod-rw

1 reply

dosubot[bot] Bot Jan 14, 2026

This cluster is showing the same stuck pattern as your previous one: it's in "Waiting for the instances to become active" with only two pods running, and the "Ready" condition is false. This usually means the operator is blocked by a missing or unusable pod/PVC, a plugin misconfiguration, or a network issue between the operator and pods (docs).

Here’s what to check and try next:

Run kubectl get pods -n <namespace> and kubectl get pvc -n <namespace> to see which pod (likely cnpg-keycloak-prod-2) or PVC is missing, stuck, or in an error state.
Use kubectl describe pod <missing-pod> -n <namespace> and kubectl describe pvc <missing-pvc> -n <namespace> to look for scheduling errors, events, or PVC status.
Check operator logs (kubectl logs -n cnpg-system deployment/cnpg-controller-manager --all-containers=true) for plugin errors, reconciliation blocks, or network timeouts.
If the missing pod is a replica and its PVC is stuck or unusable, delete both the pod and its PVC (and the WAL PVC if present). The operator should recreate them and resync from the primary (PVC logic).
If you’re using plugins (like barman-cloud), make sure all referenced ObjectStore resources exist and are correctly configured. A misconfigured or missing plugin resource can block reconciliation and leave the cluster stuck (plugin issue).
If the cluster remains stuck after these steps, or if the missing PVC is for the primary, restoring from backup may be the only safe option.

If you can, share the output of the pod and PVC lists and any relevant events or errors. That will help pinpoint whether this is a PVC, plugin, or config issue. Let me know what you find!

To reply, just mention @dosu.

Share context across your team and agents. Try Dosu.

Navigation Menu

Repository navigation

Recover an unhealthy/not ready cluster #9712

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Recover an unhealthy/not ready cluster #9712

Uh oh!

rizlas Jan 14, 2026

Replies: 2 comments · 1 reply

Uh oh!

Uh oh!

dosubot[bot] Bot Jan 14, 2026

Uh oh!

rizlas Jan 14, 2026 Author

Uh oh!

Uh oh!

dosubot[bot] Bot Jan 14, 2026

rizlas
Jan 14, 2026

Replies: 2 comments 1 reply

dosubot[bot]
Bot Jan 14, 2026

rizlas
Jan 14, 2026
Author