[Bug]: cluster stuck "Waiting for the instances to become active" after restarting node

Is there an existing issue already for this bug?

I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

I have read the troubleshooting guide and I think this is a new bug.

Version

1.23.2

What version of Kubernetes are you using?

1.30 (unsupported)

What is your Kubernetes environment?

Cloud: Amazon EKS

How did you install the operator?

Helm

What happened?

A node became "Unknown".
I restarted it manually.

I expected all the clusters to become healthy

It actually seems to be stuck, unable to reach a consistent recovery state.

Extra Details

kubectl cnpg status -v

Cluster Summary Name: postgres-advanced Namespace: advanced System ID: 7381994731694346259 PostgreSQL Image: ghcr.io/cloudnative-pg/postgis:16 Primary instance: postgres-advanced-1 Primary start time: 2024-07-11 18:56:32 +0000 UTC (uptime 2h10m42s) Status: Waiting for the instances to become active Some instances are not yet active. Please wait. Instances: 3 Ready instances: 2 Current Write LSN: 0/B4000000 (Timeline: 6 - WAL File: 0000000600000000000000B3)

PostgreSQL Configuration
archive_command = '/controller/manager wal-archive --log-destination /controller/log/postgres.json %p'
archive_mode = 'on'
archive_timeout = '5min'
cluster_name = 'postgres-advanced'
dynamic_shared_memory_type = 'posix'
full_page_writes = 'on'
hot_standby = 'true'
listen_addresses = '*'
log_destination = 'csvlog'
log_directory = '/controller/log'
log_filename = 'postgres'
log_rotation_age = '0'
log_rotation_size = '0'
log_truncate_on_rotation = 'false'
logging_collector = 'on'
max_parallel_workers = '32'
max_replication_slots = '32'
max_worker_processes = '32'
port = '5432'
restart_after_crash = 'false'
shared_memory_type = 'mmap'
shared_preload_libraries = ''
ssl = 'on'
ssl_ca_file = '/controller/certificates/client-ca.crt'
ssl_cert_file = '/controller/certificates/server.crt'
ssl_key_file = '/controller/certificates/server.key'
ssl_max_protocol_version = 'TLSv1.3'
ssl_min_protocol_version = 'TLSv1.3'
unix_socket_directories = '/controller/run'
wal_keep_size = '512MB'
wal_level = 'logical'
wal_log_hints = 'on'
wal_receiver_timeout = '5s'
wal_sender_timeout = '5s'
cnpg.config_sha256 = 'e9bd29d8d34e5c4c89b51b9e1a9df37b5ea1081dbf2516291eeb922354e53d80'

PostgreSQL HBA Rules

FIXED RULES

Grant local access ('local' user map)

local all all peer map=local

Require client certificate authentication for the streaming_replica user

hostssl postgres streaming_replica all cert
hostssl replication streaming_replica all cert
hostssl all cnpg_pooler_pgbouncer all cert

USER-DEFINED RULES

DEFAULT RULES

host all all all scram-sha-256

Certificates Status
Certificate Name Expiration Date Days Left Until Expiration

postgres-advanced-ca 2024-09-28 23:44:39 +0000 UTC 79.11
postgres-advanced-replication 2024-09-28 23:44:39 +0000 UTC 79.11
postgres-advanced-server 2024-09-28 23:44:39 +0000 UTC 79.11

Continuous Backup status
First Point of Recoverability: 2024-07-01T17:18:44Z
Working WAL archiving: OK
WALs waiting to be archived: 0
Last Archived WAL: 0000000600000000000000B3 @ 2024-07-11T19:36:34.102217Z
Last Failed WAL: 00000006.history @ 2024-07-11T18:56:32.996025Z

Physical backups
No running physical backups found

Streaming Replication status
Replication Slots Enabled
Name Sent LSN Write LSN Flush LSN Replay LSN Write Lag Flush Lag Replay Lag State Sync State Sync Priority Replication Slot Slot Restart LSN Slot WAL Status Slot Safe WAL Size

postgres-advanced-2 0/B4000000 0/B4000000 0/B4000000 0/B4000000 00:00:00 00:00:00 00:00:00 streaming async 0 active 0/B4000000 reserved NULL
postgres-advanced-3 0/B4000000 0/B4000000 0/B4000000 0/B4000000 00:00:00 00:00:00 00:00:00 streaming async 0 active 0/B4000000 reserved NULL

Unmanaged Replication Slot Status
No unmanaged replication slots found

Managed roles status
No roles managed

Tablespaces status
No managed tablespaces

Pod Disruption Budgets status
Name Role Expected Pods Current Healthy Minimum Desired Healthy Disruptions Allowed

postgres-advanced replica 2 1 1 0
postgres-advanced-primary primary 1 1 1 0

Instances status
Name Database Size Current LSN Replication role Status QoS Manager Version Node

postgres-advanced-1 51 MB 0/B4000000 Primary OK BestEffort 1.23.2 ip-10-0-39-143.us-west-2.compute.internal
postgres-advanced-2 50 MB 0/B4000000 Standby (async) OK BestEffort 1.23.2 ip-10-0-18-108.us-west-2.compute.internal
postgres-advanced-3 - - - pod not available BestEffort - ip-10-0-5-110.us-west-2.compute.internal

Helm chart version
0.21.5

Extra notes:
I have 5 instances of my app. Each application has two clusters (each with 3 replicas). One of the two clusters is not backed up.

Out of the 10 clusters:
1 of 5 of the clusters with backups are fully recovered (all similar to the errors reported below)
5 of 5 of the clusters without backups are fully recovered.

Cluster resource

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"postgresql.cnpg.io/v1","kind":"Cluster","metadata":{"annotations":{"argocd.argoproj.io/sync-options":"SkipDryRunOnMissingResource=true"},"labels":{"app.kubernetes.io/instance":"utilityai"},"name":"postgres-advanced","namespace":"advanced"},"spec":{"backup":{"barmanObjectStore":{"destinationPath":"s3://advanced-db-backup/v2","s3Credentials":{"inheritFromIAMRole":true},"wal":{"compression":"gzip","maxParallel":8}}},"bootstrap":{"recovery":{"database":"advanced-app","source":"clusterBackup"}},"externalClusters":[{"barmanObjectStore":{"destinationPath":"s3://advanced-db-backup/v1/","s3Credentials":{"inheritFromIAMRole":true},"serverName":"postgres-advanced","wal":{"compression":"gzip","maxParallel":8}},"name":"clusterBackup"}],"imageName":"ghcr.io/cloudnative-pg/postgis:16","instances":3,"serviceAccountTemplate":{"metadata":{"annotations":{"eks.amazonaws.com/role-arn":"arn:aws:iam::xxxxxxxx:role/eks-postgres-advanced"}}},"storage":{"size":"10Gi"}}}
  creationTimestamp: "2024-06-30T23:49:39Z"
  generation: 5
  labels:
    app.kubernetes.io/instance: utilityai
  name: postgres-advanced
  namespace: advanced
  resourceVersion: "63797733"
  uid: 6e546644-19bf-471c-b80b-18997e12b3b5
spec:
  affinity:
    podAntiAffinityType: preferred
  backup:
    barmanObjectStore:
      destinationPath: s3://advanced-db-backup/v2
      s3Credentials:
        inheritFromIAMRole: true
      wal:
        compression: gzip
        maxParallel: 8
    target: prefer-standby
  bootstrap:
    recovery:
      database: advanced-app
      owner: advanced-app
      source: clusterBackup
  enablePDB: true
  enableSuperuserAccess: false
  externalClusters:
  - barmanObjectStore:
      destinationPath: s3://advanced-db-backup/v1/
      s3Credentials:
        inheritFromIAMRole: true
      serverName: postgres-advanced
      wal:
        compression: gzip
        maxParallel: 8
    name: clusterBackup
  failoverDelay: 0
  imageName: ghcr.io/cloudnative-pg/postgis:16
  instances: 3
  logLevel: info
  maxSyncReplicas: 0
  minSyncReplicas: 0
  monitoring:
    customQueriesConfigMap:
    - key: queries
      name: cnpg-default-monitoring
    disableDefaultQueries: false
    enablePodMonitor: false
  postgresGID: 26
  postgresUID: 26
  postgresql:
    parameters:
      archive_mode: "on"
      archive_timeout: 5min
      dynamic_shared_memory_type: posix
      log_destination: csvlog
      log_directory: /controller/log
      log_filename: postgres
      log_rotation_age: "0"
      log_rotation_size: "0"
      log_truncate_on_rotation: "false"
      logging_collector: "on"
      max_parallel_workers: "32"
      max_replication_slots: "32"
      max_worker_processes: "32"
      shared_memory_type: mmap
      shared_preload_libraries: ""
      ssl_max_protocol_version: TLSv1.3
      ssl_min_protocol_version: TLSv1.3
      wal_keep_size: 512MB
      wal_level: logical
      wal_log_hints: "on"
      wal_receiver_timeout: 5s
      wal_sender_timeout: 5s
    syncReplicaElectionConstraint:
      enabled: false
  primaryUpdateMethod: restart
  primaryUpdateStrategy: unsupervised
  replicationSlots:
    highAvailability:
      enabled: true
      slotPrefix: _cnpg_
    synchronizeReplicas:
      enabled: true
    updateInterval: 30
  resources: {}
  serviceAccountTemplate:
    metadata:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxx:role/eks-postgres-advanced
  smartShutdownTimeout: 180
  startDelay: 3600
  stopDelay: 1800
  storage:
    resizeInUseVolumes: true
    size: 10Gi
  switchoverDelay: 3600
status:
  availableArchitectures:
  - goArch: amd64
    hash: e43340cd2ccfa2a8120cf5de6035fe4b18d799bb2feabf99a91434dd9ba92e4c
  - goArch: arm64
    hash: f453a8cb50a418ff9cac24c818b9e155b80487b4152444e48d187161ecdfc0eb
  certificates:
    clientCASecret: postgres-advanced-ca
    expirations:
      postgres-advanced-ca: 2024-09-28 23:44:39 +0000 UTC
      postgres-advanced-replication: 2024-09-28 23:44:39 +0000 UTC
      postgres-advanced-server: 2024-09-28 23:44:39 +0000 UTC
    replicationTLSSecret: postgres-advanced-replication
    serverAltDNSNames:
    - postgres-advanced-rw
    - postgres-advanced-rw.advanced
    - postgres-advanced-rw.advanced.svc
    - postgres-advanced-r
    - postgres-advanced-r.advanced
    - postgres-advanced-r.advanced.svc
    - postgres-advanced-ro
    - postgres-advanced-ro.advanced
    - postgres-advanced-ro.advanced.svc
    serverCASecret: postgres-advanced-ca
    serverTLSSecret: postgres-advanced-server
  cloudNativePGCommitHash: 4bef8412
  cloudNativePGOperatorHash: e43340cd2ccfa2a8120cf5de6035fe4b18d799bb2feabf99a91434dd9ba92e4c
  conditions:
  - lastTransitionTime: "2024-07-11T18:55:06Z"
    message: Cluster Is Not Ready
    reason: ClusterIsNotReady
    status: "False"
    type: Ready
  - lastTransitionTime: "2024-07-11T18:56:38Z"
    message: Continuous archiving is working
    reason: ContinuousArchivingSuccess
    status: "True"
    type: ContinuousArchiving
  - lastTransitionTime: "2024-07-11T00:00:13Z"
    message: Backup was successful
    reason: LastBackupSucceeded
    status: "True"
    type: LastBackupSucceeded
  configMapResourceVersion:
    metrics:
      cnpg-default-monitoring: "59241826"
  currentPrimary: postgres-advanced-1
  currentPrimaryTimestamp: "2024-07-11T18:56:31.977458Z"
  firstRecoverabilityPoint: "2024-07-01T17:18:44Z"
  firstRecoverabilityPointByMethod:
    barmanObjectStore: "2024-07-01T17:18:44Z"
  healthyPVC:
  - postgres-advanced-1
  - postgres-advanced-2
  - postgres-advanced-3
  image: ghcr.io/cloudnative-pg/postgis:16
  instanceNames:
  - postgres-advanced-1
  - postgres-advanced-2
  - postgres-advanced-3
  instances: 3
  instancesReportedState:
    postgres-advanced-1:
      isPrimary: true
      timeLineID: 6
    postgres-advanced-2:
      isPrimary: false
      timeLineID: 6
    postgres-advanced-3:
      isPrimary: false
  instancesStatus:
    healthy:
    - postgres-advanced-1
    - postgres-advanced-2
    replicating:
    - postgres-advanced-3
  lastFailedBackup: "2024-07-01T17:11:30Z"
  lastSuccessfulBackup: "2024-07-11T00:00:10Z"
  lastSuccessfulBackupByMethod:
    barmanObjectStore: "2024-07-11T00:00:10Z"
  latestGeneratedNode: 3
  managedRolesStatus: {}
  phase: Waiting for the instances to become active
  phaseReason: Some instances are not yet active. Please wait.
  poolerIntegrations:
    pgBouncerIntegration: {}
  pvcCount: 3
  readService: postgres-advanced-r
  readyInstances: 2
  secretsResourceVersion:
    applicationSecretVersion: "59244180"
    clientCaSecretVersion: "59244177"
    replicationSecretVersion: "59244179"
    serverCaSecretVersion: "59244177"
    serverSecretVersion: "59244178"
  switchReplicaClusterStatus: {}
  targetPrimary: postgres-advanced-1
  targetPrimaryTimestamp: "2024-07-11T18:55:06.821159Z"
  timelineID: 6
  topology:
    instances:
      postgres-advanced-1: {}
      postgres-advanced-2: {}
      postgres-advanced-3: {}
    nodesUsed: 3
    successfullyExtracted: true
  writeService: postgres-advanced-rw

Relevant log output

(this repeats on and on.)

{
  "level": "info",
  "ts": "2024-07-11T20:52:12Z",
  "logger": "postgres",
  "msg": "record",
  "logging_pod": "postgres-advanced-3",
  "record": {
    "log_time": "2024-07-11 20:52:12.348 UTC",
    "user_name": "postgres",
    "database_name": "postgres",
    "process_id": "283",
    "connection_from": "[local]",
    "session_id": "669045fc.11b",
    "session_line_num": "1",
    "session_start_time": "2024-07-11 20:52:12 UTC",
    "transaction_id": "0",
    "error_severity": "FATAL",
    "sql_state_code": "57P03",
    "message": "the database system is not yet accepting connections",
    "detail": "Consistent recovery state has not been yet reached.",
    "backend_type": "client backend",
    "query_id": "0"
  }
}
{
  "level": "info",
  "ts": "2024-07-11T20:52:13Z",
  "logger": "postgres",
  "msg": "record",
  "logging_pod": "postgres-advanced-3",
  "record": {
    "log_time": "2024-07-11 20:52:13.406 UTC",
    "user_name": "postgres",
    "database_name": "postgres",
    "process_id": "285",
    "connection_from": "[local]",
    "session_id": "669045fd.11d",
    "session_line_num": "1",
    "session_start_time": "2024-07-11 20:52:13 UTC",
    "transaction_id": "0",
    "error_severity": "FATAL",
    "sql_state_code": "57P03",
    "message": "the database system is not yet accepting connections",
    "detail": "Consistent recovery state has not been yet reached.",
    "backend_type": "client backend",
    "query_id": "0"
  }
}
{
  "level": "info",
  "ts": "2024-07-11T20:52:13Z",
  "logger": "postgres",
  "msg": "record",
  "logging_pod": "postgres-advanced-3",
  "record": {
    "log_time": "2024-07-11 20:52:13.408 UTC",
    "user_name": "postgres",
    "database_name": "postgres",
    "process_id": "286",
    "connection_from": "[local]",
    "session_id": "669045fd.11e",
    "session_line_num": "1",
    "session_start_time": "2024-07-11 20:52:13 UTC",
    "transaction_id": "0",
    "error_severity": "FATAL",
    "sql_state_code": "57P03",
    "message": "the database system is not yet accepting connections",
    "detail": "Consistent recovery state has not been yet reached.",
    "backend_type": "client backend",
    "query_id": "0"
  }
}
{
  "level": "info",
  "ts": "2024-07-11T20:52:13Z",
  "msg": "DB not available, will retry",
  "controller": "cluster",
  "controllerGroup": "postgresql.cnpg.io",
  "controllerKind": "Cluster",
  "Cluster": {
    "name": "postgres-advanced",
    "namespace": "advanced"
  },
  "namespace": "advanced",
  "name": "postgres-advanced",
  "reconcileID": "2b9f82bb-cb3b-4e11-b4fa-e059b4b98f8f",
  "logging_pod": "postgres-advanced-3",
  "err": "failed to connect to `user=postgres database=postgres`: /controller/run/.s.PGSQL.5432 (/controller/run): server error: FATAL: the database system is not yet accepting connections (SQLSTATE 57P03)"
}

Code of Conduct

I agree to follow this project's Code of Conduct

since posting, one of the four has become healthy (after 2+ hours). Will close unless the other three clusters continue to remain unhealthy.

still unhealthy after 19+ hours. I've tried restarting pod, no luck.

the conclusion is to be patient. everything is working after 4 days(!).

I got the same problem with a very small testing db <1MB.
It got stuck this way. is there any log indicating progress?

My database was also very small (~10MB), I do not recall any logs indicating progress until I came back after the weekend and all was well.

I just removed the PVC + pod instead of waiting :(

I faced with the same issue on empty database. The root cause was there was no new changes on primary. Cluster got healthy state when I created one change in database on primary.

@oneumyvakin

ty @oneumyvakin - same issue for me running an emtpy pg, cluster switched to healthy state after inserting some data into it

same here i just created empty database and the cluster healed. does anyone have explanation why this fixed the issue? i hate to put "magic" in our internal documentation

I was doing an operator upgrade to v1.26 and also ran into this. Mine is a sizeable cluster(2TB) with 3 replicas. Waited an hr for the replicas to be active with no luck then tried inserting dummy data. 10 mins later replicas are active.

Had the same issue. I would vote to re-open this, because it's quite confusing for the users.

Could this be a upstream issue?

@gbartolini

Same problem here with a 2-instance cluster running CNPG 1.27.0. After disk saturation on postgres-1, I manually resized the PVC from 50Gi to 80Gi, then postgres-2 was correctly promoted to primary. However, postgres-1 never reappears automatically.

Steps attempted to resolve:

Manual recreation of postgres-1 pod (successful but operator doesn't recognize it)
Manual deletion of postgres-1 pod and its PVC
Operator restart: kubectl rollout restart deployment cnpg-cloudnative-pg
Forced reconciliation with timestamp annotation
Fixed latestGeneratedNode in cluster status
Verified node resources and anti-affinity constraints

Current state:

Cluster spec: instances: 2
Cluster status: readyInstances: 1, phase: "Waiting for the instances to become active"
Only postgres-2 is running as primary
No events or logs indicating scheduling attempts for postgres-1
PVC postgres-1 was properly deleted and should be recreated

Environment:

CNPG version: 1.27.0
PostgreSQL version: 15

The operator seems to be stuck and won't attempt to create the missing replica, despite all reconciliation triggers. @gbartolini any advice on how to force the operator to recreate the missing instance?

Thx.

@AntoineMicheaOpencell

I’m facing the same issue as @AntoineMicheaOpencell. Did you found a solution? My PVC reached full capacity, which caused the database to go down. A developer manually resized the PVC, but now the cluster is stuck on “Waiting for the instances to become active.”

@AntonioMaccarini

Hi @AntonioMaccarini, I bootstrapped a new cluster from a snapshot because I tried a lot of things without success, but I suspect there's a problem with the new version of CNPG. I've been using it since the beginning and I've never faced a problem like this without any solution.

We are also facing this issue with cnpg 1.27.2 and postgres v18. First instance is working fine second one is stuck

Navigation Menu

Repository navigation

[Bug]: cluster stuck "Waiting for the instances to become active" after restarting node #5078

Is there an existing issue already for this bug?

I have read the troubleshooting guide

I am running a supported version of CloudNativePG

Version

What version of Kubernetes are you using?

What is your Kubernetes environment?

How did you install the operator?

What happened?

Extra Details

FIXED RULES

Grant local access ('local' user map)

Require client certificate authentication for the streaming_replica user

USER-DEFINED RULES

DEFAULT RULES

Cluster resource

Relevant log output

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: cluster stuck "Waiting for the instances to become active" after restarting node #5078

Description

Is there an existing issue already for this bug?

I have read the troubleshooting guide

I am running a supported version of CloudNativePG

Version

What version of Kubernetes are you using?

What is your Kubernetes environment?

How did you install the operator?

What happened?

Extra Details

FIXED RULES

Grant local access ('local' user map)

Require client certificate authentication for the streaming_replica user

USER-DEFINED RULES

DEFAULT RULES

Cluster resource

Relevant log output

Code of Conduct

Activity

MarcusDunn commented on Jul 11, 2024

MarcusDunn commented on Jul 12, 2024

MarcusDunn commented on Jul 15, 2024

smulikHakipod commented on Aug 31, 2024

MarcusDunn commented on Aug 31, 2024

smulikHakipod commented on Sep 1, 2024

oneumyvakin commented on Oct 11, 2024

maxgruber19 commented on Feb 13, 2025

henro001 commented on Apr 23, 2025

MuriukiPM commented on Jun 5, 2025

RubenNL commented on Aug 14, 2025

AntoineMicheaOpencell commented on Aug 24, 2025

AntonioMaccarini commented on Aug 29, 2025

AntoineMicheaOpencell commented on Aug 29, 2025

Mahagon commented on Jan 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions