[Bug]: First-primary bootstrap can deadlock if a status update conflicts before the initdb Job is created #11036

Is there an existing issue already for this bug?

• I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

• I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

• I have read the troubleshooting guide and I think this is a new bug.

Contact Details

[email protected]

Version

trunk (main)

What version of Kubernetes are you using?

1.34

What is your Kubernetes environment?

Cloud: Google GKE

How did you install the operator?

YAML manifest

What happened?

A Cluster can get permanently stuck while bootstrapping its first primary, ending up with no pods, no jobs, an empty phase, and a data PVC that never progresses. The operator logs Selected PVC is not ready yet, waiting for 1 second on every reconcile and never recovers on its own.

Expected: a transient API conflict during bootstrap is retried, the initdb Job gets created, and the cluster comes up. A single failed status update should not be able to wedge the cluster permanently.

What happens instead: in the first reconcile that bootstraps the primary, the operator creates the data PVC and then updates the cluster status to register the FirstPrimary phase before it creates the initdb Job. That status update runs through an optimistic-lock retry loop; if it exhausts its retry budget against concurrent writers to the cluster object, the reconcile returns while patching status: Operation cannot be fulfilled ... the object has been modified and the initdb Job is never created. The data PVC has already been persisted, carrying the cnpg.io/pvcStatus: initializing annotation.

A PVC is only promoted to ready when a Job that uses it completes, or when it is already attached to a running pod. With no initdb Job and no pod, it stays initializing. Meanwhile the PVC-status reconciler counts it as an existing instance (a still-unbound PVC counts), so Status.Instances becomes 1. From there the cluster is wedged: the first-primary bootstrap path is gated on Status.Instances == 0, so it is never re-entered and the Job is never retried; instead the operator picks the dangling primary PVC for reattachment and waits for it to become ready, which can never happen.

This matches the observed state of the stuck cluster: TargetPrimary was set (the first status write landed) but CurrentPrimary and the phase were empty (the FirstPrimary registration did not), and the PVC sat unbound with status initializing.

We have seen this intermittently in CI, where concurrent reconciles under API-server write contention make the conflict likely. It is not specific to GKE or to a storage class: a WaitForFirstConsumer class just leaves the orphaned PVC visibly Pending, but the deadlock is driven by the cnpg.io/pvcStatus annotation never reaching ready, not by the PVC's bind phase. A cluster that wins the status update bootstraps normally, which is why it is intermittent and hard to reproduce deterministically.

Code references: the bootstrap gate is in internal/controller/cluster_controller.go (if cluster.Status.Instances == 0 then createPrimaryInstance); the early return before Job creation is in createPrimaryInstance in internal/controller/cluster_create.go (status registration runs before r.Create(job)); the ready-PVC wait is in ensureInstancesAreCreated in the same file; the promotion of a PVC to ready only on a completed Job or an attached pod is in pkg/reconciler/persistentvolumeclaim/status.go; and the same file counts an unbound PVC as an instance (cluster.Status.Instances = len(sortedInstances)).

Related issues: this is the same deadlock end-state as #4147 (closed as stale), but reached without any manual intervention. #7709 (open) hits the same cnpg.io/pvcStatus readiness gate during replica scale-up from a volume snapshot. #9021 reported a similar deadlock but was self-closed by the reporter as a storage-driver misconfiguration; the operator-side readiness gate it describes is unchanged.

Cluster resource

Relevant log output

# first reconcile: PVC created, initdb event emitted, then the status update exhausts its conflict-retry budget
info   Primary instance (initdb)   {"logging_pod": "...", "reason": "CreatingInstance"}
error  Reconciler error            {"error": "while patching status: Operation cannot be fulfilled on clusters.postgresql.cnpg.io \"<cluster>\": the object has been modified; please apply your changes to the latest version and try again"}

# no "Creating new Job" line for this cluster -- the initdb Job is never created

# every subsequent reconcile, indefinitely:
info   Selected PVC is not ready yet, waiting for 1 second   {"pvc": "<cluster>-1", "status": "initializing", "instance": "<cluster>-1"}

# cluster never leaves bootstrap: 0 pods, 0 jobs, unbound PVC, TargetPrimary set but CurrentPrimary/phase empty
Ready pod number:        0
Jobs information:        (none)
Cluster target primary   <cluster>-1
Cluster current primary  (empty)
Cluster phase            (empty)
PVC name                 <cluster>-1
PVC phase                Pending

Code of Conduct

• I agree to follow this project's Code of Conduct