[Bug]: Instance recreation deadlocks when the old PVC is still terminating (join pod stuck Pending, never healed) #10985

Is there an existing issue already for this bug?

• I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

• I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

• I have read the troubleshooting guide and I think this is a new bug.

Contact Details

[email protected]

Version

trunk (main)

What version of Kubernetes are you using?

1.31

What is your Kubernetes environment?

Self-managed: kind (evaluation)

How did you install the operator?

YAML manifest

What happened?

When an instance's PVCs are deleted and the instance is recreated, the operator can wedge permanently if the old WAL PVC is still terminating at the moment it rebuilds the instance.

The data PVC and the WAL PVC are deleted independently, so they don't disappear at the same instant. If the data PVC has already finished terminating but the WAL PVC has not, the operator recreates the data PVC fresh, then calls createIfNotExists for the WAL PVC. The Create hits the still-present terminating object and returns AlreadyExists, which is swallowed (pkg/reconciler/persistentvolumeclaim/create.go), so no replacement WAL PVC is created. The operator then creates the join Job and its pod.

A moment later the old WAL PVC finishes terminating and is garbage-collected. Now there is no WAL PVC at all, and the join pod can never be scheduled (persistentvolumeclaim "<instance>-wal" not found). Crucially, nothing recovers from this: as long as the join Job exists, the reconciler returns early every loop (internal/controller/cluster_controller.go, "A job is currently running. Waiting") and never re-runs PVC reconciliation. The Job can't finish because its pod can't schedule, so the guard never clears. The instance is stuck Pending until the join times out.

The failure is silent on the operator side: there is no "creating PVC", no "already exists", and no reconcile error for the WAL PVC anywhere in the manager log. The only signal is the kube scheduler's FailedScheduling event on the wedged pod.

Observed on a main CI run reproducing it via the PGDATA Corruption e2e test (which deletes the primary's data and WAL PVCs and waits for the instance to rejoin), on kind k8s 1.31 with PG 18.4. It is a race, so it does not happen every time; it surfaces whenever the WAL PVC's deletion lags the data PVC's past the recreate. Expected behavior: the instance should be fully recreated (a new WAL PVC provisioned) and rejoin as a standby.

CI run (artifacts available while retained): https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/27937407195 (failing job kind-v1.31.14-PostgreSQL-18.4, spec tests/e2e/pg_data_corruption_test.go:215).

Cluster resource

# Any 3-instance cluster with separate WAL storage (walStorage) reproduces it
# when an instance's PVCs are deleted and recreated while the WAL PVC is still
# terminating. Reproduced by the e2e fixture tests/e2e/fixtures/pg_data_corruption.

Relevant log output

# kube scheduler events on the wedged join pod (instance recreated at 08:51:07):
08:51:07  FailedScheduling  0/4 nodes are available: persistentvolumeclaim
                            "cluster-...-no-slots-1-wal" is being deleted.
08:54:44  FailedScheduling  0/4 nodes are available: persistentvolumeclaim
                            "cluster-...-no-slots-1-wal" not found.

# operator manager log, every 5s from 08:51:07 until teardown:
08:51:07  info   Creating instance cluster-...-no-slots-1
08:51:07  info   Creating new Job
08:51:07  debug  A job is currently running. Waiting
08:51:12  debug  A job is currently running. Waiting
... (repeats; PVC reconciliation never runs again) ...

Code of Conduct

• I agree to follow this project's Code of Conduct