[Bug]: Major upgrade job starts before old pods finish terminating #10297

Is there an existing issue already for this bug?

• I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

• I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

• I have read the troubleshooting guide and I think this is a new bug.

Version

trunk (main)

What version of Kubernetes are you using?

Any

What is your Kubernetes environment?

Any

How did you install the operator?

Any

What happened?

During a major version upgrade (pg_upgrade), the upgrade job can be created and started while old instance pods are still terminating. This causes the upgrade to fail because the old PostgreSQL data directory is still in use by the terminating pod.

The issue is 100% reproducible in environments where container images are preloaded, so the upgrade job pod starts instantly. In environments where images need to be pulled, the download time masks the race condition by giving old pods time to fully terminate.

Root cause

The race condition is in deleteAllPodsInMajorUpgradePreparation (pkg/reconciler/majorupgrade/reconciler.go:131-168).

The flow:

1. Major upgrade detected, deleteAllPodsInMajorUpgradePreparation is called.
2. First reconcile: pods have no deletion timestamp, they get deleted, function returns RequeueAfter: 30s.
3. Second reconcile: pods now have a deletionTimestamp but haven't been fully removed yet (graceful termination still in progress).
4. The function skips pods with deletionTimestamp != nil and since nothing new was deleted, foundSomethingToDelete stays false.
5. Function returns nil, nil, signaling "all clear" to proceed.
6. createMajorUpgradeJob creates the upgrade job immediately.
7. The upgrade job starts while old pods are still terminating, causing pg_upgrade to fail.

The function conflates "I already issued a delete request" with "the pod is gone." Pods with a deletionTimestamp are still running during their graceful termination period.

Expected behavior

The upgrade job should only be created after all old pods and jobs have been fully terminated (removed from the API server), not just marked for deletion.

Code of Conduct

• I agree to follow this project's Code of Conduct