[Bug]: Primary pod label not removed during failover, allowing stale -rw service routing #10403

Is there an existing issue already for this bug?

• I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

• I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

• I have read the troubleshooting guide and I think this is a new bug.

Contact Details

No response

Version

trunk (main)

What version of Kubernetes are you using?

No response

What is your Kubernetes environment?

Other

How did you install the operator?

No response

What happened?

During failover, the old primary's pod retains the cnpg.io/instanceRole: primary label longer than necessary. Since the -rw service routes traffic based on this label, the old primary can continue receiving connections until the label is eventually reconciled. This is a defense-in-depth issue that could compound with other problems (e.g. premature failover triggered by transient network loss).

Two label reconciliation gaps have been identified:

1. Primary label not removed when failover starts

When the operator initiates failover (replicas.go:139, setting targetPrimary to PendingFailoverMarker), it does not remove the primary label from the old primary's pod. The old primary should be cut off from the -rw service immediately at this point, since the operator has already committed to failing over.

2. Primary labels not reconciled when dual primaries are detected

When the old primary comes back and both pods report as primary, the dual-primary detection at cluster_controller.go:417 returns early before ReconcileMetadata at line 438 is reached. This means pod labels are never updated during the dual-primary state, and the -rw service keeps routing to both primaries until the old one self-heals.

Proposed fix

1. When failover is initiated (setting targetPrimary to PendingFailoverMarker), immediately strip the primary label from the old primary's pod. This cuts it off from the -rw service before the new primary is even elected.
2. When dual primaries are detected, remove the primary label from all pods declaring themselves primary, instead of returning early and skipping ReconcileMetadata.

Cluster resource

Relevant log output

Code of Conduct

• I agree to follow this project's Code of Conduct