[Bug]: replica switchover demotion token is wiped to empty when the reconcile is requeued mid-transition #11074

Is there an existing issue already for this bug?

• I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

• I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

• I have read the troubleshooting guide and I think this is a new bug.

Contact Details

No response

Version

trunk (main)

What version of Kubernetes are you using?

1.34

What is your Kubernetes environment?

Cloud: Azure AKS

How did you install the operator?

YAML manifest

What happened?

During a controlled replica-cluster switchover, the operator writes the demotion token to Cluster.Status.DemotionToken and then immediately wipes it back to an empty string. The cluster completes the demotion and the former primary comes up as a designated primary in recovery, but Status.DemotionToken is left empty, so the token needed to promote the new primary is gone.

This is a contract mismatch between generateDemotionToken and its caller reconcileDemotionToken in pkg/reconciler/replicaclusterswitch.

generateDemotionToken (shutdown_wal.go) returns ("", nil) as a "no change, skip" sentinel when the recomputed token already matches the stored one. The caller reconcileDemotionToken (reconciler.go) treats the return value as authoritative and writes it unconditionally whenever it differs from the stored value:

demotionToken, err := generateDemotionToken(...) // returns "" on the no-change path
...
if cluster.Status.DemotionToken != demotionToken { // "<token>" != "" -> true
    cluster.Status.DemotionToken = demotionToken    // overwrites the good token with ""
    cli.Status().Patch(...)
}

In the normal path this is harmless because cleanupTransitionMetadata removes the ReplicaClusterDesignatedPrimaryTransition condition in the same reconcile, closing the IsDesignatedPrimaryTransitionCompleted gate before the function can run again. The bug surfaces whenever the reconcile is requeued in the window between writing the token and removing that condition: on requeue the gate is still open, generateDemotionToken hits the no-change short-circuit and returns "", and the caller overwrites the previously-stored token with empty.

Any requeue in that window triggers it (optimistic-lock conflict, controller restart, a failing webhook call). I hit it on AKS where the validating webhook call failed through the Konnectivity proxy right after the token was written:

03:48:32.0017  patching the demotionToken  value=[eyJ...]  previousValue=[]
03:48:32.040   replica_cluster_cleanup_transition: removing all the unnecessary metadata
03:48:32.160   Reconciler error: while cleaning up demotion transition metadata: ... failed calling webhook "vcluster.cnpg.io"
03:48:32.3415  shutdown_checkpoint: no changes in the token value, skipping
03:48:32.3416  patching the demotionToken  value=[]  previousValue=[eyJ...]

The cleanup retried and succeeded afterwards, so the switchover proceeded but the token ended up empty. A client reading Status.DemotionToken to promote the new primary then gets an empty value.

This is reproducible by the replica_mode_cluster_test.go demote/promote switchover e2e and is environment-independent (the feature was introduced in #4411 and the code is shared across release branches); cloud environments just make the mid-transition requeue likely enough to hit regularly.

Cluster resource

No response

Relevant log output

2026-06-29T03:48:32.001Z  replica_cluster                      patching the demotionToken in the  cluster status   value=[eyJ...] previousValue=[]
2026-06-29T03:48:32.040Z  replica_cluster_cleanup_transition   removing all the unnecessary metadata from the cluster object
2026-06-29T03:48:32.160Z                                       Reconciler error   error="while cleaning up demotion transition metadata: Internal error occurred: failed calling webhook \"vcluster.cnpg.io\": ..."
2026-06-29T03:48:32.341Z  shutdown_checkpoint                  no changes in the token value, skipping
2026-06-29T03:48:32.341Z  replica_cluster                      patching the demotionToken in the  cluster status   value=[] previousValue=[eyJ...]
2026-06-29T03:48:32.371Z  replica_cluster_cleanup_transition   removing all the unnecessary metadata from the cluster object

Code of Conduct

• I agree to follow this project's Code of Conduct