[Bug]: Race condition in unschedulable node auto-switchover #11293

Activity

1. chunngaiau

   assigned

   gbartolini

   on Aug 4, 2026
2. chunngaiau

   added

   bug 🐛Something isn't workingSomething isn't working

   triagePending triagePending triage

   on Aug 4, 2026
3. dosubot commented on Aug 4, 2026

   

   dosubotBot

   on Aug 4, 2026 – with DosuBot

   More actions

   Your analysis is spot on, and this is indeed the same class of race condition as #11114 but triggered via a different code path.

   Looking at the code, setPrimaryOnSchedulableNode selects a candidate and immediately calls setPrimaryInstance() to designate the new target — without first shutting down or demoting the old primary. This is the same pattern as updatePrimaryPod in the upgrade path.

   What makes this particularly concerning is that the normal failover path in reconcileTargetPrimaryForNonReplicaCluster is actually safer — it first sets a PendingFailoverMarker, marks the old primary as unhealthy, waits for WAL receivers to go down, and only then selects and promotes the new primary [1]. setPrimaryOnSchedulableNode skips all of these safety steps and goes straight to promotion.

   With method: any, number: 1, PostgreSQL can acknowledge commits to either replica. So the 45 commits that landed on -3 but not -2 between target selection and primary shutdown are exactly the expected failure mode of this race.

   Mitigations available today (same as #11114):

   • method: first makes the synchronous standby deterministic, significantly reducing (but not fully eliminating) the race window [2]
   • primaryUpdateMethod: restart avoids the switchover path entirely, though it doesn't help here since the trigger is node drain, not a config update [2]

   v1.30 Primary Lease: The primary lease mechanism merged in v1.30 adds a mutex at promotion time — the new primary must acquire the lease before promoting, and the old primary must release it during shutdown. You mention you're on v1.30 already, so the lease should be active. However, the lease prevents dual-primary but doesn't fully close the window where setPrimaryOnSchedulableNode selects the target before the old primary stops accepting writes. The fix likely needs the same approach as the failover path: use PendingFailoverMarker → demote old primary → wait for WAL receivers down → then select the best candidate by LSN.

   As @ardentperf noted on #11114, the operator would ideally use the lease to ensure the old primary is confirmed shut down before gathering LSNs and selecting a new primary [3].

   To reply, just mention @dosu.

   ---

   Share context across your team and agents. Try Dosu.

   
4. chunngaiau commented on Aug 4, 2026

   

   chunngaiau

   on Aug 4, 2026

   Author

   More actions

   Happy to contribute a fix to this issue. I've opened a draft PR with a potential fix #11294. Also, credit to #11185 for the inspiration of how to fix the previous issue and bringing both the failover and switchover logic into the same code paths.

   Appreciate any feedback as this is my first contribution to the project.
5. ardentperf commented on Aug 4, 2026

   

   ardentperf

   on Aug 4, 2026

   Last edited by ardentperf

   Contributor

   More actions

   Postgres's design for sync replication is that it will write a commit record to the local WAL however it does NOT send the acknowledgment to the client if there is a network partition and it cannot replicate the commit record. From the clients perspective, the transactions are in-doubt and uncommitted (even if a local WAL log temporarily has the records, until rewind).

   Split brain isn’t a good way to describe this. There are not divergent timelines of client-committed transaction, there is no loss of data known by the client as committed, and there are not two pods processing & acknowledging writes at the same time. During a network partition, all backends and clients will hang on IPC:SyncRep waits.

   It is expected to see a few COMMIT records in the WAL. Those in-doubt COMMITs will always be written to the WAL, even when the new primary starts after the old shuts down according to the wall clock. Clients never receive messages saying these COMMITs were successful.

   The in-doubt transactions may later become visible if there is no cluster reconfiguration and the existing primary is restarted after resolution of the network partition. On the other hand, the in-doubt transactions should never be visible to any client if a new primary is elected and the old primary is rewound after shutdown. Importantly: the thing that should never happen is in-doubt transactions being visible to a client and then later disappearing due to rewind and cluster reconfiguration.
6. chunngaiau commented on Aug 5, 2026

   

   chunngaiau

   on Aug 5, 2026

   Author

   More actions

   @ardentperf Thanks for the reply and sorry if I have misunderstood the Postgres mechanics here.

   There are not divergent timelines of client-committed transaction, there is no loss of data known by the client as committed, and there are not two pods processing & acknowledging writes at the same time.

   I agree maybe "split-brain" is the not the right word here as only one of the instances are accepting writes at any time (I am new to the terminology). "Data loss" might also be wrong as the data is still there on the diverged replica. But I think there is a loss of client-committed transactions (transactions for which the client has received an acknowledgement) and not just a loss of in-doubt transactions.

   My understanding is that the commits would've been acknowledged by the primary and its clients as long as it's been replicated to one of the replicas (we are running a 3-instance cluster with ANY 1 synchronous replication), but that may not be the pre-selected new primary. From the client's perspective, it would've received acknowledgement of the commit from the old primary, but after the switchover, the commit isn't visible anymore on the new primary.

   The issue here is that the new primary is selected while the old primary is still accepting writes. The old primary can accept COMMITs, write that to WAL, wait until it is acknowledged by any replica and return success to the client between when the new primary was selected and when it is finally promoted. In this period, there is no guarantee that all acknowledged commits are replicated to the pre-selected new primary. In fact, the pre-selected new primary could have lost connection to the new primary immediately after being selected as the new primary. It's worth noting that in smart and fast shutdown, Postgres will try to flush all remaining WAL to its replicas, but this is not guaranteed and the replicas have to be connected.

   I realise I am only talking theoretically here based on tracing the code and I agree seeing commit records with pg_waldump only indicates a possible but not definitive issue. Unfortunately I don't have clear evidence the commits were indeed acknowledged to the client. I'll see if I can come up with a reproducer example to help with the discussions, although this bug relies on very specific timings of the switchover process, so I'm not sure how feasible it is.
7. ardentperf commented on Aug 5, 2026

   

   ardentperf

   on Aug 5, 2026

   Contributor

   More actions

   The issue here is that the new primary is selected while the old primary is still accepting writes. The old primary can accept COMMITs, write that to WAL, wait until it is acknowledged by any replica and return success to the client between when the new primary was selected and when it is finally promoted. In this period, there is no guarantee that all acknowledged commits are replicated to the pre-selected new primary. In fact, the pre-selected new primary could have lost connection to the new primary immediately after being selected as the new primary.

   CNPG's quorum-based failover should ensure that whenever a new primary is selected, it contains all COMMITs/transactions that have been acknowledged to clients

   https://cloudnative-pg.io/docs/devel/failover/#failover-quorum-quorum-based-failover

   some example network partition scenarios are at the 20 minute mark in last year's kubecon presentation https://youtu.be/iQUOO3-JRK4?si=_nDfBoJn8jsgRmiJ&t=1200

   👍React with 👍1Reacted by maxLengdell

   👀React with 👀1Reacted by Chun Ngai Au
8. chunngaiau commented on Aug 7, 2026

   

   chunngaiau

   on Aug 7, 2026

   Author

   More actions

   CNPG's quorum-based failover should ensure that whenever a new primary is selected, it contains all COMMITs/transactions that have been acknowledged to clients

   I agree and I think the failover code you are referring to is quite robust. Thank you for linking the presentation.

   To clarify, there are currently three ways automatic cluster reconfiguration happens:

   1. Failover when the primary is unhealthy. This is robust and there is no issue here.
   2. Switchover to update the primary pod. Referenced in [Bug]: Potential race condition in primary update switchover #11114 bug report.
   3. Switchover to move primary to a schedulable node when the primary node is tainted. Referenced in this bug report.

   All three uses slightly different code paths on the operator. The setPrimaryOnSchedulableNode method referenced in this bug report picks the first (sorted by LSN) replica that is on a schedulable node. It does not check if the number of candidates satisfy the failover quorum. The failover quorum check is only performed in the failover code path in the reconcileTargetPrimaryForNonReplicaCluster method. This was confirmed by the lack of quorum check logs in our cluster during the node pool upgrade which taints the primary node.

   This would be fine in most cases though because the setPrimaryOnSchedulableNode method waits until all replicas are on schedulable nodes before initiating the switchover so the chance of it failing the quorum check is pretty low. Although this isn't the case when there are failed pods.

   Nevertheless, even if the failover quorum was checked, the issue remains. If we trace the failover code in reconcileTargetPrimaryForNonReplicaCluster method, it first sets the target primary to a pending marker, waits for all WAL receivers (and indirectly for the primary) to shut down before picking the new primary with the highest LSN. AFAIK this is the correct sequence of steps, however the switchover code in setPrimaryOnSchedulableNode here (and updatePrimaryPod here from #11114) does not follow these steps. They pick the new primary directly without using the pending marker and it's done when the old primary is still accepting writes. This may lead to a less advanced replica being promoted to a new primary, which we observed in both bug reports.

   I'm not sure if it was an intentional design decision to have separate code paths for failover and switchover. My understanding is that they should use the same mechanics for cluster reconfiguration and hence should share the same code.
9. chadlangston commented on Aug 10, 2026

   

   chadlangston

   on Aug 10, 2026

   More actions

   Thank you for working on this!

   I experienced this bug recently during routine k8s maintenance.

   I have a five instance cnpg cluster.

   When I cordoned the node running the primary, cnpg promoted one of my replicas to be the new primary before shutting down the existing primary.

   This resulted in incompatible timelines and replica crash loops. The controller restored the original primary with pg_rewind.

   The remaining replicas had to be recreated on new PVs from the new primary.
10. ardentperf commented on Aug 13, 2026

    

    ardentperf

    on Aug 13, 2026

    Last edited by ardentperf

    Contributor

    More actions

    @chunngaiau thanks for the analysis here. i want to walk through the code more closely to better understand the path with node taints. good questions

    @chadlangston want to be careful about drawing conclusions from wall-clock comparisons or pg_rewind execution. bumping the timeline and running rewind doesn't indicate that any transactions were lost (this can potentially happen in a healthy setup, depending how the orchestration works). the case i most recently saw of a replica crash loop with an incompatible timeline was related to a bug in barman-cloud rather than cnpg itself (notably, it did not involve any lost transactions). the focus should be on client-committed transactions, and kits like jepsen are a good way to test for this since they record client-side data and reconcile it with the database. what were your sync replication settings?
11. github-actions commented on Nov 9, 2026

    

    github-actionsBot

    on Nov 9, 2026 – with GitHub Actions

    Contributor

    More actions

    This issue is stale because it has been open for 60 days with no activity.
12. github-actions

    added

    Stale

    on Nov 9, 2026