Repository navigation
[Bug]: Race condition in replica cluster switch prevents designated primary transition completion #9591
Description
Is there an existing issue already for this bug?
- I have searched for an existing issue, and could not find anything. I believe this is a new bug.
I have read the troubleshooting guide
- I have read the troubleshooting guide and I think this is a new bug.
I am running a supported version of CloudNativePG
- I have read the troubleshooting guide and I think this is a new bug.
Contact Details
No response
Version
trunk (main)
What version of Kubernetes are you using?
1.34
What is your Kubernetes environment?
Self-managed: kind (evaluation)
How did you install the operator?
YAML manifest
What happened?
During a replica cluster switch operation, a rare race condition (observed in ~1-5% of test runs) occurs where the operator and instance manager concurrently modify the cluster object, resulting in an optimistic lock conflict that prevents the designated primary transition from completing.
From E2E test failure: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/20547048546
Relevant log output
Attempted transition:
{"ts":"2025-12-28T02:52:54.810548Z",
"msg":"Setting myself as the current designated primary"}Optimistic lock conflict:
{"ts":"2025-12-28T02:52:55.372302Z",
"error":"Operation cannot be fulfilled... the object has been modified"}Test timeout:
Timed out after 300.000s.
Expected pod to be in recovery mode
Code of Conduct
- I agree to follow this project's Code of Conduct
Reacted by dosubot
Activity
fix(replica_cluster): check condition for designated primary transiti…
mnencia commentedon Dec 30, 2025
The conflict scenario seen in the logs is actually very rare because
RetryOnConflictusesDefaultBackoffwith 4 retry steps. For the race condition to manifest, it requires 5 consecutive optimistic lock conflicts - meaning the operator must update the cluster status at precisely the right moment to conflict with each of the 5 retry attempts.This explains why the E2E test failure is intermittent: it only occurs when the timing aligns perfectly between the operator's updates and the instance manager's retries across all 5 attempts.
However, once the bug triggers (all five retries have been exhausted), the sentinel flips from TRUE to FALSE, and subsequent reconciliation loops early return, causing the operator to wait indefinitely. The fix ensures the sentinel remains TRUE throughout the transition, allowing natural reconciliation retries.
fix(replica_cluster): resolve race condition in designated primary tr…
fix(replica_cluster): resolve race condition in designated primary tr…
fix(replica_cluster): resolve race condition in designated primary tr…
fix(replica_cluster): resolve race condition in designated primary tr…
fix(replica_cluster): resolve race condition in designated primary tr…
fix(replica_cluster): resolve race condition in designated primary tr…
fix(replica_cluster): resolve race condition in designated primary tr…
fix(replica_cluster): resolve race condition in designated primary tr…