[Bug]: Database, Publication, and Subscription status is stale after the cluster is demoted to a replica #10867

Is there an existing issue already for this bug?

• I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

• I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

• I have read the troubleshooting guide and I think this is a new bug.

Contact Details

No response

Version

trunk (main)

What version of Kubernetes are you using?

1.36 (not yet in the dropdown; kind v1.36.1)

What is your Kubernetes environment?

Self-managed: kind (evaluation)

How did you install the operator?

YAML manifest

What happened?

Once a Database, Publication, or Subscription has been reconciled, the instance-side reconciler returns without requeueing when Generation == Status.ObservedGeneration (database_controller.go, publication_controller.go, and subscription_controller.go all share the gate). The cluster.IsReplica() check that reports the "cluster is a replica" condition runs after that early exit, so it is only reachable for objects that have not been reconciled yet.

When a cluster is later demoted to a replica in a distributed topology, the objects it had applied keep reporting their last primary-side status indefinitely, while an identical object created after the demotion reports unknown with the replica message. The stale applied: true is misleading: the object is read-only on the demoted cluster and its real state is now owned by the new primary's own resources.

To reproduce:

1. Create a Database on a primary cluster and wait for applied: true.
2. Demote the cluster to a replica (for example by changing .spec.replica.primary in a distributed topology).
3. The Database keeps applied: true, while a second Database created after the demotion reports the replica condition instead.

I expected both objects to report the same replica condition after the demotion.

Reproduced on a kind cluster with the operator built from current main (2731fb5), demoting via replica.enabled: true with an external source; see the log output below. db-c was applied before the demotion and never changes; db-d was created afterwards.

This is adjacent to, but not covered by, #10853: that PR only exempts deletion from the replica gate (IsReplica() && GetDeletionTimestamp().IsZero()), so a deleted object no longer gets stuck in Terminating on a replica. The generation early-exit still precedes the replica gate for live objects, so an already-applied object never reaches it and its status is never re-evaluated after the demotion. The DatabaseRole controller proposed in #6155 inherits the same pattern.

Cluster resource

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: repro
  namespace: repro
spec:
  instances: 1
  storage:
    size: 512Mi
  externalClusters:
  - name: fake-source
    connectionParameters:
      host: fake-source.invalid
      user: streaming_replica
      dbname: postgres
  replica:
    enabled: true
    source: fake-source

Relevant log output

$ kubectl get database -n repro -o custom-columns='NAME:.metadata.name,APPLIED:.status.applied,MESSAGE:.status.message'
NAME   APPLIED   MESSAGE
db-c   true      <none>
db-d   <none>    waiting for the cluster to become primary

Code of Conduct

• I agree to follow this project's Code of Conduct