fix(cluster): strip stale primary label during failover - #10409

When the operator initiates a failover, the old primary's pod keeps its
`cnpg.io/instanceRole=primary` label until `ReconcileMetadata` runs. But
`ReconcileMetadata` is skipped during the entire failover window (the
`CurrentPrimary != TargetPrimary` guard returns early), so the `-rw`
service keeps routing to the old primary. If the old primary comes back
(e.g. after a temporary network partition), replicas reconnect through
the `-rw` service, satisfy the sync quorum, and writes committed on the
stale primary are lost to `pg_rewind`.

Introduce a third value for the instance role label, `unhealthy`, and
apply it to the old primary as soon as failover starts. Since neither
the `-rw` nor the `-ro` service selector matches `unhealthy`, the pod is
immediately isolated from all service traffic for the duration of the
failover window. `ReconcileMetadata` restores the `replica` label once
`CurrentPrimary == TargetPrimary`.

The label is applied best-effort at the point where failover is
initiated, and re-applied on every pass of the reconcile loop while the
failover is in progress so transient API errors are retried
automatically.

Note: stripping the label removes the pod from the service Endpoints,
but does not drop TCP connections already established by a replica's
walreceiver. This fix closes the reconnection window; established
connections must still be terminated by the Postgres-level promotion on
the new primary.

Closes #10403

Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
(cherry picked from commit 57436b2)

When the operator initiates a failover, the old primary's pod keeps its
`cnpg.io/instanceRole=primary` label until `ReconcileMetadata` runs. But
`ReconcileMetadata` is skipped during the entire failover window (the
`CurrentPrimary != TargetPrimary` guard returns early), so the `-rw`
service keeps routing to the old primary. If the old primary comes back
(e.g. after a temporary network partition), replicas reconnect through
the `-rw` service, satisfy the sync quorum, and writes committed on the
stale primary are lost to `pg_rewind`.

Introduce a third value for the instance role label, `unhealthy`, and
apply it to the old primary as soon as failover starts. Since neither
the `-rw` nor the `-ro` service selector matches `unhealthy`, the pod is
immediately isolated from all service traffic for the duration of the
failover window. `ReconcileMetadata` restores the `replica` label once
`CurrentPrimary == TargetPrimary`.

The label is applied best-effort at the point where failover is
initiated, and re-applied on every pass of the reconcile loop while the
failover is in progress so transient API errors are retried
automatically.

Note: stripping the label removes the pod from the service Endpoints,
but does not drop TCP connections already established by a replica's
walreceiver. This fix closes the reconnection window; established
connections must still be terminated by the Postgres-level promotion on
the new primary.

Closes #10403

Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
(cherry picked from commit 57436b2)

When the operator initiates a failover, the old primary's pod keeps its
`cnpg.io/instanceRole=primary` label until `ReconcileMetadata` runs. But
`ReconcileMetadata` is skipped during the entire failover window (the
`CurrentPrimary != TargetPrimary` guard returns early), so the `-rw`
service keeps routing to the old primary. If the old primary comes back
(e.g. after a temporary network partition), replicas reconnect through
the `-rw` service, satisfy the sync quorum, and writes committed on the
stale primary are lost to `pg_rewind`.

Introduce a third value for the instance role label, `unhealthy`, and
apply it to the old primary as soon as failover starts. Since neither
the `-rw` nor the `-ro` service selector matches `unhealthy`, the pod is
immediately isolated from all service traffic for the duration of the
failover window. `ReconcileMetadata` restores the `replica` label once
`CurrentPrimary == TargetPrimary`.

The label is applied best-effort at the point where failover is
initiated, and re-applied on every pass of the reconcile loop while the
failover is in progress so transient API errors are retried
automatically.

Note: stripping the label removes the pod from the service Endpoints,
but does not drop TCP connections already established by a replica's
walreceiver. This fix closes the reconnection window; established
connections must still be terminated by the Postgres-level promotion on
the new primary.

Closes #10403

Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
(cherry picked from commit 57436b2)

##### [\`v1.29.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.29.1)

**Release date:** May 8, 2026

##### Security and Supply Chain

- **`CVE-2026-44477` / `GHSA-423p-g724-fr39`: metrics exporter privilege escalation**: the metrics exporter no longer authenticates as the `postgres` superuser. It now uses a dedicated `cnpg_metrics_exporter` role with `pg_monitor` privileges only, closing a chain that let a low-privilege database user gain PostgreSQL superuser. ([`GHSA-423p-g724-fr39`](GHSA-423p-g724-fr39)) <!-- 1.29 1.28 1.25 -->

  Upgrade impact: custom monitoring queries that read user-owned tables, or use `target_databases: '*'` against databases where `PUBLIC CONNECT` has been revoked, need explicit `GRANT` statements to `cnpg_metrics_exporter`. See ["Custom query privileges and safety"](../monitoring.md#custom-query-privileges-and-safety) and ["Manually creating the metrics exporter role"](../monitoring.md#manually-creating-the-metrics-exporter-role) in the monitoring documentation.

  For replica clusters, upgrade the source primary cluster before any replica clusters that consume from it. The `cnpg_metrics_exporter` role is created on the source primary and replicates downstream; a replica cluster upgraded first will scrape against a missing role until the source primary upgrades. The manual-recovery section linked above also covers replica clusters.

- **Schema-qualified catalog references in default monitoring queries**: hardened the shipped monitoring configuration and documentation samples by qualifying every `pg_catalog` object explicitly. Unqualified references resolve through `search_path`, which a database user can manipulate to shadow built-in objects. ([#10576](cloudnative-pg/cloudnative-pg#10576)) <!-- 1.29 1.28 1.25 -->

- **Discoverable SBOM and provenance attestations**: SBOM and SLSA provenance attached to operator container images now follow the OCI 1.1 Referrers spec, so standard registry tooling and supply-chain scanners can discover them automatically. ([#10601](cloudnative-pg/cloudnative-pg#10601)) <!-- 1.29 1.28 1.25 -->

- **CVE remediation in `github.com/jackc/pgx/v5`**: bumped to v5.9.2 to pick up upstream fixes for `CVE-2026-33816` (memory-safety in `pgproto3`) and `GHSA-j88v-2chj-qfwx` (SQL injection via simple-protocol dollar-quoted string handling). ([#10437](cloudnative-pg/cloudnative-pg#10437), [#10499](cloudnative-pg/cloudnative-pg#10499))

- **CVE remediation in the Go runtime**: built with Go 1.26.3 to pick up upstream fixes in `crypto/x509`, `crypto/tls`, `net/http`, and `net` (CVE-2026-32280, CVE-2026-32281, CVE-2026-33810, CVE-2026-33814, CVE-2026-33811, CVE-2026-39825). ([#10463](cloudnative-pg/cloudnative-pg#10463), [#10647](cloudnative-pg/cloudnative-pg#10647)) <!-- 1.29 1.28 1.25 -->

- **Build pipeline hardening**: the Go 1.26.3 bump also addresses CVE-2026-42501 (`cmd/go` module-checksum validation), reducing supply-chain exposure during release builds. The affected code paths are not reachable from the running operator. ([#10647](cloudnative-pg/cloudnative-pg#10647)) <!-- 1.29 1.28 1.25 -->

##### Changes

- Switched TLS peer verification from `VerifyPeerCertificate` to `VerifyConnection`, which runs on every completed handshake (the former is skipped on resumed TLS 1.3 sessions). Session resumption is not enabled in CloudNativePG today, so this has no observable effect, but it future-proofs verification if session caching is introduced later. ([#10478](cloudnative-pg/cloudnative-pg#10478)) <!-- 1.29 1.28 1.25 -->

##### Fixes

- Fixed a failover window where the former primary kept its primary label. If it returned during failover (for example, after a transient network partition), the `-rw` service kept routing to it, replicas could reconnect, and committed writes were lost to `pg_rewind`. The old primary is now labeled `unhealthy` to isolate it from service traffic during failover. ([#10409](cloudnative-pg/cloudnative-pg#10409)) <!-- 1.29 1.28 1.25 -->

- Fixed failover not being triggered when the node hosting the primary becomes unreachable. The operator now reads the pod's `Ready` condition (flipped to `False` by the node controller when the kubelet stops reporting) instead of `ContainersReady`, which stays stale as `True` in that scenario. Combined with the spurious-failover guard ([#10445](cloudnative-pg/cloudnative-pg#10445)), failover triggers only when Kubernetes itself marks the pod not Ready. ([#10448](cloudnative-pg/cloudnative-pg#10448)) <!-- 1.29 1.28 1.25 -->

- Fixed spurious failovers caused by transient failures on the primary's HTTP status endpoint. ([#10445](cloudnative-pg/cloudnative-pg#10445)) <!-- 1.29 1.28 1.25 -->

- Fixed escaping of backslashes and control characters in PostgreSQL configuration values. Previously, such characters in parameters like `log_line_prefix` could corrupt the configuration file or be silently stripped at runtime. ([#10515](cloudnative-pg/cloudnative-pg#10515)) <!-- 1.29 1.28 1.25 -->

- Fixed `restore_command` construction to shell-quote each argument. Values such as a `destinationPath` containing whitespace (for example, `s3://my bucket/wal`) were word-split by the POSIX shell and passed to the WAL restore tool as separate arguments. ([#10518](cloudnative-pg/cloudnative-pg#10518)) <!-- 1.29 1.28 1.25 -->

- Tightened `recoveryTarget` validation in the admission webhook: `targetXID` must now be a non-negative 32-bit integer, and `targetName` must be shorter than 64 bytes and free of ASCII control characters. Malformed values are rejected at admission instead of failing later during PostgreSQL recovery. ([#10565](cloudnative-pg/cloudnative-pg#10565)) <!-- 1.29 1.28 1.25 -->

- Fixed snapshot restores failing when leftover `pgsql_tmp*` directories were present in the data directory. ([#10447](cloudnative-pg/cloudnative-pg#10447)) <!-- 1.29 1.28 1.25 -->

- Fixed a deadlock occurring when PVC storage size and resource requests are changed simultaneously. ([#10427](cloudnative-pg/cloudnative-pg#10427)) <!-- 1.29 1.28 1.25 -->