fix: avoid spurious failovers when the primary status endpoint fails transiently - #10445

…transiently (#10445)

When the operator calls /pg/status on the primary during a
reconciliation loop and the request fails, a failover could be triggered
even if the primary pod was still Ready from the kubelet's perspective.

Since a successful /pg/status implies a passing readiness probe but not
vice versa, a failing /pg/status on an otherwise Ready pod is a strong
signal of a transient condition (e.g. a network hiccup between the
operator and the pod) rather than a genuine PostgreSQL health problem.

The reconciliation loop now requeues instead of proceeding when the
current primary equals the target primary, the pod is Ready, but
/pg/status is failing, applying the same conservative approach already
used when the kubelet has not yet refreshed a positive readiness probe.

A follow-up PR (#10509) will surface the deferred failover as a cluster
phase so the state is visible via `kubectl get cluster`.

See #10403

Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
(cherry picked from commit 96e445c)

…transiently (#10445)

When the operator calls /pg/status on the primary during a
reconciliation loop and the request fails, a failover could be triggered
even if the primary pod was still Ready from the kubelet's perspective.

Since a successful /pg/status implies a passing readiness probe but not
vice versa, a failing /pg/status on an otherwise Ready pod is a strong
signal of a transient condition (e.g. a network hiccup between the
operator and the pod) rather than a genuine PostgreSQL health problem.

The reconciliation loop now requeues instead of proceeding when the
current primary equals the target primary, the pod is Ready, but
/pg/status is failing, applying the same conservative approach already
used when the kubelet has not yet refreshed a positive readiness probe.

A follow-up PR (#10509) will surface the deferred failover as a cluster
phase so the state is visible via `kubectl get cluster`.

See #10403

Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
(cherry picked from commit 96e445c)

…transiently (#10445)

When the operator calls /pg/status on the primary during a
reconciliation loop and the request fails, a failover could be triggered
even if the primary pod was still Ready from the kubelet's perspective.

Since a successful /pg/status implies a passing readiness probe but not
vice versa, a failing /pg/status on an otherwise Ready pod is a strong
signal of a transient condition (e.g. a network hiccup between the
operator and the pod) rather than a genuine PostgreSQL health problem.

The reconciliation loop now requeues instead of proceeding when the
current primary equals the target primary, the pod is Ready, but
/pg/status is failing, applying the same conservative approach already
used when the kubelet has not yet refreshed a positive readiness probe.

A follow-up PR (#10509) will surface the deferred failover as a cluster
phase so the state is visible via `kubectl get cluster`.

See #10403

Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
(cherry picked from commit 96e445c)

The prior commit on this branch (f1204b22c) surfaced the /pg/status
unreachability guard as a new v1 Phase, PhaseWaitingForPrimaryStatus.
On reflection, Event is a better fit for this observation:

- The signal is transient (a reconcile-loop observation, not a
  lifecycle state). Phase is single-slot and would flap between
  Healthy and Waiting on each 10s requeue, polluting the one-line
  `kubectl get cluster` view rather than informing it.
- Phase permanently extends the v1 API enum; Events do not.
- The Kubernetes event recorder natively deduplicates by
  (involvedObject, reason, message), so a sustained outage rolls up
  into one logical event with a growing count and moving
  lastTimestamp.

This commit drops PhaseWaitingForPrimaryStatus and its RegisterPhase
call, and emits a Warning PrimaryStatusUnreachable event from
evaluatePodReadinessGuards in its place. The event message is kept
stable (no %v of the error) to preserve dedup; the underlying error
is logged at the guard site via contextLogger.Warning with an 'err'
structured field, and the event message points users there.

IsPodReadyAndNotReporting now returns (bool, error) so the guard's
log line can carry the instance manager's error without a second
iteration over the status list. The helper has a single caller, so
the signature change is zero blast radius.

The (ctrl.Result, error) signature of evaluatePodReadinessGuards
introduced in f1204b22c is reverted to a single ctrl.Result return:
without the RegisterPhase call, the function has no error source,
and golangci-lint's unparam rightly flags result 1 as always nil.

Deliberately out of scope (follow-ups):
- No max-duration escape when operator-to-primary networking is
  persistently broken while the kubelet readiness probe keeps
  passing. /readyz and /pg/status are served by different handlers
  in the instance manager and can diverge.
- No new Prometheus metric for time since last successful /pg/status.
- No event emission on the kubelet-stale-readiness branch (expected
  sub-second, event noise not worth the added signal).

Depends on #10445.

Signed-off-by: Armando Ruocco <[email protected]>

The prior commit on this branch (f1204b22c) surfaced the /pg/status
unreachability guard as a new v1 Phase, PhaseWaitingForPrimaryStatus.
On reflection, Event is a better fit for this observation:

- The signal is transient (a reconcile-loop observation, not a
  lifecycle state). Phase is single-slot and would flap between
  Healthy and Waiting on each 10s requeue, polluting the one-line
  `kubectl get cluster` view rather than informing it.
- Phase permanently extends the v1 API enum; Events do not.
- The Kubernetes event recorder natively deduplicates by
  (involvedObject, reason, message), so a sustained outage rolls up
  into one logical event with a growing count and moving
  lastTimestamp.

This commit drops PhaseWaitingForPrimaryStatus and its RegisterPhase
call, and emits a Warning PrimaryStatusUnreachable event from
evaluatePodReadinessGuards in its place. The event message is kept
stable (no %v of the error) to preserve dedup; the underlying error
is logged at the guard site via contextLogger.Warning with an 'err'
structured field, and the event message points users there.

IsPodReadyAndNotReporting now returns (bool, error) so the guard's
log line can carry the instance manager's error without a second
iteration over the status list. The helper has a single caller, so
the signature change is zero blast radius.

The (ctrl.Result, error) signature of evaluatePodReadinessGuards
introduced in f1204b22c is reverted to a single ctrl.Result return:
without the RegisterPhase call, the function has no error source,
and golangci-lint's unparam rightly flags result 1 as always nil.

Deliberately out of scope (follow-ups):
- No max-duration escape when operator-to-primary networking is
  persistently broken while the kubelet readiness probe keeps
  passing. /readyz and /pg/status are served by different handlers
  in the instance manager and can diverge.
- No new Prometheus metric for time since last successful /pg/status.
- No event emission on the kubelet-stale-readiness branch (expected
  sub-second, event noise not worth the added signal).

Depends on #10445.

Signed-off-by: Armando Ruocco <[email protected]>

The prior commit on this branch (f1204b22c) surfaced the /pg/status
unreachability guard as a new v1 Phase, PhaseWaitingForPrimaryStatus.
On reflection, Event is a better fit for this observation:

- The signal is transient (a reconcile-loop observation, not a
  lifecycle state). Phase is single-slot and would flap between
  Healthy and Waiting on each 10s requeue, polluting the one-line
  `kubectl get cluster` view rather than informing it.
- Phase permanently extends the v1 API enum; Events do not.
- The Kubernetes event recorder natively deduplicates by
  (involvedObject, reason, message), so a sustained outage rolls up
  into one logical event with a growing count and moving
  lastTimestamp.

This commit drops PhaseWaitingForPrimaryStatus and its RegisterPhase
call, and emits a Warning PrimaryStatusUnreachable event from
evaluatePodReadinessGuards in its place. The event message is kept
stable (no %v of the error) to preserve dedup; the underlying error
is logged at the guard site via contextLogger.Warning with an 'err'
structured field, and the event message points users there.

IsPodReadyAndNotReporting now returns (bool, error) so the guard's
log line can carry the instance manager's error without a second
iteration over the status list. The helper has a single caller, so
the signature change is zero blast radius.

The (ctrl.Result, error) signature of evaluatePodReadinessGuards
introduced in f1204b22c is reverted to a single ctrl.Result return:
without the RegisterPhase call, the function has no error source,
and golangci-lint's unparam rightly flags result 1 as always nil.

Deliberately out of scope (follow-ups):
- No max-duration escape when operator-to-primary networking is
  persistently broken while the kubelet readiness probe keeps
  passing. /readyz and /pg/status are served by different handlers
  in the instance manager and can diverge.
- No new Prometheus metric for time since last successful /pg/status.
- No event emission on the kubelet-stale-readiness branch (expected
  sub-second, event noise not worth the added signal).

Depends on #10445.

Signed-off-by: Armando Ruocco <[email protected]>

##### [\`v1.29.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.29.1)

**Release date:** May 8, 2026

##### Security and Supply Chain

- **`CVE-2026-44477` / `GHSA-423p-g724-fr39`: metrics exporter privilege escalation**: the metrics exporter no longer authenticates as the `postgres` superuser. It now uses a dedicated `cnpg_metrics_exporter` role with `pg_monitor` privileges only, closing a chain that let a low-privilege database user gain PostgreSQL superuser. ([`GHSA-423p-g724-fr39`](GHSA-423p-g724-fr39)) <!-- 1.29 1.28 1.25 -->

  Upgrade impact: custom monitoring queries that read user-owned tables, or use `target_databases: '*'` against databases where `PUBLIC CONNECT` has been revoked, need explicit `GRANT` statements to `cnpg_metrics_exporter`. See ["Custom query privileges and safety"](../monitoring.md#custom-query-privileges-and-safety) and ["Manually creating the metrics exporter role"](../monitoring.md#manually-creating-the-metrics-exporter-role) in the monitoring documentation.

  For replica clusters, upgrade the source primary cluster before any replica clusters that consume from it. The `cnpg_metrics_exporter` role is created on the source primary and replicates downstream; a replica cluster upgraded first will scrape against a missing role until the source primary upgrades. The manual-recovery section linked above also covers replica clusters.

- **Schema-qualified catalog references in default monitoring queries**: hardened the shipped monitoring configuration and documentation samples by qualifying every `pg_catalog` object explicitly. Unqualified references resolve through `search_path`, which a database user can manipulate to shadow built-in objects. ([#10576](cloudnative-pg/cloudnative-pg#10576)) <!-- 1.29 1.28 1.25 -->

- **Discoverable SBOM and provenance attestations**: SBOM and SLSA provenance attached to operator container images now follow the OCI 1.1 Referrers spec, so standard registry tooling and supply-chain scanners can discover them automatically. ([#10601](cloudnative-pg/cloudnative-pg#10601)) <!-- 1.29 1.28 1.25 -->

- **CVE remediation in `github.com/jackc/pgx/v5`**: bumped to v5.9.2 to pick up upstream fixes for `CVE-2026-33816` (memory-safety in `pgproto3`) and `GHSA-j88v-2chj-qfwx` (SQL injection via simple-protocol dollar-quoted string handling). ([#10437](cloudnative-pg/cloudnative-pg#10437), [#10499](cloudnative-pg/cloudnative-pg#10499))

- **CVE remediation in the Go runtime**: built with Go 1.26.3 to pick up upstream fixes in `crypto/x509`, `crypto/tls`, `net/http`, and `net` (CVE-2026-32280, CVE-2026-32281, CVE-2026-33810, CVE-2026-33814, CVE-2026-33811, CVE-2026-39825). ([#10463](cloudnative-pg/cloudnative-pg#10463), [#10647](cloudnative-pg/cloudnative-pg#10647)) <!-- 1.29 1.28 1.25 -->

- **Build pipeline hardening**: the Go 1.26.3 bump also addresses CVE-2026-42501 (`cmd/go` module-checksum validation), reducing supply-chain exposure during release builds. The affected code paths are not reachable from the running operator. ([#10647](cloudnative-pg/cloudnative-pg#10647)) <!-- 1.29 1.28 1.25 -->

##### Changes

- Switched TLS peer verification from `VerifyPeerCertificate` to `VerifyConnection`, which runs on every completed handshake (the former is skipped on resumed TLS 1.3 sessions). Session resumption is not enabled in CloudNativePG today, so this has no observable effect, but it future-proofs verification if session caching is introduced later. ([#10478](cloudnative-pg/cloudnative-pg#10478)) <!-- 1.29 1.28 1.25 -->

##### Fixes

- Fixed a failover window where the former primary kept its primary label. If it returned during failover (for example, after a transient network partition), the `-rw` service kept routing to it, replicas could reconnect, and committed writes were lost to `pg_rewind`. The old primary is now labeled `unhealthy` to isolate it from service traffic during failover. ([#10409](cloudnative-pg/cloudnative-pg#10409)) <!-- 1.29 1.28 1.25 -->

- Fixed failover not being triggered when the node hosting the primary becomes unreachable. The operator now reads the pod's `Ready` condition (flipped to `False` by the node controller when the kubelet stops reporting) instead of `ContainersReady`, which stays stale as `True` in that scenario. Combined with the spurious-failover guard ([#10445](cloudnative-pg/cloudnative-pg#10445)), failover triggers only when Kubernetes itself marks the pod not Ready. ([#10448](cloudnative-pg/cloudnative-pg#10448)) <!-- 1.29 1.28 1.25 -->

- Fixed spurious failovers caused by transient failures on the primary's HTTP status endpoint. ([#10445](cloudnative-pg/cloudnative-pg#10445)) <!-- 1.29 1.28 1.25 -->

- Fixed escaping of backslashes and control characters in PostgreSQL configuration values. Previously, such characters in parameters like `log_line_prefix` could corrupt the configuration file or be silently stripped at runtime. ([#10515](cloudnative-pg/cloudnative-pg#10515)) <!-- 1.29 1.28 1.25 -->

- Fixed `restore_command` construction to shell-quote each argument. Values such as a `destinationPath` containing whitespace (for example, `s3://my bucket/wal`) were word-split by the POSIX shell and passed to the WAL restore tool as separate arguments. ([#10518](cloudnative-pg/cloudnative-pg#10518)) <!-- 1.29 1.28 1.25 -->

- Tightened `recoveryTarget` validation in the admission webhook: `targetXID` must now be a non-negative 32-bit integer, and `targetName` must be shorter than 64 bytes and free of ASCII control characters. Malformed values are rejected at admission instead of failing later during PostgreSQL recovery. ([#10565](cloudnative-pg/cloudnative-pg#10565)) <!-- 1.29 1.28 1.25 -->

- Fixed snapshot restores failing when leftover `pgsql_tmp*` directories were present in the data directory. ([#10447](cloudnative-pg/cloudnative-pg#10447)) <!-- 1.29 1.28 1.25 -->

- Fixed a deadlock occurring when PVC storage size and resource requests are changed simultaneously. ([#10427](cloudnative-pg/cloudnative-pg#10427)) <!-- 1.29 1.28 1.25 -->

The prior commit on this branch (f1204b22c) surfaced the /pg/status
unreachability guard as a new v1 Phase, PhaseWaitingForPrimaryStatus.
On reflection, Event is a better fit for this observation:

- The signal is transient (a reconcile-loop observation, not a
  lifecycle state). Phase is single-slot and would flap between
  Healthy and Waiting on each 10s requeue, polluting the one-line
  `kubectl get cluster` view rather than informing it.
- Phase permanently extends the v1 API enum; Events do not.
- The Kubernetes event recorder natively deduplicates by
  (involvedObject, reason, message), so a sustained outage rolls up
  into one logical event with a growing count and moving
  lastTimestamp.

This commit drops PhaseWaitingForPrimaryStatus and its RegisterPhase
call, and emits a Warning PrimaryStatusUnreachable event from
evaluatePodReadinessGuards in its place. The event message is kept
stable (no %v of the error) to preserve dedup; the underlying error
is logged at the guard site via contextLogger.Warning with an 'err'
structured field, and the event message points users there.

IsPodReadyAndNotReporting now returns (bool, error) so the guard's
log line can carry the instance manager's error without a second
iteration over the status list. The helper has a single caller, so
the signature change is zero blast radius.

The (ctrl.Result, error) signature of evaluatePodReadinessGuards
introduced in f1204b22c is reverted to a single ctrl.Result return:
without the RegisterPhase call, the function has no error source,
and golangci-lint's unparam rightly flags result 1 as always nil.

Deliberately out of scope (follow-ups):
- No max-duration escape when operator-to-primary networking is
  persistently broken while the kubelet readiness probe keeps
  passing. /readyz and /pg/status are served by different handlers
  in the instance manager and can diverge.
- No new Prometheus metric for time since last successful /pg/status.
- No event emission on the kubelet-stale-readiness branch (expected
  sub-second, event noise not worth the added signal).

Depends on #10445.

Signed-off-by: Armando Ruocco <[email protected]>

…10509)

When the current primary pod is Ready from the kubelet's perspective but the
operator's `/pg/status` check fails, the parent PR (#10445) defers failover
and requeues on a 10s loop. Previously this loop was silent from the user's
point of view. This PR surfaces the deferral as a `Warning`
`PrimaryStatusCheckFailed` event on the Cluster, emitted from
`evaluatePodReadinessGuards`.

Users see the event in `kubectl describe cluster` and via the events API;
the Kubernetes event recorder deduplicates by (involvedObject, reason,
message), so a sustained outage aggregates into a single Event object with
a growing count. The event message is kept stable (no error interpolation)
to preserve dedup; the underlying error is logged at the guard site via
`contextLogger.Warning(..., "err", statusErr)` and the event message
points users there.

Closes #10516

Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>

…10509)

When the current primary pod is Ready from the kubelet's perspective but the
operator's `/pg/status` check fails, the parent PR (#10445) defers failover
and requeues on a 10s loop. Previously this loop was silent from the user's
point of view. This PR surfaces the deferral as a `Warning`
`PrimaryStatusCheckFailed` event on the Cluster, emitted from
`evaluatePodReadinessGuards`.

Users see the event in `kubectl describe cluster` and via the events API;
the Kubernetes event recorder deduplicates by (involvedObject, reason,
message), so a sustained outage aggregates into a single Event object with
a growing count. The event message is kept stable (no error interpolation)
to preserve dedup; the underlying error is logged at the guard site via
`contextLogger.Warning(..., "err", statusErr)` and the event message
points users there.

Closes #10516

Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
(cherry picked from commit c6d3a94)

…10509)

When the current primary pod is Ready from the kubelet's perspective but the
operator's `/pg/status` check fails, the parent PR (#10445) defers failover
and requeues on a 10s loop. Previously this loop was silent from the user's
point of view. This PR surfaces the deferral as a `Warning`
`PrimaryStatusCheckFailed` event on the Cluster, emitted from
`evaluatePodReadinessGuards`.

Users see the event in `kubectl describe cluster` and via the events API;
the Kubernetes event recorder deduplicates by (involvedObject, reason,
message), so a sustained outage aggregates into a single Event object with
a growing count. The event message is kept stable (no error interpolation)
to preserve dedup; the underlying error is logged at the guard site via
`contextLogger.Warning(..., "err", statusErr)` and the event message
points users there.

Closes #10516

Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
(cherry picked from commit c6d3a94)

…10509)

When the current primary pod is Ready from the kubelet's perspective but the
operator's `/pg/status` check fails, the parent PR (#10445) defers failover
and requeues on a 10s loop. Previously this loop was silent from the user's
point of view. This PR surfaces the deferral as a `Warning`
`PrimaryStatusCheckFailed` event on the Cluster, emitted from
`evaluatePodReadinessGuards`.

Users see the event in `kubectl describe cluster` and via the events API;
the Kubernetes event recorder deduplicates by (involvedObject, reason,
message), so a sustained outage aggregates into a single Event object with
a growing count. The event message is kept stable (no error interpolation)
to preserve dedup; the underlying error is logged at the guard site via
`contextLogger.Warning(..., "err", statusErr)` and the event message
points users there.

Closes #10516

Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
(cherry picked from commit c6d3a94)