Skip to content

fix: trigger failover when the primary node becomes unreachable - #10448

Merged
mnencia merged 5 commits into
cloudnative-pg:mainfrom
leonardoce:dev/containerready
Apr 22, 2026
Merged

mnencia merged 5 commits into
cloudnative-pg:mainfrom
leonardoce:dev/containerready

Conversation

@leonardoce

@leonardoce leonardoce commented Apr 15, 2026 •

Copy link
Copy Markdown
Contributor

When the node hosting the primary loses network connectivity, the kubelet can no longer update the pod status. The pod's ContainersReady condition is left stale as True, while Kubernetes actively sets Ready to False via the node controller. By checking Ready instead of ContainersReady, the operator can now correctly detect that the primary is unavailable and proceed with a failover.

This change is safe in combination with the previous patch: when the status endpoint fails transiently the pod's Ready condition is still True, so the operator requeues rather than triggering a spurious failover.

See #10403

@leonardoce
leonardoce requested a review from a team April 15, 2026 13:04
@dosubot dosubot Bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Apr 15, 2026
@cnpg-bot cnpg-bot added backport-requested ◀️ This pull request should be backported to all supported releases release-1.25 release-1.28 labels Apr 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

❗ By default, the pull request is configured to backport to all release branches.

  • To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
  • To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

@dosubot dosubot Bot added the bug 🐛 Something isn't working label Apr 15, 2026
@leonardoce

Copy link
Copy Markdown
Contributor Author

/test

@github-actions

Copy link
Copy Markdown
Contributor

@cnpg-bot cnpg-bot added the ok to merge 👌 This PR can be merged label Apr 15, 2026
@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Apr 20, 2026
@mnencia
mnencia force-pushed the dev/containerready branch from 19cd925 to 5016f80 Compare April 21, 2026 12:46
@mnencia
mnencia requested a review from jsilvela as a code owner April 21, 2026 13:16
@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Apr 21, 2026
leonardoce and others added 4 commits April 22, 2026 10:33
When the node hosting the primary loses network connectivity, the
kubelet can no longer update the pod status. The pod's ContainersReady
condition is left stale as True, while Kubernetes actively sets Ready
to False via the node controller. By checking Ready instead of
ContainersReady, the operator can now correctly detect that the primary
is unavailable and proceed with a failover.

This change is safe in combination with the previous patch: when the
status endpoint fails transiently the pod's Ready condition is still
True, so the operator requeues rather than triggering a spurious
failover.

Signed-off-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Reflect that --node-monitor-grace-period defaults to 50s on Kubernetes
1.32+ and document that the kubelet isolation check stops the isolated
primary locally before taint-based eviction removes the pod.

Signed-off-by: Marco Nenciarini <[email protected]>
@mnencia
mnencia force-pushed the dev/containerready branch from 50218d1 to f224cca Compare April 22, 2026 08:34
- Refresh the PostgresqlStatus.IsPodReady field comment so it matches the
  new helper godoc (PodReady condition, driven by kubelet and the node
  lifecycle controller).
- Caveat the helper godoc's 30s figure with user overrides via
  `.spec.probes.readiness`.
- Cover the node-unreachable shape (ContainersReady=True + PodReady=False)
  at the controller test level: findDeletableInstance must skip such a
  Pod even when it has the highest serial.

Signed-off-by: Armando Ruocco <[email protected]>
@mnencia
mnencia merged commit 0083245 into cloudnative-pg:main Apr 22, 2026
35 checks passed
cnpg-bot pushed a commit that referenced this pull request Apr 22, 2026
When the node hosting the primary loses network connectivity, the
kubelet can no longer update the pod status. The pod's ContainersReady
condition is left stale as True, while Kubernetes actively sets Ready to
False via the node controller. By checking Ready instead of
ContainersReady, the operator can now correctly detect that the primary
is unavailable and proceed with a failover.

This change is safe in combination with the previous patch: when the
status endpoint fails transiently the pod's Ready condition is still
True, so the operator requeues rather than triggering a spurious
failover.

See #10403

Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
(cherry picked from commit 0083245)
cnpg-bot pushed a commit that referenced this pull request Apr 22, 2026
When the node hosting the primary loses network connectivity, the
kubelet can no longer update the pod status. The pod's ContainersReady
condition is left stale as True, while Kubernetes actively sets Ready to
False via the node controller. By checking Ready instead of
ContainersReady, the operator can now correctly detect that the primary
is unavailable and proceed with a failover.

This change is safe in combination with the previous patch: when the
status endpoint fails transiently the pod's Ready condition is still
True, so the operator requeues rather than triggering a spurious
failover.

See #10403

Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
(cherry picked from commit 0083245)
cnp-autobot pushed a commit to EnterpriseDB/cloudnative-pg that referenced this pull request Apr 23, 2026
…dnative-pg#10448)

When the node hosting the primary loses network connectivity, the
kubelet can no longer update the pod status. The pod's ContainersReady
condition is left stale as True, while Kubernetes actively sets Ready to
False via the node controller. By checking Ready instead of
ContainersReady, the operator can now correctly detect that the primary
is unavailable and proceed with a failover.

This change is safe in combination with the previous patch: when the
status endpoint fails transiently the pod's Ready condition is still
True, so the operator requeues rather than triggering a spurious
failover.

See cloudnative-pg#10403

Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Armando Ruocco <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Co-authored-by: Marco Nenciarini <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]>
(cherry picked from commit 0083245)
sdwilsh pushed a commit to sdwilsh/ansible-playbooks that referenced this pull request May 11, 2026
##### [\`v1.29.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.29.1)

**Release date:** May 8, 2026

##### Security and Supply Chain

- **`CVE-2026-44477` / `GHSA-423p-g724-fr39`: metrics exporter privilege escalation**: the metrics exporter no longer authenticates as the `postgres` superuser. It now uses a dedicated `cnpg_metrics_exporter` role with `pg_monitor` privileges only, closing a chain that let a low-privilege database user gain PostgreSQL superuser. ([`GHSA-423p-g724-fr39`](GHSA-423p-g724-fr39)) <!-- 1.29 1.28 1.25 -->

  Upgrade impact: custom monitoring queries that read user-owned tables, or use `target_databases: '*'` against databases where `PUBLIC CONNECT` has been revoked, need explicit `GRANT` statements to `cnpg_metrics_exporter`. See ["Custom query privileges and safety"](../monitoring.md#custom-query-privileges-and-safety) and ["Manually creating the metrics exporter role"](../monitoring.md#manually-creating-the-metrics-exporter-role) in the monitoring documentation.

  For replica clusters, upgrade the source primary cluster before any replica clusters that consume from it. The `cnpg_metrics_exporter` role is created on the source primary and replicates downstream; a replica cluster upgraded first will scrape against a missing role until the source primary upgrades. The manual-recovery section linked above also covers replica clusters.

- **Schema-qualified catalog references in default monitoring queries**: hardened the shipped monitoring configuration and documentation samples by qualifying every `pg_catalog` object explicitly. Unqualified references resolve through `search_path`, which a database user can manipulate to shadow built-in objects. ([#10576](cloudnative-pg/cloudnative-pg#10576)) <!-- 1.29 1.28 1.25 -->

- **Discoverable SBOM and provenance attestations**: SBOM and SLSA provenance attached to operator container images now follow the OCI 1.1 Referrers spec, so standard registry tooling and supply-chain scanners can discover them automatically. ([#10601](cloudnative-pg/cloudnative-pg#10601)) <!-- 1.29 1.28 1.25 -->

- **CVE remediation in `github.com/jackc/pgx/v5`**: bumped to v5.9.2 to pick up upstream fixes for `CVE-2026-33816` (memory-safety in `pgproto3`) and `GHSA-j88v-2chj-qfwx` (SQL injection via simple-protocol dollar-quoted string handling). ([#10437](cloudnative-pg/cloudnative-pg#10437), [#10499](cloudnative-pg/cloudnative-pg#10499))

- **CVE remediation in the Go runtime**: built with Go 1.26.3 to pick up upstream fixes in `crypto/x509`, `crypto/tls`, `net/http`, and `net` (CVE-2026-32280, CVE-2026-32281, CVE-2026-33810, CVE-2026-33814, CVE-2026-33811, CVE-2026-39825). ([#10463](cloudnative-pg/cloudnative-pg#10463), [#10647](cloudnative-pg/cloudnative-pg#10647)) <!-- 1.29 1.28 1.25 -->

- **Build pipeline hardening**: the Go 1.26.3 bump also addresses CVE-2026-42501 (`cmd/go` module-checksum validation), reducing supply-chain exposure during release builds. The affected code paths are not reachable from the running operator. ([#10647](cloudnative-pg/cloudnative-pg#10647)) <!-- 1.29 1.28 1.25 -->

##### Changes

- Switched TLS peer verification from `VerifyPeerCertificate` to `VerifyConnection`, which runs on every completed handshake (the former is skipped on resumed TLS 1.3 sessions). Session resumption is not enabled in CloudNativePG today, so this has no observable effect, but it future-proofs verification if session caching is introduced later. ([#10478](cloudnative-pg/cloudnative-pg#10478)) <!-- 1.29 1.28 1.25 -->

##### Fixes

- Fixed a failover window where the former primary kept its primary label. If it returned during failover (for example, after a transient network partition), the `-rw` service kept routing to it, replicas could reconnect, and committed writes were lost to `pg_rewind`. The old primary is now labeled `unhealthy` to isolate it from service traffic during failover. ([#10409](cloudnative-pg/cloudnative-pg#10409)) <!-- 1.29 1.28 1.25 -->

- Fixed failover not being triggered when the node hosting the primary becomes unreachable. The operator now reads the pod's `Ready` condition (flipped to `False` by the node controller when the kubelet stops reporting) instead of `ContainersReady`, which stays stale as `True` in that scenario. Combined with the spurious-failover guard ([#10445](cloudnative-pg/cloudnative-pg#10445)), failover triggers only when Kubernetes itself marks the pod not Ready. ([#10448](cloudnative-pg/cloudnative-pg#10448)) <!-- 1.29 1.28 1.25 -->

- Fixed spurious failovers caused by transient failures on the primary's HTTP status endpoint. ([#10445](cloudnative-pg/cloudnative-pg#10445)) <!-- 1.29 1.28 1.25 -->

- Fixed escaping of backslashes and control characters in PostgreSQL configuration values. Previously, such characters in parameters like `log_line_prefix` could corrupt the configuration file or be silently stripped at runtime. ([#10515](cloudnative-pg/cloudnative-pg#10515)) <!-- 1.29 1.28 1.25 -->

- Fixed `restore_command` construction to shell-quote each argument. Values such as a `destinationPath` containing whitespace (for example, `s3://my bucket/wal`) were word-split by the POSIX shell and passed to the WAL restore tool as separate arguments. ([#10518](cloudnative-pg/cloudnative-pg#10518)) <!-- 1.29 1.28 1.25 -->

- Tightened `recoveryTarget` validation in the admission webhook: `targetXID` must now be a non-negative 32-bit integer, and `targetName` must be shorter than 64 bytes and free of ASCII control characters. Malformed values are rejected at admission instead of failing later during PostgreSQL recovery. ([#10565](cloudnative-pg/cloudnative-pg#10565)) <!-- 1.29 1.28 1.25 -->

- Fixed snapshot restores failing when leftover `pgsql_tmp*` directories were present in the data directory. ([#10447](cloudnative-pg/cloudnative-pg#10447)) <!-- 1.29 1.28 1.25 -->

- Fixed a deadlock occurring when PVC storage size and resource requests are changed simultaneously. ([#10427](cloudnative-pg/cloudnative-pg#10427)) <!-- 1.29 1.28 1.25 -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-requested ◀️ This pull request should be backported to all supported releases bug 🐛 Something isn't working lgtm This PR has been approved by a maintainer ok to merge 👌 This PR can be merged release-1.28 size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants