Repository navigation
fix: trigger failover when the primary node becomes unreachable - #10448
Merged
Merged
Conversation
Contributor
|
❗ By default, the pull request is configured to backport to all release branches.
|
Contributor
Author
|
/test |
Contributor
|
@leonardoce, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/24456492900 |
armru
approved these changes
Apr 20, 2026
mnencia
force-pushed
the
dev/containerready
branch
from
April 21, 2026 12:46
19cd925 to
5016f80
Compare
mnencia
approved these changes
Apr 21, 2026
mnencia
force-pushed
the
dev/containerready
branch
from
April 21, 2026 16:26
5b84ea0 to
58f7f4d
Compare
mnencia
force-pushed
the
dev/containerready
branch
from
April 22, 2026 08:11
58f7f4d to
50218d1
Compare
When the node hosting the primary loses network connectivity, the kubelet can no longer update the pod status. The pod's ContainersReady condition is left stale as True, while Kubernetes actively sets Ready to False via the node controller. By checking Ready instead of ContainersReady, the operator can now correctly detect that the primary is unavailable and proceed with a failover. This change is safe in combination with the previous patch: when the status endpoint fails transiently the pod's Ready condition is still True, so the operator requeues rather than triggering a spurious failover. Signed-off-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Armando Ruocco <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Signed-off-by: Marco Nenciarini <[email protected]>
Signed-off-by: Marco Nenciarini <[email protected]>
Reflect that --node-monitor-grace-period defaults to 50s on Kubernetes 1.32+ and document that the kubelet isolation check stops the isolated primary locally before taint-based eviction removes the pod. Signed-off-by: Marco Nenciarini <[email protected]>
mnencia
force-pushed
the
dev/containerready
branch
from
April 22, 2026 08:34
50218d1 to
f224cca
Compare
- Refresh the PostgresqlStatus.IsPodReady field comment so it matches the new helper godoc (PodReady condition, driven by kubelet and the node lifecycle controller). - Caveat the helper godoc's 30s figure with user overrides via `.spec.probes.readiness`. - Cover the node-unreachable shape (ContainersReady=True + PodReady=False) at the controller test level: findDeletableInstance must skip such a Pod even when it has the highest serial. Signed-off-by: Armando Ruocco <[email protected]>
cnpg-bot
pushed a commit
that referenced
this pull request
Apr 22, 2026
When the node hosting the primary loses network connectivity, the kubelet can no longer update the pod status. The pod's ContainersReady condition is left stale as True, while Kubernetes actively sets Ready to False via the node controller. By checking Ready instead of ContainersReady, the operator can now correctly detect that the primary is unavailable and proceed with a failover. This change is safe in combination with the previous patch: when the status endpoint fails transiently the pod's Ready condition is still True, so the operator requeues rather than triggering a spurious failover. See #10403 Signed-off-by: Leonardo Cecchi <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Signed-off-by: Marco Nenciarini <[email protected]> Co-authored-by: Marco Nenciarini <[email protected]> Co-authored-by: Armando Ruocco <[email protected]> (cherry picked from commit 0083245)
cnpg-bot
pushed a commit
that referenced
this pull request
Apr 22, 2026
When the node hosting the primary loses network connectivity, the kubelet can no longer update the pod status. The pod's ContainersReady condition is left stale as True, while Kubernetes actively sets Ready to False via the node controller. By checking Ready instead of ContainersReady, the operator can now correctly detect that the primary is unavailable and proceed with a failover. This change is safe in combination with the previous patch: when the status endpoint fails transiently the pod's Ready condition is still True, so the operator requeues rather than triggering a spurious failover. See #10403 Signed-off-by: Leonardo Cecchi <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Signed-off-by: Marco Nenciarini <[email protected]> Co-authored-by: Marco Nenciarini <[email protected]> Co-authored-by: Armando Ruocco <[email protected]> (cherry picked from commit 0083245)
cnp-autobot
pushed a commit
to EnterpriseDB/cloudnative-pg
that referenced
this pull request
Apr 23, 2026
…dnative-pg#10448) When the node hosting the primary loses network connectivity, the kubelet can no longer update the pod status. The pod's ContainersReady condition is left stale as True, while Kubernetes actively sets Ready to False via the node controller. By checking Ready instead of ContainersReady, the operator can now correctly detect that the primary is unavailable and proceed with a failover. This change is safe in combination with the previous patch: when the status endpoint fails transiently the pod's Ready condition is still True, so the operator requeues rather than triggering a spurious failover. See cloudnative-pg#10403 Signed-off-by: Leonardo Cecchi <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Signed-off-by: Marco Nenciarini <[email protected]> Co-authored-by: Marco Nenciarini <[email protected]> Co-authored-by: Armando Ruocco <[email protected]> (cherry picked from commit 0083245)
sdwilsh
pushed a commit
to sdwilsh/ansible-playbooks
that referenced
this pull request
May 11, 2026
##### [\`v1.29.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.29.1) **Release date:** May 8, 2026 ##### Security and Supply Chain - **`CVE-2026-44477` / `GHSA-423p-g724-fr39`: metrics exporter privilege escalation**: the metrics exporter no longer authenticates as the `postgres` superuser. It now uses a dedicated `cnpg_metrics_exporter` role with `pg_monitor` privileges only, closing a chain that let a low-privilege database user gain PostgreSQL superuser. ([`GHSA-423p-g724-fr39`](GHSA-423p-g724-fr39)) <!-- 1.29 1.28 1.25 --> Upgrade impact: custom monitoring queries that read user-owned tables, or use `target_databases: '*'` against databases where `PUBLIC CONNECT` has been revoked, need explicit `GRANT` statements to `cnpg_metrics_exporter`. See ["Custom query privileges and safety"](../monitoring.md#custom-query-privileges-and-safety) and ["Manually creating the metrics exporter role"](../monitoring.md#manually-creating-the-metrics-exporter-role) in the monitoring documentation. For replica clusters, upgrade the source primary cluster before any replica clusters that consume from it. The `cnpg_metrics_exporter` role is created on the source primary and replicates downstream; a replica cluster upgraded first will scrape against a missing role until the source primary upgrades. The manual-recovery section linked above also covers replica clusters. - **Schema-qualified catalog references in default monitoring queries**: hardened the shipped monitoring configuration and documentation samples by qualifying every `pg_catalog` object explicitly. Unqualified references resolve through `search_path`, which a database user can manipulate to shadow built-in objects. ([#10576](cloudnative-pg/cloudnative-pg#10576)) <!-- 1.29 1.28 1.25 --> - **Discoverable SBOM and provenance attestations**: SBOM and SLSA provenance attached to operator container images now follow the OCI 1.1 Referrers spec, so standard registry tooling and supply-chain scanners can discover them automatically. ([#10601](cloudnative-pg/cloudnative-pg#10601)) <!-- 1.29 1.28 1.25 --> - **CVE remediation in `github.com/jackc/pgx/v5`**: bumped to v5.9.2 to pick up upstream fixes for `CVE-2026-33816` (memory-safety in `pgproto3`) and `GHSA-j88v-2chj-qfwx` (SQL injection via simple-protocol dollar-quoted string handling). ([#10437](cloudnative-pg/cloudnative-pg#10437), [#10499](cloudnative-pg/cloudnative-pg#10499)) - **CVE remediation in the Go runtime**: built with Go 1.26.3 to pick up upstream fixes in `crypto/x509`, `crypto/tls`, `net/http`, and `net` (CVE-2026-32280, CVE-2026-32281, CVE-2026-33810, CVE-2026-33814, CVE-2026-33811, CVE-2026-39825). ([#10463](cloudnative-pg/cloudnative-pg#10463), [#10647](cloudnative-pg/cloudnative-pg#10647)) <!-- 1.29 1.28 1.25 --> - **Build pipeline hardening**: the Go 1.26.3 bump also addresses CVE-2026-42501 (`cmd/go` module-checksum validation), reducing supply-chain exposure during release builds. The affected code paths are not reachable from the running operator. ([#10647](cloudnative-pg/cloudnative-pg#10647)) <!-- 1.29 1.28 1.25 --> ##### Changes - Switched TLS peer verification from `VerifyPeerCertificate` to `VerifyConnection`, which runs on every completed handshake (the former is skipped on resumed TLS 1.3 sessions). Session resumption is not enabled in CloudNativePG today, so this has no observable effect, but it future-proofs verification if session caching is introduced later. ([#10478](cloudnative-pg/cloudnative-pg#10478)) <!-- 1.29 1.28 1.25 --> ##### Fixes - Fixed a failover window where the former primary kept its primary label. If it returned during failover (for example, after a transient network partition), the `-rw` service kept routing to it, replicas could reconnect, and committed writes were lost to `pg_rewind`. The old primary is now labeled `unhealthy` to isolate it from service traffic during failover. ([#10409](cloudnative-pg/cloudnative-pg#10409)) <!-- 1.29 1.28 1.25 --> - Fixed failover not being triggered when the node hosting the primary becomes unreachable. The operator now reads the pod's `Ready` condition (flipped to `False` by the node controller when the kubelet stops reporting) instead of `ContainersReady`, which stays stale as `True` in that scenario. Combined with the spurious-failover guard ([#10445](cloudnative-pg/cloudnative-pg#10445)), failover triggers only when Kubernetes itself marks the pod not Ready. ([#10448](cloudnative-pg/cloudnative-pg#10448)) <!-- 1.29 1.28 1.25 --> - Fixed spurious failovers caused by transient failures on the primary's HTTP status endpoint. ([#10445](cloudnative-pg/cloudnative-pg#10445)) <!-- 1.29 1.28 1.25 --> - Fixed escaping of backslashes and control characters in PostgreSQL configuration values. Previously, such characters in parameters like `log_line_prefix` could corrupt the configuration file or be silently stripped at runtime. ([#10515](cloudnative-pg/cloudnative-pg#10515)) <!-- 1.29 1.28 1.25 --> - Fixed `restore_command` construction to shell-quote each argument. Values such as a `destinationPath` containing whitespace (for example, `s3://my bucket/wal`) were word-split by the POSIX shell and passed to the WAL restore tool as separate arguments. ([#10518](cloudnative-pg/cloudnative-pg#10518)) <!-- 1.29 1.28 1.25 --> - Tightened `recoveryTarget` validation in the admission webhook: `targetXID` must now be a non-negative 32-bit integer, and `targetName` must be shorter than 64 bytes and free of ASCII control characters. Malformed values are rejected at admission instead of failing later during PostgreSQL recovery. ([#10565](cloudnative-pg/cloudnative-pg#10565)) <!-- 1.29 1.28 1.25 --> - Fixed snapshot restores failing when leftover `pgsql_tmp*` directories were present in the data directory. ([#10447](cloudnative-pg/cloudnative-pg#10447)) <!-- 1.29 1.28 1.25 --> - Fixed a deadlock occurring when PVC storage size and resource requests are changed simultaneously. ([#10427](cloudnative-pg/cloudnative-pg#10427)) <!-- 1.29 1.28 1.25 -->
1 task
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When the node hosting the primary loses network connectivity, the kubelet can no longer update the pod status. The pod's ContainersReady condition is left stale as True, while Kubernetes actively sets Ready to False via the node controller. By checking Ready instead of ContainersReady, the operator can now correctly detect that the primary is unavailable and proceed with a failover.
This change is safe in combination with the previous patch: when the status endpoint fails transiently the pod's Ready condition is still True, so the operator requeues rather than triggering a spurious failover.
See #10403