Skip to content

Commit 0083245

Browse files
leonardocemnenciaarmru
authored
fix: trigger failover when the primary node becomes unreachable (#10448)
When the node hosting the primary loses network connectivity, the kubelet can no longer update the pod status. The pod's ContainersReady condition is left stale as True, while Kubernetes actively sets Ready to False via the node controller. By checking Ready instead of ContainersReady, the operator can now correctly detect that the primary is unavailable and proceed with a failover. This change is safe in combination with the previous patch: when the status endpoint fails transiently the pod's Ready condition is still True, so the operator requeues rather than triggering a spurious failover. See #10403 Signed-off-by: Leonardo Cecchi <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Signed-off-by: Marco Nenciarini <[email protected]> Co-authored-by: Marco Nenciarini <[email protected]> Co-authored-by: Armando Ruocco <[email protected]>
1 parent 9309106 commit 0083245

8 files changed

Lines changed: 141 additions & 22 deletions

File tree

‎api/v1/base_funcs_test.go‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ var _ = Describe("Properly builds ListStatusPods", func() {
7171
Status: corev1.PodStatus{
7272
Conditions: []corev1.PodCondition{
7373
{
74-
Type: corev1.ContainersReady,
74+
Type: corev1.PodReady,
7575
Status: corev1.ConditionTrue,
7676
},
7777
},
@@ -85,7 +85,7 @@ var _ = Describe("Properly builds ListStatusPods", func() {
8585
Phase: corev1.PodRunning,
8686
Conditions: []corev1.PodCondition{
8787
{
88-
Type: corev1.ContainersReady,
88+
Type: corev1.PodReady,
8989
Status: corev1.ConditionFalse,
9090
},
9191
},
@@ -99,7 +99,7 @@ var _ = Describe("Properly builds ListStatusPods", func() {
9999
Phase: corev1.PodFailed,
100100
Conditions: []corev1.PodCondition{
101101
{
102-
Type: corev1.ContainersReady,
102+
Type: corev1.PodReady,
103103
Status: corev1.ConditionFalse,
104104
},
105105
},
@@ -116,7 +116,7 @@ var _ = Describe("Properly builds ListStatusPods", func() {
116116
Phase: corev1.PodRunning,
117117
Conditions: []corev1.PodCondition{
118118
{
119-
Type: corev1.ContainersReady,
119+
Type: corev1.PodReady,
120120
Status: corev1.ConditionTrue,
121121
},
122122
},

‎docs/src/failover.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,49 @@ expected outage.
103103
Enabling a new configuration option to delay failover provides a mechanism to
104104
prevent premature failover for short-lived network or node instability.
105105

106+
## Detection of node-level failures
107+
108+
When the node hosting the primary becomes unreachable (for example, due to a
109+
kubelet crash or a network partition between the node and the Kubernetes API
110+
server), the operator relies on the pod's `Ready` condition to decide that the
111+
primary is no longer serviceable. While the node is healthy the kubelet keeps
112+
that condition up to date from the readiness probe; once the node stops
113+
reporting, the Kubernetes node lifecycle controller is the one that flips the
114+
condition to `False` as soon as it declares the node `Unknown`.
115+
116+
With stock kube-controller-manager settings, the transition is governed by
117+
`--node-monitor-grace-period` (default `40s` on Kubernetes 1.29-1.31, raised
118+
to `50s` in 1.32 and later): after that window the controller marks the node
119+
`Unknown` and, in the same monitoring pass, issues a patch per pod on that
120+
node to flip the `Ready` condition. In practice the operator observes the
121+
primary as unready about **40 to 55 seconds** after the node becomes
122+
unreachable (the grace period plus up to one `--node-monitor-period` poll,
123+
default `5s`). Managed Kubernetes distributions (GKE, EKS, AKS) may tune
124+
these values; consult the provider's documentation if the observed timing
125+
does not match. After that, the failover procedure starts (further gated by
126+
`.spec.failoverDelay`).
127+
128+
The `Ready` condition flip is not subject to the rate limiters that throttle
129+
pod *eviction* during partial-zonal or large-cluster disruptions
130+
(`--node-eviction-rate`, `--secondary-node-eviction-rate`,
131+
`--unhealthy-zone-threshold`). The operator reacts to the condition flip as
132+
soon as the controller emits the patch, regardless of the zone or cluster-wide
133+
health state.
134+
135+
Pod *eviction* (actual deletion from the unreachable node) is a separate
136+
mechanism, driven by `tolerationSeconds` on the
137+
`node.kubernetes.io/unreachable` `NoExecute` taint (`300s` by default). That
138+
timer does not hold up the operator's failover decision; CloudNativePG
139+
promotes a new primary as soon as the `Ready` condition flips. By that point
140+
the kubelet on the isolated node has already stopped the old PostgreSQL
141+
container locally: with the default
142+
`.spec.probes.liveness.isolationCheck.enabled: true`, the instance manager
143+
fails its own liveness probe once it can reach neither the API server nor
144+
the rest of the cluster, and the kubelet kills the container within
145+
approximately three probe periods (`~30s`). Full high availability
146+
(recreation of the old primary on a healthy node by the operator) is still
147+
gated on the taint-based eviction actually deleting the pod.
148+
106149
## Failover Quorum (Quorum-based Failover)
107150

108151
Failover quorum is a mechanism that enhances data durability and safety during

‎internal/controller/replicas_test.go‎

Lines changed: 34 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ var _ = Describe("Sacrificial Pod detection", func() {
5050
Status: corev1.PodStatus{
5151
Conditions: []corev1.PodCondition{
5252
{
53-
Type: corev1.ContainersReady,
53+
Type: corev1.PodReady,
5454
Status: corev1.ConditionTrue,
5555
},
5656
},
@@ -68,7 +68,7 @@ var _ = Describe("Sacrificial Pod detection", func() {
6868
Status: corev1.PodStatus{
6969
Conditions: []corev1.PodCondition{
7070
{
71-
Type: corev1.ContainersReady,
71+
Type: corev1.PodReady,
7272
Status: corev1.ConditionTrue,
7373
},
7474
},
@@ -86,7 +86,7 @@ var _ = Describe("Sacrificial Pod detection", func() {
8686
Status: corev1.PodStatus{
8787
Conditions: []corev1.PodCondition{
8888
{
89-
Type: corev1.ContainersReady,
89+
Type: corev1.PodReady,
9090
Status: corev1.ConditionFalse,
9191
},
9292
},
@@ -101,10 +101,35 @@ var _ = Describe("Sacrificial Pod detection", func() {
101101
utils.ClusterSerialAnnotationName: "4",
102102
},
103103
},
104+
Status: corev1.PodStatus{
105+
Conditions: []corev1.PodCondition{
106+
{
107+
Type: corev1.PodReady,
108+
Status: corev1.ConditionFalse,
109+
},
110+
},
111+
},
112+
}
113+
114+
// unreachable mimics a Pod whose node has stopped reporting to the API
115+
// server: the node lifecycle controller flips PodReady to False, but the
116+
// stale ContainersReady left by the kubelet remains True.
117+
unreachable := corev1.Pod{
118+
ObjectMeta: metav1.ObjectMeta{
119+
Name: "unreachable",
120+
Namespace: "default",
121+
Annotations: map[string]string{
122+
utils.ClusterSerialAnnotationName: "5",
123+
},
124+
},
104125
Status: corev1.PodStatus{
105126
Conditions: []corev1.PodCondition{
106127
{
107128
Type: corev1.ContainersReady,
129+
Status: corev1.ConditionTrue,
130+
},
131+
{
132+
Type: corev1.PodReady,
108133
Status: corev1.ConditionFalse,
109134
},
110135
},
@@ -134,6 +159,12 @@ var _ = Describe("Sacrificial Pod detection", func() {
134159
Expect(resultName).ToNot(BeEmpty())
135160
Expect(resultName).To(Equal("car-2"))
136161
})
162+
163+
It("skips a Pod whose node is unreachable even if it has the highest serial", func() {
164+
podList := []corev1.Pod{car1, car2, unreachable}
165+
resultName := findDeletableInstance(&apiv1.Cluster{}, podList)
166+
Expect(resultName).To(Equal("car-2"))
167+
})
137168
})
138169

139170
var _ = Describe("markOldPrimaryAsUnhealthy", func() {

‎internal/controller/suite_test.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -333,7 +333,7 @@ func generateFakeClusterPods(
333333
Phase: corev1.PodRunning,
334334
Conditions: []corev1.PodCondition{
335335
{
336-
Type: corev1.ContainersReady,
336+
Type: corev1.PodReady,
337337
Status: corev1.ConditionTrue,
338338
},
339339
},

‎pkg/postgres/status.go‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -92,12 +92,14 @@ type PostgresqlStatus struct {
9292
// allowing detection of restarts that don't change the container ID or executable hash.
9393
SessionID string `json:"sessionID"`
9494

95-
// This field represents the Kubelet point-of-view of the readiness
96-
// status of this instance and may be slightly stale when the Kubelet has
97-
// not still invoked the readiness probe.
95+
// IsPodReady mirrors the Pod's PodReady condition, as set by the kubelet
96+
// while the node is healthy and by the node lifecycle controller once the
97+
// node stops reporting. It is a readiness signal driven by the readiness
98+
// probe plus any readiness gates, so it may be briefly stale compared to
99+
// the actual PostgreSQL state.
98100
//
99-
// If you want to check the latest detected status of PostgreSQL, you
100-
// need to call HasHTTPStatus().
101+
// For the latest PostgreSQL health as seen by the instance manager, use
102+
// HasHTTPStatus() instead.
101103
//
102104
// This field is never populated in the instance manager.
103105
IsPodReady bool `json:"isPodReady"`

‎pkg/reconciler/backup/volumesnapshot/offline_test.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ var _ = Describe("offlineExecutor", func() {
5353
Status: corev1.PodStatus{
5454
Conditions: []corev1.PodCondition{
5555
{
56-
Type: corev1.ContainersReady,
56+
Type: corev1.PodReady,
5757
Status: corev1.ConditionTrue,
5858
},
5959
},

‎pkg/utils/pod_conditions.go‎

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,10 +26,35 @@ import (
2626

2727
var utilsLog = log.WithName("utils")
2828

29-
// IsPodReady check if a Pod is ready or not
29+
// IsPodReady reports whether the Pod has its PodReady condition set to True.
30+
//
31+
// The PodReady condition is kept up-to-date by two independent control-plane
32+
// actors:
33+
//
34+
// - the kubelet on the pod's node, while the node is healthy: the transition
35+
// from True to False happens within FailureThreshold consecutive probe
36+
// periods of the underlying container becoming unhealthy (about 30s with
37+
// stock defaults: PeriodSeconds=10, FailureThreshold=3, unless the user
38+
// overrides them via `.spec.probes.readiness`).
39+
// - the node lifecycle controller when the node stops reporting to the API
40+
// server; once the node transitions to `Unknown` (after
41+
// `--node-monitor-grace-period`, 40s by default on Kubernetes 1.29-1.31
42+
// and 50s on 1.32+) the controller calls `MarkPodsNotReady`, which flips
43+
// PodReady to False on every pod of that node. With stock defaults the
44+
// operator observes the flip about 40 to 55 seconds after the node
45+
// becomes unreachable.
46+
//
47+
// Note that `tolerationSeconds` on the `node.kubernetes.io/unreachable` taint
48+
// (300s by default) controls pod eviction, not this condition: PodReady has
49+
// already flipped to False well before eviction happens.
50+
//
51+
// This helper is the source of truth for "is this pod serviceable right now?"
52+
// across the operator: failover election, backup target selection,
53+
// failover-quorum reachability, offline volume-snapshot fencing, and
54+
// user-visible cluster health classification.
3055
func IsPodReady(pod corev1.Pod) bool {
3156
for _, c := range pod.Status.Conditions {
32-
if c.Type == corev1.ContainersReady && c.Status == corev1.ConditionTrue {
57+
if c.Type == corev1.PodReady && c.Status == corev1.ConditionTrue {
3358
return true
3459
}
3560
}

‎pkg/utils/pod_conditions_test.go‎

Lines changed: 24 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ var _ = Describe("Pod conditions test suite", func() {
4343
Status: corev1.PodStatus{
4444
Conditions: []corev1.PodCondition{
4545
{
46-
Type: corev1.ContainersReady,
46+
Type: corev1.PodReady,
4747
Status: corev1.ConditionTrue,
4848
},
4949
},
@@ -53,11 +53,29 @@ var _ = Describe("Pod conditions test suite", func() {
5353
})
5454

5555
It("Detects not ready PODs are not running", func() {
56+
pod := corev1.Pod{
57+
Status: corev1.PodStatus{
58+
Conditions: []corev1.PodCondition{
59+
{
60+
Type: corev1.PodReady,
61+
Status: corev1.ConditionFalse,
62+
},
63+
},
64+
},
65+
}
66+
Expect(IsPodReady(pod)).To(BeFalse())
67+
})
68+
69+
It("Returns false when ContainersReady is True but PodReady is False", func() {
5670
pod := corev1.Pod{
5771
Status: corev1.PodStatus{
5872
Conditions: []corev1.PodCondition{
5973
{
6074
Type: corev1.ContainersReady,
75+
Status: corev1.ConditionTrue,
76+
},
77+
{
78+
Type: corev1.PodReady,
6179
Status: corev1.ConditionFalse,
6280
},
6381
},
@@ -80,7 +98,7 @@ var _ = Describe("Pod conditions test suite", func() {
8098
Status: corev1.PodStatus{
8199
Conditions: []corev1.PodCondition{
82100
{
83-
Type: corev1.ContainersReady,
101+
Type: corev1.PodReady,
84102
Status: corev1.ConditionTrue,
85103
},
86104
},
@@ -95,7 +113,7 @@ var _ = Describe("Pod conditions test suite", func() {
95113
Status: corev1.PodStatus{
96114
Conditions: []corev1.PodCondition{
97115
{
98-
Type: corev1.ContainersReady,
116+
Type: corev1.PodReady,
99117
Status: corev1.ConditionTrue,
100118
},
101119
},
@@ -110,7 +128,7 @@ var _ = Describe("Pod conditions test suite", func() {
110128
Status: corev1.PodStatus{
111129
Conditions: []corev1.PodCondition{
112130
{
113-
Type: corev1.ContainersReady,
131+
Type: corev1.PodReady,
114132
Status: corev1.ConditionFalse,
115133
},
116134
},
@@ -125,7 +143,7 @@ var _ = Describe("Pod conditions test suite", func() {
125143
Status: corev1.PodStatus{
126144
Conditions: []corev1.PodCondition{
127145
{
128-
Type: corev1.ContainersReady,
146+
Type: corev1.PodReady,
129147
Status: corev1.ConditionFalse,
130148
},
131149
},
@@ -157,7 +175,7 @@ var _ = Describe("Pod conditions test suite", func() {
157175
Phase: corev1.PodRunning,
158176
Conditions: []corev1.PodCondition{
159177
{
160-
Type: corev1.ContainersReady,
178+
Type: corev1.PodReady,
161179
Status: corev1.ConditionTrue,
162180
},
163181
},

0 commit comments

Comments
 (0)