Skip to content

Commit ec41c7c

Browse files
gbartolinileonardocejsilvelamnencia
committed
docs: document how to simulate a failover (#2911)
Warning against using `kubectl delete pod --grace-period=0`. Closes #2910 Signed-off-by: Gabriele Bartolini <[email protected]> Signed-off-by: Leonardo Cecchi <[email protected]> Co-authored-by: Leonardo Cecchi <[email protected]> Co-authored-by: Jaime Silvela <[email protected]> Co-authored-by: Marco Nenciarini <[email protected]> (cherry picked from commit 6a24e5d)
1 parent 5f8c7a3 commit ec41c7c

1 file changed

Lines changed: 28 additions & 0 deletions

File tree

‎docs/src/failure_modes.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,34 @@ or starting from a physical backup of the *primary* otherwise.
7272

7373
Self-healing will happen as soon as the *apiserver* is notified.
7474

75+
You can trigger a sudden failure on a given pod of the cluster using the
76+
following generic command:
77+
78+
```sh
79+
kubectl delete -n [namespace] \
80+
pod/[cluster-name]-[serial] --grace-period=1
81+
```
82+
83+
For example, if you want to simulate a real failure on the primary and trigger
84+
the failover process, you can run:
85+
86+
```sh
87+
kubectl delete pod [primary pod] --grace-period=1
88+
```
89+
90+
!!! Warning
91+
Never use `--grace-period=0` in your failover simulation tests, as this
92+
might produce misleading results with your PostgreSQL cluster. A grace
93+
period of 0 guarantees that the pod is immediately removed from the
94+
Kubernetes API server, without first ensuring that the PID 1 process of
95+
the `postgres` container (the instance manager) is shut down - contrary
96+
to what would happen in case of a real failure (e.g. unplug the power cord
97+
cable or network partitioning).
98+
As a result, the operator doesn't see the pod of the primary anymore, and
99+
triggers a failover promoting the most aligned standby, without
100+
the guarantee that the primary had been shut down.
101+
102+
75103
### Readiness probe failure
76104

77105
After 3 failures, the pod will be considered *not ready*. The pod will still

0 commit comments

Comments
 (0)