Skip to content

Commit 6c14dc3

Browse files
gbartolinileonardoce
authored andcommitted
docs: document how to simulate a failover
Warning against using `kubectl delete pod --grace-period=0`. Closes #2910 Signed-off-by: Gabriele Bartolini <[email protected]>
1 parent 649de3c commit 6c14dc3

1 file changed

Lines changed: 27 additions & 0 deletions

File tree

‎docs/src/failure_modes.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,33 @@ or starting from a physical backup of the *primary* otherwise.
7272

7373
Self-healing will happen as soon as the *apiserver* is notified.
7474

75+
You can trigger a sudden failure on a given pod of the cluster using the
76+
following generic command:
77+
78+
```sh
79+
kubectl delete -n [namespace] \
80+
pod/[cluster-name]-[serial] --grace-period=1
81+
```
82+
83+
For example, if you want to simulate a real failure on the primary and trigger
84+
the failover process, you can run:
85+
86+
```sh
87+
kubectl delete pod [primary pod] --grace-period=1
88+
```
89+
90+
!!! Warning
91+
Never use `--grace-period=0` in your failover simulation tests, as this
92+
might produce misleading results with your PostgreSQL cluster. A grace
93+
period of 0 guarantees that the pod is immediately removed from the
94+
Kubernetes API server, without guaranteeing that the PID 1 process of
95+
the `postgres` container (the instance manager) is shut down - contrary
96+
to what would happen in case of a real failure (e.g. unplug the power cord cable).
97+
As a result, the operator doesn't see the pod of the primary anymore, and
98+
triggers an automated failover promoting the most aligned standby without
99+
the guarantee that the primary had been shut down.
100+
101+
75102
### Readiness probe failure
76103

77104
After 3 failures, the pod will be considered *not ready*. The pod will still

0 commit comments

Comments
 (0)