Skip to content

Commit 248ee34

Browse files
YanniHu1996litaocdlarmrugbartolini
committed
feat: change default value of stopDelay to 1800 (#2848)
IMPORTANT: The default value of `stopDelay` has now been changed to 1800 seconds, equivalent of half an hour. Previously, it was set to 30 seconds. This patch also introduces a new parameter, called `smartStopDelay`, to control the window of time reserved for the smart shutdown of Postgres to complete. The general formula to compute the overall timeout to stop Postgres is `max(stopDelay - smartStopDelay, 30)`. Closes #2844 Signed-off-by: YanniHu1996 <[email protected]> Signed-off-by: Tao Li <[email protected]> Signed-off-by: Armando Ruocco <[email protected]> Signed-off-by: Gabriele Bartolini <[email protected]> Co-authored-by: Tao Li <[email protected]> Co-authored-by: Armando Ruocco <[email protected]> Co-authored-by: Gabriele Bartolini <[email protected]>
1 parent 397b897 commit 248ee34

19 files changed

Lines changed: 96 additions & 32 deletions

‎.wordlist-en-custom.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -984,6 +984,7 @@ sig
984984
sigs
985985
singlenamespace
986986
slotPrefix
987+
smartStopDelay
987988
snapshotted
988989
snapshotting
989990
sourceNamespace

‎api/v1/cluster_types.go‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -237,11 +237,17 @@ type ClusterSpec struct {
237237
MaxStartDelay int32 `json:"startDelay,omitempty"`
238238

239239
// The time in seconds that is allowed for a PostgreSQL instance to
240-
// gracefully shutdown (default 30)
241-
// +kubebuilder:default:=30
240+
// gracefully shutdown (default 1800)
241+
// +kubebuilder:default:=1800
242242
// +optional
243243
MaxStopDelay int32 `json:"stopDelay,omitempty"`
244244

245+
// The time in seconds that controls the window of time reserved for the smart shutdown of Postgres to complete.
246+
// this formula to compute the timeout of smart shutdown is `max(stopDelay - smartStopDelay, 30)`,
247+
// +kubebuilder:default:=180
248+
// +optional
249+
SmartStopDelay int32 `json:"smartStopDelay,omitempty"`
250+
245251
// The time in seconds that is allowed for a primary PostgreSQL instance
246252
// to gracefully shutdown during a switchover.
247253
// Default value is 3600 seconds (1 hour).
@@ -2323,7 +2329,15 @@ func (cluster *Cluster) GetMaxStopDelay() int32 {
23232329
if cluster.Spec.MaxStopDelay > 0 {
23242330
return cluster.Spec.MaxStopDelay
23252331
}
2326-
return 30
2332+
return 1800
2333+
}
2334+
2335+
// GetSmartStopDelay is used to compute the timeout of smart shutdown by the formula `stopDelay - smartStopDelay`
2336+
func (cluster *Cluster) GetSmartStopDelay() int32 {
2337+
if cluster.Spec.SmartStopDelay > 0 {
2338+
return cluster.Spec.SmartStopDelay
2339+
}
2340+
return 180
23272341
}
23282342

23292343
// GetMaxSwitchoverDelay get the amount of time PostgreSQL has to stop before switchover

‎config/crd/bases/postgresql.cnpg.io_clusters.yaml‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2973,6 +2973,14 @@ spec:
29732973
required:
29742974
- metadata
29752975
type: object
2976+
smartStopDelay:
2977+
default: 180
2978+
description: The time in seconds that controls the window of time
2979+
reserved for the smart shutdown of Postgres to complete. this formula
2980+
to compute the timeout of smart shutdown is `max(stopDelay - smartStopDelay,
2981+
30)`,
2982+
format: int32
2983+
type: integer
29762984
startDelay:
29772985
default: 3600
29782986
description: 'The time in seconds that is allowed for a PostgreSQL
@@ -2982,9 +2990,9 @@ spec:
29822990
format: int32
29832991
type: integer
29842992
stopDelay:
2985-
default: 30
2993+
default: 1800
29862994
description: The time in seconds that is allowed for a PostgreSQL
2987-
instance to gracefully shutdown (default 30)
2995+
instance to gracefully shutdown (default 1800)
29882996
format: int32
29892997
type: integer
29902998
storage:

‎docs/src/cloudnative-pg.v1.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1432,7 +1432,15 @@ ceiling(startDelay / 10).</p>
14321432
</td>
14331433
<td>
14341434
<p>The time in seconds that is allowed for a PostgreSQL instance to
1435-
gracefully shutdown (default 30)</p>
1435+
gracefully shutdown (default 1800)</p>
1436+
</td>
1437+
</tr>
1438+
<tr><td><code>smartStopDelay</code><br/>
1439+
<i>int32</i>
1440+
</td>
1441+
<td>
1442+
<p>The time in seconds that controls the window of time reserved for the smart shutdown of Postgres to complete.
1443+
this formula to compute the timeout of smart shutdown is <code>max(stopDelay - smartStopDelay, 30)</code>,</p>
14361444
</td>
14371445
</tr>
14381446
<tr><td><code>switchoverDelay</code><br/>

‎docs/src/fencing.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -78,8 +78,8 @@ kubectl cnpg fencing off cluster-example "*"
7878

7979
Once an instance is set for fencing, the procedure to shut down the
8080
`postmaster` process is initiated. This consists of an initial smart shutdown
81-
with a timeout set to `.spec.stopDelay`, followed by a fast shutdown if
82-
required. Then:
81+
with a timeout set to `.spec.smartStopDelay`, followed by a fast shutdown if
82+
required for up to `.spec.stopDelay` seconds. Then:
8383

8484
- the Pod will be kept alive
8585

‎docs/src/instance_manager.md‎

Lines changed: 8 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -48,20 +48,22 @@ When a Pod running Postgres is deleted, either manually or by Kubernetes
4848
following a node drain operation, the kubelet will send a termination signal to the
4949
instance manager, and the instance manager will take care of shutting down
5050
PostgreSQL in an appropriate way.
51-
The `.spec.stopDelay`, expressed in seconds, is the amount of time
52-
given to PostgreSQL to shut down. The value defaults to 30 seconds.
51+
The `.spec.smartStopDelay` and `.spec.stopDelay` options, expressed in seconds,
52+
control the amount of time given to PostgreSQL to shut down. The values default
53+
to 180 and 1800 seconds, respectively.
5354

5455
The shutdown procedure is composed of two steps:
5556

5657
1. The instance manager requests a **smart** shut down, disallowing any
57-
new connection to PostgreSQL. This step will last for half of the
58-
time set in `.spec.stopDelay`.
58+
new connection to PostgreSQL. This step will last for up to
59+
`.spec.smartStopDelay` seconds.
5960

6061
2. If PostgreSQL is still up, the instance manager requests a **fast**
6162
shut down, terminating any existing connection and exiting promptly.
6263
If the instance is archiving and/or streaming WAL files, the process
63-
will wait for up to the remaining half of the time set in `.spec.stopDelay`
64-
to complete the operation and then forcibly shut down.
64+
will wait for up to the remaining time set in `.spec.stopDelay` to complete the
65+
operation and then forcibly shut down. Such timeout is calculated using the
66+
following formula: `max(stopDelay - smartStopDelay, 30)`.
6567

6668
!!! Important
6769
In order to avoid any data loss in the Postgres cluster, which impacts

‎internal/cmd/manager/instance/run/lifecycle/lifecycle.go‎

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -124,11 +124,9 @@ func (i *PostgresLifecycle) Start(ctx context.Context) error {
124124
// to our process. In this case we terminate as fast as we can,
125125
// otherwise we'll receive a SIGKILL by the Kubelet, possibly
126126
// resulting in a data corruption.
127-
//
128-
// This is why we are trying a smart shutdown for half-time
129-
// of our stop delay, and then we proceed.
130-
log.Info("Received termination signal", "signal", sig)
131-
if err := tryShuttingDownSmartFast(i.instance.MaxStopDelay/2, i.instance); err != nil {
127+
smartShutdownTimeout := i.instance.GetSmartShutdownTimeout()
128+
log.Info("Received termination signal", "signal", sig, "smartShutdownTimeout", smartShutdownTimeout)
129+
if err := tryShuttingDownSmartFast(smartShutdownTimeout, i.instance); err != nil {
132130
log.Error(err, "error while shutting down instance, proceeding")
133131
}
134132
return nil

‎internal/management/controller/instance_controller.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -881,6 +881,7 @@ func (r *InstanceReconciler) reconcileInstance(cluster *apiv1.Cluster) {
881881
r.instance.PgCtlTimeoutForPromotion = cluster.GetPgCtlTimeoutForPromotion()
882882
r.instance.MaxSwitchoverDelay = cluster.GetMaxSwitchoverDelay()
883883
r.instance.MaxStopDelay = cluster.GetMaxStopDelay()
884+
r.instance.SmartStopDelay = cluster.GetSmartStopDelay()
884885
}
885886

886887
func (r *InstanceReconciler) reconcileCheckWalArchiveFile(cluster *apiv1.Cluster) error {

‎pkg/management/postgres/instance.go‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,9 @@ type Instance struct {
167167
// MaxStopDelay is the current MaxStopDelay of the cluster
168168
MaxStopDelay int32
169169

170+
// SmartStopDelay is used to compute the timeout of smart shutdown by the formula `stopDelay - smartStopDelay`
171+
SmartStopDelay int32
172+
170173
// canCheckReadiness specifies whether the instance can start being checked for readiness
171174
// Is set to true before the instance is run and to false once it exits,
172175
// it's used by the readiness probe to know whether it should be short-circuited
@@ -257,6 +260,17 @@ func (instance *Instance) VerifyPgDataCoherence(ctx context.Context) error {
257260
return WritePostgresUserMaps(instance.PgData)
258261
}
259262

263+
// GetSmartShutdownTimeout gets the duration in seconds as the timeout of smart shutdown
264+
// we calculate smart shutdown with following formula
265+
// max(stopDelay - smartStopDelay, 30)
266+
func (instance *Instance) GetSmartShutdownTimeout() int32 {
267+
timeout := instance.MaxStopDelay - instance.SmartStopDelay
268+
if timeout < 30 {
269+
timeout = 30
270+
}
271+
return timeout
272+
}
273+
260274
// InstanceCommand are commands for the goroutine managing postgres
261275
type InstanceCommand string
262276

‎pkg/management/postgres/instance_test.go‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -224,3 +224,15 @@ var _ = Describe("check atomic bool", func() {
224224
Expect(unAvailable).To(BeTrue())
225225
})
226226
})
227+
228+
var _ = Describe("testing the compute of smart shutdown time", func() {
229+
It("should be 30 as the result of 'MaxStopDelay - SmartStopDelay' is less than it", func() {
230+
instance := Instance{MaxStopDelay: 100, SmartStopDelay: 100}
231+
Expect(instance.GetSmartShutdownTimeout()).To(BeNumerically("==", 30))
232+
})
233+
234+
It("should be the result of 'MaxStopDelay - SmartStopDelay'", func() {
235+
instance := Instance{MaxStopDelay: 200, SmartStopDelay: 100}
236+
Expect(instance.GetSmartShutdownTimeout()).To(BeNumerically("==", 100))
237+
})
238+
})

0 commit comments

Comments
 (0)