You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6e3da63
Browse filesBrowse the repository at this point in the historyBrowse files
fix(postgres): guard the disable-wedge, role-scope the freeze count, and harden edges
Address the remaining IvanHunters review points on #3954:
- [MAJOR] Disabling autoscaling wedged the HelmRelease under synchronous quorum.
An autoscaled + maxSyncReplicas>=1 cluster runs with instances seeded from
effectiveMin while replicas stays at its default, so flipping autoscaling off
rendered instances: <replicas> next to the quorum and CNPG's webhook rejected
it ("maxSyncReplicas must be lower than the number of instances"). Add a
symmetric render-time guard on the static-count path (covers off/dryRun/
transition): fail with an actionable message when replicas <= maxSyncReplicas.
- [MINOR] Role-scope the freeze brake's currentInstances count
(label_cnpg_io_instance_role=~".+") so lingering CNPG join/bootstrap Job pods
(cnpg.io/cluster but jobRole, not instanceRole) don't inflate it and nudge
desired up while the brake is meant to hold it.
- [MINOR] Default maxReplicationLagSeconds to 0 (brake opt-in): only the
non-braking path is validated live, so ship the validated default and let the
freeze be enabled explicitly until its branch is exercised under load.
- [MINOR] Document the floor-change and non-quorum-disable dips (same three-way-
merge hazard) and the pin-replicas-first mitigation.
- [NIT] Coerce the cozy-lib helper's `paused` (eq "true") so a caller passing the
string "false" does not pause the ScaledObject.
- [NIT] Trim the keda.clusterDomain injection comment so it no longer ships ~8
verbatim lines into the cozystack-values Secret on every cluster.
Regenerate types/schema/README/postgres-rd. New tests cover the disable-wedge
guard, the role-scoped freeze count, and the string-"false" paused coercion.
Signed-off-by: Alexey Artamonov <[email protected]>
Copy file name to clipboardExpand all lines: packages/apps/postgres/README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -250,7 +250,7 @@ For `sslmode=verify-full` to work, the CA bundle retrieved above must be saved t
250
250
| `autoscaling.maxReplicas` | Maximum total instances; raised to the quorum floor if the floor exceeds it. | `int` | `6` |
251
251
| `autoscaling.metric` | Read-load signal to scale on. | `string` | `ReadConnections` |
252
252
| `autoscaling.target` | Target read load per read-serving replica (unit depends on `metric`: active connections, or CPU millicores). Dimensionless integer - a resource.Quantity here would accept `150m`, which PromQL reads as 150 minutes and KEDA cannot use as a threshold. | `int` | `150` |
253
-
| `autoscaling.maxReplicationLagSeconds` | Freeze scaling while replication lag exceeds this and the primary is writing. | `int` | `30` |
253
+
| `autoscaling.maxReplicationLagSeconds` | Freeze scaling while replication lag exceeds this (seconds) and the primary is writing. 0 disables the brake. Default 0: the freeze branch has only been validated on a live cluster in its non-braking (pass-through) form, so the lag brake is opt-in until the braking path is exercised under load; set a positive value (e.g. 30) to enable it. | `int` | `0` |
254
254
| `autoscaling.dryRun` | Recommendation mode: keep the static count and render the ScaledObject paused (KEDA does not actuate). Permanent, unlike `transition`. | `bool` | `false` |
255
255
| `autoscaling.transition` | Migration phase 1: render `.spec.instances` from `replicas` (which the operator stages to the current live count) AND pause the ScaledObject, so KEDA stands up ready-but-not-actuating. Flip to false (phase 2) to hand the runtime count to KEDA. This shortens the handoff (KEDA is already warm, no cold-start) but does not make it atomic — see the enablement note for the residual dip and how to eliminate it. | `bool` | `false` |
Copy file name to clipboardExpand all lines: packages/apps/postgres/templates/db.yaml
+12Lines changed: 12 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -22,6 +22,18 @@ spec:
22
22
# subresource (design proposal §3). A brief cold-start window at 1 is expected
23
23
# until KEDA scales up to the floor. See templates/scaledobject.yaml.
24
24
{{- else }}
25
+
{{- /* When the chart declares the static instance count, CNPG's admission webhook
26
+
rejects the Cluster if maxSyncReplicas >= instances ("maxSyncReplicas must be
27
+
lower than the number of instances"). This bites the DISABLE path in
28
+
particular: an autoscaled + sync-quorum cluster runs with instances seeded
29
+
from effectiveMin while replicas stays at its default, so flipping autoscaling
30
+
off would render instances: <replicas> next to the quorum and wedge the
31
+
HelmRelease upgrade with an opaque webhook error. Fail closed with an
32
+
actionable message (symmetric to the enable-path guards). Also covers the
33
+
dryRun/transition modes, which likewise render the static count. */ -}}
34
+
{{- if le (int .Values.replicas) (int .Values.quorum.maxSyncReplicas) }}
35
+
{{- fail (printf "postgres: replicas (%d) must be greater than quorum.maxSyncReplicas (%d) - CNPG rejects a Cluster whose maxSyncReplicas is not lower than its instance count. When disabling autoscaling on a synchronous-quorum cluster, first raise replicas to at least the live instance count (kubectl scale --subresource=scale with the HelmRelease suspended)." (int .Values.replicas) (int .Values.quorum.maxSyncReplicas)) }}
# the frozen summand is floored with `or vector(0)` so a kube-state-metrics
294
296
# scrape gap (empty count) cannot empty the whole `+` sum and defeat the
295
297
# base term's own floor
@@ -340,7 +342,7 @@ tests:
340
342
maxReplicas: 6
341
343
target: 150
342
344
quorum:
343
-
maxSyncReplicas: 2
345
+
maxSyncReplicas: 1
344
346
asserts:
345
347
- equal:
346
348
path: spec.instances
@@ -354,6 +356,23 @@ tests:
354
356
value: "true"
355
357
template: templates/scaledobject.yaml
356
358
359
+
- it: fails closed when the static instance count would violate synchronous quorum (disable-wedge guard)
360
+
templates:
361
+
- templates/db.yaml
362
+
set:
363
+
# autoscaling off: the chart renders the static instances: <replicas>. With
364
+
# replicas at its default 2 and maxSyncReplicas 2, CNPG would reject the
365
+
# Cluster (maxSyncReplicas must be < instances). This is exactly the wedge a
366
+
# tenant hits when DISABLING autoscaling on a quorum cluster whose replicas
367
+
# was never raised off the default. Must fail closed at render, not at the webhook.
368
+
autoscaling:
369
+
enabled: false
370
+
quorum:
371
+
maxSyncReplicas: 2
372
+
asserts:
373
+
- failedTemplate:
374
+
errorMessage: "postgres: replicas (2) must be greater than quorum.maxSyncReplicas (2) - CNPG rejects a Cluster whose maxSyncReplicas is not lower than its instance count. When disabling autoscaling on a synchronous-quorum cluster, first raise replicas to at least the live instance count (kubectl scale --subresource=scale with the HelmRelease suspended)."
Copy file name to clipboardExpand all lines: packages/apps/postgres/values.schema.json
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -228,9 +228,9 @@
228
228
"default": 6
229
229
},
230
230
"maxReplicationLagSeconds": {
231
-
"description": "Freeze scaling while replication lag exceeds this and the primary is writing.",
231
+
"description": "Freeze scaling while replication lag exceeds this (seconds) and the primary is writing. 0 disables the brake. Default 0: the freeze branch has only been validated on a live cluster in its non-braking (pass-through) form, so the lag brake is opt-in until the braking path is exercised under load; set a positive value (e.g. 30) to enable it.",
Copy file name to clipboardExpand all lines: packages/apps/postgres/values.yaml
+11-2Lines changed: 11 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -150,7 +150,7 @@ quorum:
150
150
## @field {int} maxReplicas - Maximum total instances; raised to the quorum floor if the floor exceeds it.
151
151
## @field {ReadMetric} metric - Read-load signal to scale on.
152
152
## @field {int} target - Target read load per read-serving replica (unit depends on `metric`: active connections, or CPU millicores). Dimensionless integer - a resource.Quantity here would accept `150m`, which PromQL reads as 150 minutes and KEDA cannot use as a threshold.
153
-
## @field {int} maxReplicationLagSeconds - Freeze scaling while replication lag exceeds this and the primary is writing.
153
+
## @field {int} maxReplicationLagSeconds - Freeze scaling while replication lag exceeds this (seconds) and the primary is writing. 0 disables the brake. Default 0: the freeze branch has only been validated on a live cluster in its non-braking (pass-through) form, so the lag brake is opt-in until the braking path is exercised under load; set a positive value (e.g. 30) to enable it.
154
154
## @field {bool} dryRun - Recommendation mode: keep the static count and render the ScaledObject paused (KEDA does not actuate). Permanent, unlike `transition`.
155
155
## @field {bool} transition - Migration phase 1: render `.spec.instances` from `replicas` (which the operator stages to the current live count) AND pause the ScaledObject, so KEDA stands up ready-but-not-actuating. Flip to false (phase 2) to hand the runtime count to KEDA. This shortens the handoff (KEDA is already warm, no cold-start) but does not make it atomic — see the enablement note for the residual dip and how to eliminate it.
156
156
@@ -183,14 +183,23 @@ quorum:
183
183
## pin the live count with `kubectl scale --subresource=scale` (or set replicas on the
184
184
## CNPG Cluster) so KEDA/CNPG holds it, run the phase-2 apply, then resume. Absent that,
185
185
## expect a brief read-replica disruption during the handoff.
186
+
## The same hazard applies to two other live changes, for the same three-way-merge
187
+
## reason: (1) raising `minReplicas` or `quorum.maxSyncReplicas` on a running autoscaled
188
+
## cluster re-renders the seeded `instances` to the new, higher effectiveMin — harmless
189
+
## (scale-up) — but LOWERING either re-renders a smaller effectiveMin, so if KEDA had
190
+
## scaled above it CNPG decommissions the surplus instances (deleting their PVCs) before
191
+
## the HPA re-clones them; and (2) in the non-quorum case, disabling autoscaling snaps
192
+
## `instances` from the live count back to `.Values.replicas`. Before a floor change or a
193
+
## disable, pin `replicas` to the current live instance count (`kubectl scale
194
+
## --subresource=scale`, HelmRelease suspended) so the re-render is a no-op.
0 commit comments