Skip to content

Commit 6e3da63

Browse files
committed
fix(postgres): guard the disable-wedge, role-scope the freeze count, and harden edges
Address the remaining IvanHunters review points on #3954: - [MAJOR] Disabling autoscaling wedged the HelmRelease under synchronous quorum. An autoscaled + maxSyncReplicas>=1 cluster runs with instances seeded from effectiveMin while replicas stays at its default, so flipping autoscaling off rendered instances: <replicas> next to the quorum and CNPG's webhook rejected it ("maxSyncReplicas must be lower than the number of instances"). Add a symmetric render-time guard on the static-count path (covers off/dryRun/ transition): fail with an actionable message when replicas <= maxSyncReplicas. - [MINOR] Role-scope the freeze brake's currentInstances count (label_cnpg_io_instance_role=~".+") so lingering CNPG join/bootstrap Job pods (cnpg.io/cluster but jobRole, not instanceRole) don't inflate it and nudge desired up while the brake is meant to hold it. - [MINOR] Default maxReplicationLagSeconds to 0 (brake opt-in): only the non-braking path is validated live, so ship the validated default and let the freeze be enabled explicitly until its branch is exercised under load. - [MINOR] Document the floor-change and non-quorum-disable dips (same three-way- merge hazard) and the pin-replicas-first mitigation. - [NIT] Coerce the cozy-lib helper's `paused` (eq "true") so a caller passing the string "false" does not pause the ScaledObject. - [NIT] Trim the keda.clusterDomain injection comment so it no longer ships ~8 verbatim lines into the cozystack-values Secret on every cluster. Regenerate types/schema/README/postgres-rd. New tests cover the disable-wedge guard, the role-scoped freeze count, and the string-"false" paused coercion. Signed-off-by: Alexey Artamonov <[email protected]>
1 parent e8609ac commit 6e3da63

11 files changed

Lines changed: 76 additions & 22 deletions

File tree

‎api/apps/v1alpha1/postgresql/types.go‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/postgres/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -250,7 +250,7 @@ For `sslmode=verify-full` to work, the CA bundle retrieved above must be saved t
250250
| `autoscaling.maxReplicas` | Maximum total instances; raised to the quorum floor if the floor exceeds it. | `int` | `6` |
251251
| `autoscaling.metric` | Read-load signal to scale on. | `string` | `ReadConnections` |
252252
| `autoscaling.target` | Target read load per read-serving replica (unit depends on `metric`: active connections, or CPU millicores). Dimensionless integer - a resource.Quantity here would accept `150m`, which PromQL reads as 150 minutes and KEDA cannot use as a threshold. | `int` | `150` |
253-
| `autoscaling.maxReplicationLagSeconds` | Freeze scaling while replication lag exceeds this and the primary is writing. | `int` | `30` |
253+
| `autoscaling.maxReplicationLagSeconds` | Freeze scaling while replication lag exceeds this (seconds) and the primary is writing. 0 disables the brake. Default 0: the freeze branch has only been validated on a live cluster in its non-braking (pass-through) form, so the lag brake is opt-in until the braking path is exercised under load; set a positive value (e.g. 30) to enable it. | `int` | `0` |
254254
| `autoscaling.dryRun` | Recommendation mode: keep the static count and render the ScaledObject paused (KEDA does not actuate). Permanent, unlike `transition`. | `bool` | `false` |
255255
| `autoscaling.transition` | Migration phase 1: render `.spec.instances` from `replicas` (which the operator stages to the current live count) AND pause the ScaledObject, so KEDA stands up ready-but-not-actuating. Flip to false (phase 2) to hand the runtime count to KEDA. This shortens the handoff (KEDA is already warm, no cold-start) but does not make it atomic — see the enablement note for the residual dip and how to eliminate it. | `bool` | `false` |
256256

‎packages/apps/postgres/templates/_autoscaling.tpl‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,13 @@ tuned per the proposal's PoC; the base (no-lag) path is validated live.
118118
{{- $lagHigh := printf "((max(max_over_time(cnpg_pg_replication_lag{namespace=%q}[%s]) %s) > bool %d) or vector(0))" $ns $cooldown $joinCluster $maxLag -}}
119119
{{- $writing := printf "((max(rate(cnpg_collector_wal_records{namespace=%q}[5m]) %s) > bool 0) or vector(0))" $ns $joinCluster -}}
120120
{{- $braking := printf "(%s * %s)" $lagHigh $writing -}}
121-
{{- $frozen := printf "(count(kube_pod_labels{namespace=%q,label_cnpg_io_cluster=%q}) * %v)" $ns $rel $target -}}
121+
{{- /* currentInstances: count only instance pods. Restrict to pods carrying an
122+
instance role (label_cnpg_io_instance_role=~".+") like the read-load term
123+
above; CNPG stamps cnpg.io/cluster on its bootstrap/join Job pods too (they
124+
carry jobRole, not instanceRole), and kube-state-metrics reports them until
125+
GC. Without this filter a lingering join Job pod inflates the frozen count
126+
while the brake is engaged, nudging desired up instead of holding it. */ -}}
127+
{{- $frozen := printf "(count(kube_pod_labels{namespace=%q,label_cnpg_io_cluster=%q,label_cnpg_io_instance_role=~\".+\"}) * %v)" $ns $rel $target -}}
122128
{{- /* base when not braking, frozen (= currentInstances * target) when braking.
123129
The frozen summand is floored with `or vector(0)`: PromQL `+` is a set
124130
intersection, so if count(kube_pod_labels) is momentarily empty (a

‎packages/apps/postgres/templates/db.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,18 @@ spec:
2222
# subresource (design proposal §3). A brief cold-start window at 1 is expected
2323
# until KEDA scales up to the floor. See templates/scaledobject.yaml.
2424
{{- else }}
25+
{{- /* When the chart declares the static instance count, CNPG's admission webhook
26+
rejects the Cluster if maxSyncReplicas >= instances ("maxSyncReplicas must be
27+
lower than the number of instances"). This bites the DISABLE path in
28+
particular: an autoscaled + sync-quorum cluster runs with instances seeded
29+
from effectiveMin while replicas stays at its default, so flipping autoscaling
30+
off would render instances: <replicas> next to the quorum and wedge the
31+
HelmRelease upgrade with an opaque webhook error. Fail closed with an
32+
actionable message (symmetric to the enable-path guards). Also covers the
33+
dryRun/transition modes, which likewise render the static count. */ -}}
34+
{{- if le (int .Values.replicas) (int .Values.quorum.maxSyncReplicas) }}
35+
{{- fail (printf "postgres: replicas (%d) must be greater than quorum.maxSyncReplicas (%d) - CNPG rejects a Cluster whose maxSyncReplicas is not lower than its instance count. When disabling autoscaling on a synchronous-quorum cluster, first raise replicas to at least the live instance count (kubectl scale --subresource=scale with the HelmRelease suspended)." (int .Values.replicas) (int .Values.quorum.maxSyncReplicas)) }}
36+
{{- end }}
2537
instances: {{ .Values.replicas }}
2638
{{- end }}
2739
{{- if and $tlsEnabled .Values.external }}

‎packages/apps/postgres/tests/autoscaling_test.yaml‎

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -286,10 +286,12 @@ tests:
286286
- matchRegex:
287287
path: spec.triggers[0].metadata.query
288288
pattern: rate\(cnpg_collector_wal_records.*> bool 0\) or vector\(0\)
289-
# frozen branch pins desired to the current instance count under lag
289+
# frozen branch pins desired to the current instance count under lag, counting
290+
# only instance pods (role-scoped) so lingering CNPG join/bootstrap Job pods
291+
# (which carry cnpg.io/cluster but jobRole, not instanceRole) don't inflate it
290292
- matchRegex:
291293
path: spec.triggers[0].metadata.query
292-
pattern: count\(kube_pod_labels.* \* 150
294+
pattern: count\(kube_pod_labels\{namespace="tenant-acme",label_cnpg_io_cluster="postgres-db",label_cnpg_io_instance_role=~".\+"\}\) \* 150
293295
# the frozen summand is floored with `or vector(0)` so a kube-state-metrics
294296
# scrape gap (empty count) cannot empty the whole `+` sum and defeat the
295297
# base term's own floor
@@ -340,7 +342,7 @@ tests:
340342
maxReplicas: 6
341343
target: 150
342344
quorum:
343-
maxSyncReplicas: 2
345+
maxSyncReplicas: 1
344346
asserts:
345347
- equal:
346348
path: spec.instances
@@ -354,6 +356,23 @@ tests:
354356
value: "true"
355357
template: templates/scaledobject.yaml
356358

359+
- it: fails closed when the static instance count would violate synchronous quorum (disable-wedge guard)
360+
templates:
361+
- templates/db.yaml
362+
set:
363+
# autoscaling off: the chart renders the static instances: <replicas>. With
364+
# replicas at its default 2 and maxSyncReplicas 2, CNPG would reject the
365+
# Cluster (maxSyncReplicas must be < instances). This is exactly the wedge a
366+
# tenant hits when DISABLING autoscaling on a quorum cluster whose replicas
367+
# was never raised off the default. Must fail closed at render, not at the webhook.
368+
autoscaling:
369+
enabled: false
370+
quorum:
371+
maxSyncReplicas: 2
372+
asserts:
373+
- failedTemplate:
374+
errorMessage: "postgres: replicas (2) must be greater than quorum.maxSyncReplicas (2) - CNPG rejects a Cluster whose maxSyncReplicas is not lower than its instance count. When disabling autoscaling on a synchronous-quorum cluster, first raise replicas to at least the live instance count (kubectl scale --subresource=scale with the HelmRelease suspended)."
375+
357376
- it: rejects a non-positive autoscaling target
358377
templates:
359378
- templates/scaledobject.yaml

‎packages/apps/postgres/values.schema.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -228,9 +228,9 @@
228228
"default": 6
229229
},
230230
"maxReplicationLagSeconds": {
231-
"description": "Freeze scaling while replication lag exceeds this and the primary is writing.",
231+
"description": "Freeze scaling while replication lag exceeds this (seconds) and the primary is writing. 0 disables the brake. Default 0: the freeze branch has only been validated on a live cluster in its non-braking (pass-through) form, so the lag brake is opt-in until the braking path is exercised under load; set a positive value (e.g. 30) to enable it.",
232232
"type": "integer",
233-
"default": 30
233+
"default": 0
234234
},
235235
"metric": {
236236
"description": "Read-load signal to scale on.",

‎packages/apps/postgres/values.yaml‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -150,7 +150,7 @@ quorum:
150150
## @field {int} maxReplicas - Maximum total instances; raised to the quorum floor if the floor exceeds it.
151151
## @field {ReadMetric} metric - Read-load signal to scale on.
152152
## @field {int} target - Target read load per read-serving replica (unit depends on `metric`: active connections, or CPU millicores). Dimensionless integer - a resource.Quantity here would accept `150m`, which PromQL reads as 150 minutes and KEDA cannot use as a threshold.
153-
## @field {int} maxReplicationLagSeconds - Freeze scaling while replication lag exceeds this and the primary is writing.
153+
## @field {int} maxReplicationLagSeconds - Freeze scaling while replication lag exceeds this (seconds) and the primary is writing. 0 disables the brake. Default 0: the freeze branch has only been validated on a live cluster in its non-braking (pass-through) form, so the lag brake is opt-in until the braking path is exercised under load; set a positive value (e.g. 30) to enable it.
154154
## @field {bool} dryRun - Recommendation mode: keep the static count and render the ScaledObject paused (KEDA does not actuate). Permanent, unlike `transition`.
155155
## @field {bool} transition - Migration phase 1: render `.spec.instances` from `replicas` (which the operator stages to the current live count) AND pause the ScaledObject, so KEDA stands up ready-but-not-actuating. Flip to false (phase 2) to hand the runtime count to KEDA. This shortens the handoff (KEDA is already warm, no cold-start) but does not make it atomic — see the enablement note for the residual dip and how to eliminate it.
156156

@@ -183,14 +183,23 @@ quorum:
183183
## pin the live count with `kubectl scale --subresource=scale` (or set replicas on the
184184
## CNPG Cluster) so KEDA/CNPG holds it, run the phase-2 apply, then resume. Absent that,
185185
## expect a brief read-replica disruption during the handoff.
186+
## The same hazard applies to two other live changes, for the same three-way-merge
187+
## reason: (1) raising `minReplicas` or `quorum.maxSyncReplicas` on a running autoscaled
188+
## cluster re-renders the seeded `instances` to the new, higher effectiveMin — harmless
189+
## (scale-up) — but LOWERING either re-renders a smaller effectiveMin, so if KEDA had
190+
## scaled above it CNPG decommissions the surplus instances (deleting their PVCs) before
191+
## the HPA re-clones them; and (2) in the non-quorum case, disabling autoscaling snaps
192+
## `instances` from the live count back to `.Values.replicas`. Before a floor change or a
193+
## disable, pin `replicas` to the current live instance count (`kubectl scale
194+
## --subresource=scale`, HelmRelease suspended) so the re-render is a no-op.
186195
## @param {Autoscaling} autoscaling - Read-replica autoscaling configuration.
187196
autoscaling:
188197
enabled: false
189198
minReplicas: 2
190199
maxReplicas: 6
191200
metric: ReadConnections
192201
target: 150
193-
maxReplicationLagSeconds: 30
202+
maxReplicationLagSeconds: 0
194203
dryRun: false
195204
transition: false
196205

‎packages/core/platform/templates/apps.yaml‎

Lines changed: 2 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -18,14 +18,8 @@ metadata:
1818
type: Opaque
1919
stringData:
2020
values.yaml: |
21-
# KEDA's clusterDomain is a vendored-subchart value a chart template cannot
22-
# derive from _cluster; inject it here from the authoritative
23-
# networking.clusterDomain so KEDA's metrics-apiserver dials the operator at the
24-
# right domain on clusters whose domain is not the cozy.local default (otherwise
25-
# the external-metrics gRPC target is NXDOMAIN and every DB HPA stalls). Ignored
26-
# by packages without a keda subchart (no app/extra schema sets root
27-
# additionalProperties:false), and the keda package keeps cozy.local as its own
28-
# fallback default.
21+
# keda.clusterDomain: authoritative cluster domain injected into the KEDA subchart
22+
# (a subchart value a template cannot derive from _cluster). Rationale: packages/system/keda.
2923
keda:
3024
clusterDomain: {{ .Values.networking.clusterDomain | quote }}
3125
_cluster:

‎packages/library/cozy-lib/templates/_keda.tpl‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,10 @@ Parameters:
8989
{{- end -}}
9090
{{- $labels := merge (dict "app.kubernetes.io/managed-by" "cozystack") (default (dict) .labels) -}}
9191
{{- $annotations := default (dict) .annotations -}}
92-
{{- if .paused -}}
92+
{{- /* Coerce rather than truthiness-test: a caller passing the string "false"
93+
(easy to produce from an include without an `| eq "true"` guard) must NOT
94+
pause. Only a real true / "true" pauses. */ -}}
95+
{{- if eq (printf "%v" (default false .paused)) "true" -}}
9396
{{- $annotations = merge (dict "autoscaling.keda.sh/paused" "true") $annotations -}}
9497
{{- end -}}
9598
apiVersion: keda.sh/v1alpha1

0 commit comments

Comments
 (0)