Skip to content

Commit 66145e7

Browse files
committed
fix(postgres): seed spec.instances at the quorum floor under autoscaling
Under active autoscaling the chart previously omitted .spec.instances entirely, letting the KEDA HPA own it via /scale. But CNPG defaults an absent .spec.instances to 1, and its admission webhook rejects a Cluster where maxSyncReplicas >= instances ("maxSyncReplicas must be lower than the number of instances"). So a fresh database created with autoscaling.enabled: true AND synchronous quorum (maxSyncReplicas >= 1) was rejected at admission before KEDA could ever scale it to effectiveMin (= maxSyncReplicas+1) — the whole tenant HelmRelease failed. The default maxSyncReplicas: 0 path was unaffected, which is why it went unnoticed. Verified live: a rendered Cluster with maxSyncReplicas: 1 and an omitted instances is rejected by the dev-cluster webhook; with the seeded floor it is admitted. Render instances: effectiveMin under active autoscaling instead of omitting it, so a fresh Cluster starts at a valid, quorum-satisfying count. Because the chart emits the same constant floor on every reconcile, Helm's three-way merge is a no-op on the field and never reverts KEDA's higher live value, so the HPA still owns the runtime count. Update the enablement docs (a new DB can enable directly even with synchronous replication; existing DBs above the floor rebase to it unless the two-phase transition flow is used) and add render tests for the fresh create + maxSyncReplicas>=1 case. Addresses the fresh-create-with-quorum admission gap found in branch review. Signed-off-by: Alexey Artamonov <[email protected]>
1 parent d4ba7bb commit 66145e7

7 files changed

Lines changed: 74 additions & 31 deletions

File tree

‎api/apps/v1alpha1/postgresql/types.go‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/postgres/README.md‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -242,17 +242,17 @@ For `sslmode=verify-full` to work, the CA bundle retrieved above must be saved t
242242

243243
### Autoscaling
244244

245-
| Name | Description | Type | Value |
246-
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | ----------------- |
247-
| `autoscaling` | Read-replica autoscaling configuration. | `object` | `{}` |
248-
| `autoscaling.enabled` | Enable horizontal autoscaling of read replicas. Requires the platform `keda` package to be enabled by an administrator first; the chart fails to render (rather than break the release) if KEDA is not installed. | `bool` | `false` |
249-
| `autoscaling.minReplicas` | Minimum total instances; raised to the synchronous-quorum floor (`maxSyncReplicas + 1`) and to 2 when either is higher. | `int` | `2` |
250-
| `autoscaling.maxReplicas` | Maximum total instances; raised to the quorum floor if the floor exceeds it. | `int` | `6` |
251-
| `autoscaling.metric` | Read-load signal to scale on. | `string` | `ReadConnections` |
252-
| `autoscaling.target` | Target read load per read-serving replica (unit depends on `metric`: active connections, or CPU millicores). Dimensionless integer - a resource.Quantity here would accept `150m`, which PromQL reads as 150 minutes and KEDA cannot use as a threshold. | `int` | `150` |
253-
| `autoscaling.maxReplicationLagSeconds` | Freeze scaling while replication lag exceeds this and the primary is writing. | `int` | `30` |
254-
| `autoscaling.dryRun` | Recommendation mode: keep the static count and render the ScaledObject paused (KEDA does not actuate). Permanent, unlike `transition`. | `bool` | `false` |
255-
| `autoscaling.transition` | Migration phase 1: keep `.spec.instances` rendered as a floor (from `replicas`, which the operator stages to the live count) AND pause the ScaledObject, so KEDA stands up without contending for the field. Flip to false (phase 2) to drop the static field and hand the count to KEDA. See the enablement note below. | `bool` | `false` |
245+
| Name | Description | Type | Value |
246+
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----------------- |
247+
| `autoscaling` | Read-replica autoscaling configuration. | `object` | `{}` |
248+
| `autoscaling.enabled` | Enable horizontal autoscaling of read replicas. Requires the platform `keda` package to be enabled by an administrator first; the chart fails to render (rather than break the release) if KEDA is not installed. | `bool` | `false` |
249+
| `autoscaling.minReplicas` | Minimum total instances; raised to the synchronous-quorum floor (`maxSyncReplicas + 1`) and to 2 when either is higher. | `int` | `2` |
250+
| `autoscaling.maxReplicas` | Maximum total instances; raised to the quorum floor if the floor exceeds it. | `int` | `6` |
251+
| `autoscaling.metric` | Read-load signal to scale on. | `string` | `ReadConnections` |
252+
| `autoscaling.target` | Target read load per read-serving replica (unit depends on `metric`: active connections, or CPU millicores). Dimensionless integer - a resource.Quantity here would accept `150m`, which PromQL reads as 150 minutes and KEDA cannot use as a threshold. | `int` | `150` |
253+
| `autoscaling.maxReplicationLagSeconds` | Freeze scaling while replication lag exceeds this and the primary is writing. | `int` | `30` |
254+
| `autoscaling.dryRun` | Recommendation mode: keep the static count and render the ScaledObject paused (KEDA does not actuate). Permanent, unlike `transition`. | `bool` | `false` |
255+
| `autoscaling.transition` | Migration phase 1: render `.spec.instances` from `replicas` (which the operator stages to the current live count) AND pause the ScaledObject, so KEDA stands up without actuating. Flip to false (phase 2) to hand the runtime count to KEDA. Only needed to avoid a dip when enabling on an existing cluster sized above the quorum floor; see the enablement note below. | `bool` | `false` |
256256

257257

258258
### Users configuration

‎packages/apps/postgres/templates/db.yaml‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,18 @@ kind: Cluster
66
metadata:
77
name: {{ .Release.Name }}
88
spec:
9-
{{- if not $autoscaling }}
10-
# Under active autoscaling the KEDA-managed HPA owns .spec.instances via the
11-
# scale subresource, so the chart must not declare it (a declared value would
12-
# be reverted by Flux). See templates/scaledobject.yaml and the design proposal.
9+
{{- if $autoscaling }}
10+
# Under active autoscaling the KEDA-managed HPA owns .spec.instances at runtime
11+
# via the /scale subresource. The chart still seeds the synchronous-quorum floor
12+
# (effectiveMin) instead of omitting the field: CNPG defaults an absent
13+
# .spec.instances to 1, which its admission webhook rejects when
14+
# maxSyncReplicas>=1 ("maxSyncReplicas must be lower than the number of
15+
# instances"), so a fresh autoscaled + sync-quorum Cluster would never be
16+
# admitted. Because the chart renders the same constant floor on every
17+
# reconcile, Helm's three-way merge is a no-op on this field and never reverts
18+
# KEDA's higher live value. See templates/scaledobject.yaml and the design proposal.
19+
instances: {{ include "postgres.autoscaling.effectiveMin" . }}
20+
{{- else }}
1321
instances: {{ .Values.replicas }}
1422
{{- end }}
1523
{{- if and $tlsEnabled .Values.external }}

‎packages/apps/postgres/tests/autoscaling_test.yaml‎

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ tests:
4545
- hasDocuments:
4646
count: 0
4747

48-
- it: drops spec.instances from the Cluster and floors the WorkloadMonitor under active autoscaling
48+
- it: seeds spec.instances at the quorum floor and floors the WorkloadMonitor under active autoscaling
4949
templates:
5050
- templates/db.yaml
5151
set:
@@ -57,8 +57,13 @@ tests:
5757
quorum:
5858
maxSyncReplicas: 2
5959
asserts:
60-
- notExists:
60+
# The field is seeded (not omitted) to effectiveMin = max(minReplicas=2,
61+
# maxSyncReplicas+1=3, 2) = 3 so a fresh Cluster is admitted at a valid,
62+
# quorum-satisfying count; KEDA then owns the runtime value via /scale and
63+
# the constant render keeps three-way merge from reverting it.
64+
- equal:
6165
path: spec.instances
66+
value: 3
6267
documentSelector:
6368
path: kind
6469
value: Cluster
@@ -70,6 +75,29 @@ tests:
7075
path: kind
7176
value: WorkloadMonitor
7277

78+
- it: seeds a quorum-valid instance count on a fresh autoscaled cluster with synchronous replication
79+
templates:
80+
- templates/db.yaml
81+
set:
82+
autoscaling:
83+
enabled: true
84+
minReplicas: 2
85+
maxReplicas: 6
86+
target: 150
87+
quorum:
88+
maxSyncReplicas: 1
89+
asserts:
90+
# Regression guard for the CNPG admission webhook: an absent .spec.instances
91+
# defaults to 1, and "maxSyncReplicas must be lower than the number of
92+
# instances" rejects the Cluster when maxSyncReplicas>=1. effectiveMin here is
93+
# max(2, 1+1, 2) = 2 > maxSyncReplicas=1, so the fresh Cluster is admitted.
94+
- equal:
95+
path: spec.instances
96+
value: 2
97+
documentSelector:
98+
path: kind
99+
value: Cluster
100+
73101
- it: renders a KEDA ScaledObject with quorum-floored bounds under active autoscaling
74102
templates:
75103
- templates/scaledobject.yaml

‎packages/apps/postgres/values.schema.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -252,7 +252,7 @@
252252
"default": 150
253253
},
254254
"transition": {
255-
"description": "Migration phase 1: keep `.spec.instances` rendered as a floor (from `replicas`, which the operator stages to the live count) AND pause the ScaledObject, so KEDA stands up without contending for the field. Flip to false (phase 2) to drop the static field and hand the count to KEDA. See the enablement note below.",
255+
"description": "Migration phase 1: render `.spec.instances` from `replicas` (which the operator stages to the current live count) AND pause the ScaledObject, so KEDA stands up without actuating. Flip to false (phase 2) to hand the runtime count to KEDA. Only needed to avoid a dip when enabling on an existing cluster sized above the quorum floor; see the enablement note below.",
256256
"type": "boolean",
257257
"default": false
258258
}

‎packages/apps/postgres/values.yaml‎

Lines changed: 18 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -152,22 +152,29 @@ quorum:
152152
## @field {int} target - Target read load per read-serving replica (unit depends on `metric`: active connections, or CPU millicores). Dimensionless integer - a resource.Quantity here would accept `150m`, which PromQL reads as 150 minutes and KEDA cannot use as a threshold.
153153
## @field {int} maxReplicationLagSeconds - Freeze scaling while replication lag exceeds this and the primary is writing.
154154
## @field {bool} dryRun - Recommendation mode: keep the static count and render the ScaledObject paused (KEDA does not actuate). Permanent, unlike `transition`.
155-
## @field {bool} transition - Migration phase 1: keep `.spec.instances` rendered as a floor (from `replicas`, which the operator stages to the live count) AND pause the ScaledObject, so KEDA stands up without contending for the field. Flip to false (phase 2) to drop the static field and hand the count to KEDA. See the enablement note below.
155+
## @field {bool} transition - Migration phase 1: render `.spec.instances` from `replicas` (which the operator stages to the current live count) AND pause the ScaledObject, so KEDA stands up without actuating. Flip to false (phase 2) to hand the runtime count to KEDA. Only needed to avoid a dip when enabling on an existing cluster sized above the quorum floor; see the enablement note below.
156156

157157
## @enum {string} ReadMetric - Read-load metric type.
158158
## @value ReadConnections - Active read connections on the standby pods.
159159
## @value ReadCPUUtilization - CPU (millicores) of the postgres container on the standby pods.
160160

161-
## Enabling autoscaling on an EXISTING database is a two-phase, count-preserving
162-
## migration — a naive enable drops `.spec.instances` and CNPG defaults to 1.
163-
## Phase 1: stage `replicas` to the current live instance count and set
164-
## `autoscaling: {enabled: true, transition: true, ...}` — the field is held as a
165-
## floor and the ScaledObject comes up paused (no writer conflict); verify KEDA is
166-
## healthy. Phase 2: set `transition: false`. Because helm-controller applies via
167-
## Helm's three-way merge (not SSA), suspend the HelmRelease across the phase-2
168-
## apply so dropping the static field does not race CNPG's default-to-1 (the exact
169-
## safe handoff is confirmed on a live cluster per the design proposal). A NEW
170-
## database can skip the migration — set `enabled: true` directly.
161+
## Enabling autoscaling seeds `.spec.instances` at the quorum floor
162+
## (effectiveMin = max(minReplicas, maxSyncReplicas+1, 2)) and lets the KEDA HPA own
163+
## the runtime count via the /scale subresource. The chart re-renders that same
164+
## constant floor on every reconcile, so Helm's three-way merge is a no-op on the
165+
## field and never reverts KEDA's higher live value.
166+
## A NEW database can enable autoscaling directly — the seeded floor is a valid,
167+
## quorum-satisfying count, so CNPG's admission webhook accepts it even with
168+
## synchronous replication (maxSyncReplicas>=1), where an absent/defaulted
169+
## instances=1 would be rejected.
170+
## On an EXISTING database sized above the floor, a direct enable rebases the
171+
## Cluster to effectiveMin before KEDA scales it back up by load. To avoid that dip,
172+
## use the two-phase flow: (phase 1) stage `replicas` to the live count and set
173+
## `autoscaling: {enabled: true, transition: true, ...}` — the live count is held
174+
## and the ScaledObject comes up paused; verify KEDA is healthy. (phase 2) set
175+
## `transition: false` to hand the count to KEDA. Suspend the HelmRelease across the
176+
## phase-2 apply so the render/actuation handoff does not race (helm-controller uses
177+
## Helm's three-way merge, not SSA).
171178
## @param {Autoscaling} autoscaling - Read-replica autoscaling configuration.
172179
autoscaling:
173180
enabled: false

0 commit comments

Comments
 (0)