Skip to content

Commit 1e70234

Browse files
committed
docs(dha): address branch-review — commit two-phase migration, consistency nits
- §Upgrade: commit the enable AND disable migration to an explicit two-phase order in prose (observe live count -> stage -> stand up under a floor -> hand the field off via SSA field-manager release), leaving only the SSA-vs-helm-merge handoff detail as the single PoC item; the disable mirror stages .Values.replicas to the live count so it cannot shrink a running cluster - fix the rendered ScaledObject name (postgres-db, not db) to match its scaleTargetRef and the release name - make the minReplicaCount=3 example reproducible (note maxSyncReplicas=2) - §4: reconcile the per-object serverAddress argument with the shared-root vmselect example (MVP reads the root vmselect scoped by label; per-object serverAddress enables per-tenant stacks later) - PoC checklist: verify spec.subresources.scale on the pinned CNPG version Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 8eb3684 commit 1e70234

1 file changed

Lines changed: 7 additions & 7 deletions

File tree

  • design-proposals/database-horizontal-autoscaling

‎design-proposals/database-horizontal-autoscaling/README.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ The conditional keys off `autoscaling.enabled`, **not** off presence of the fiel
9090
An HPA object cannot carry a query — its metric spec holds only a name and a selector — so the query must live where the metrics-API backend reads it, and the options differ sharply:
9191

9292
- **prometheus-adapter — ruled out.** Its queries live in one global ConfigMap, so a per-application query means per-application adapter config plus a reload — a registration step for every database. It also speaks to a single upstream URL, while every tenant's metrics live behind a different vmselect.
93-
- **KEDA — recommended.** The query lives inline in a namespaced `ScaledObject` that the chart renders exactly where it would have rendered an HPA; there is no global config and no registration step, and KEDA generates and manages the HPA itself. Everything this design needs passes through: `scaleTargetRef` accepts any CR with a `scale` subresource (CNPG `Cluster` qualifies), `minReplicaCount`/`maxReplicaCount` take the template-computed bounds, `advanced.horizontalPodAutoscalerConfig.behavior` carries the scale-down policies verbatim, and `serverAddress` is per-object — so each tenant's `ScaledObject` points at its own vmselect, which a single-upstream adapter cannot do.
93+
- **KEDA — recommended.** The query lives inline in a namespaced `ScaledObject` that the chart renders exactly where it would have rendered an HPA; there is no global config and no registration step, and KEDA generates and manages the HPA itself. Everything this design needs passes through: `scaleTargetRef` accepts any CR with a `scale` subresource (CNPG `Cluster` qualifies), `minReplicaCount`/`maxReplicaCount` take the template-computed bounds, `advanced.horizontalPodAutoscalerConfig.behavior` carries the scale-down policies verbatim, and `serverAddress` is per-object. In the MVP every `ScaledObject` reads the shared root vmselect (`vmselect-shortterm.tenant-root.svc`) with the query scoped by namespace/lineage labels; the per-object `serverAddress` is the property that lets a tenant with its own isolated monitoring stack point at its own vmselect later without any central reconfiguration — the thing a single-upstream adapter cannot do.
9494
- **kube-metrics-adapter (Zalando)** is the lighter alternative — the query lives in annotations on the HPA — but it is a much smaller project and its per-tenant-server story is weaker.
9595

9696
Because the query is authored by the chart template (the tenant supplies only numbers through values), the mandatory-scoping rule — no raw tenant PromQL against shared vmselect — is satisfied by construction. The cost is that **KEDA becomes a new platform component**: a cluster-singleton that claims the `external.metrics.k8s.io` APIService (nothing serves it in Cozystack today), shared by any future feature that needs custom-metric autoscaling.
@@ -131,10 +131,10 @@ The chart renders (reference only — the tenant never authors this):
131131
```yaml
132132
apiVersion: keda.sh/v1alpha1
133133
kind: ScaledObject
134-
metadata: { name: db, namespace: tenant-acme }
134+
metadata: { name: postgres-db, namespace: tenant-acme } # both rendered from the release name
135135
spec:
136136
scaleTargetRef: { apiVersion: postgresql.cnpg.io/v1, kind: Cluster, name: postgres-db }
137-
minReplicaCount: 3 # max(minReplicas, maxSyncReplicas+1, 2)
137+
minReplicaCount: 3 # max(minReplicas=2, quorum.maxSyncReplicas+1, 2); =3 here with maxSyncReplicas=2
138138
maxReplicaCount: 6
139139
advanced:
140140
horizontalPodAutoscalerConfig:
@@ -154,9 +154,9 @@ When `autoscaling.enabled` is false, nothing changes — the chart templates `re
154154
## Upgrade and rollback compatibility
155155

156156
- **Opt-in and off by default.** The chart conditional is inert unless `autoscaling.enabled` is set; KEDA and the alert/dashboard bundle are optional platform packages. Existing clusters are unaffected.
157-
- **Enabling autoscaling on an existing database — the one real migration, and it needs a deterministic two-phase order.** Flipping `autoscaling.enabled` removes `instances` from the rendered CR, and Helm's three-way merge deletes a key present in the old manifest and absent from the new one **regardless of who last wrote it** — so simply pre-setting `.spec.instances` through the scale subresource does **not** save it: the upgrade deletes the field, CNPG defaults to **1 instance**, and the HPA only re-raises it after CNPG has already begun removing standbys. A safe rollout therefore needs a real two-phase design — e.g. a transition window in which the chart templates `.spec.instances` as a floor (rendered in both the old and new manifest so three-way merge never sees it disappear) while KEDA takes over, then a second phase that drops the floor once the HPA is the established writer. The precise operation order must be worked out and exercised on a dev cluster before MVP; "must be tested" is a gate, not the mechanism.
157+
- **Enabling autoscaling on an existing database — the one real migration.** The hazard: flipping `autoscaling.enabled` removes `instances` from the rendered CR, and Helm's three-way merge deletes a key present in the old manifest and absent from the new one **regardless of who last wrote it**, so a naive flip deletes the field, CNPG defaults to **1 instance**, and the HPA only re-raises it after CNPG has already begun removing standbys. The committed two-phase order that avoids this: **(phase 1 — stand up under a floor, no field removal)** the operator reads the live `.status.instances` (= N) and sets `.Values.replicas = N` (a no-op to the running cluster); then sets `autoscaling.enabled: true` with a transition sub-flag that keeps the chart rendering `instances: {{ .Values.replicas }}` (= N) **alongside** the new `ScaledObject` (whose `minReplicaCount` is pinned to N). The field never leaves the manifest, so three-way merge never deletes it; Flux and the HPA both target N, so neither fights; KEDA comes up healthy and begins observing load. **(phase 2 — hand the field off)** once the `ScaledObject`/HPA is Ready, clear the transition flag so the chart stops rendering `instances`. This is the single present→absent transition, and it is safe only if Flux relinquishes its claim on `.spec.instances` while the HPA keeps writing it — i.e. the handoff must ride on server-side-apply field-manager ownership (Flux drops the field from *its* managed-fields; the HPA's scale-subresource writes keep the value alive), not on Helm's classic three-way delete. Confirming that the platform's Flux/helm-controller path performs this as an SSA release rather than a hard delete — and, if it does not, pinning `.spec.instances` via the scale subresource across the phase-2 apply as a fallback — is the one migration detail the PoC must settle. Steady state after phase 2 is safe: with the field absent from every subsequent render, three-way merge leaves the HPA-managed value untouched.
158158
- **Steady state after migration is correct.** With the field absent from both the previous and the current render, three-way merge leaves the HPA-set `.spec.instances` untouched.
159-
- **Rollback must be count-preserving too.** Setting `autoscaling.enabled: false` re-introduces `instances: {{ .Values.replicas }}` — and `replicas` defaults to `2`, so a naive disable would shrink a live cluster the HPA had grown to, say, 6. The disable path must first observe the current instance count and stage it into `.Values.replicas` (or hold the count via the scale subresource through the transition) before the static field is reintroduced — the mirror image of the enable migration. Only then is it fully reversible; no data migration is involved either way.
159+
- **Disabling must be count-preserving too — the mirror sequence.** Setting `autoscaling.enabled: false` re-introduces `instances: {{ .Values.replicas }}`, and `replicas` defaults to `2`, so a naive disable would shrink a live cluster the HPA had grown to, say, 6. The committed order: **(phase 1)** the operator reads the live `.status.instances` (= M, the count the HPA is currently holding) and sets `.Values.replicas = M`; **(phase 2)** clears `autoscaling.enabled` and deletes the `ScaledObject` in the same apply — the chart re-renders `instances: M`, which matches the live count, so Flux reasserts the current value rather than dropping to the default. Only with `.Values.replicas` staged to the live count first is the disable a no-op to the running cluster; no data migration is involved either way.
160160
- **Cold start.** Until KEDA's HPA takes its first sample it holds at `minReplicaCount`; a brief window at the floor is expected.
161161
- **Enablement constraint — `minReplicas ≥ 2` changes single-instance footprint.** Enabling autoscaling on a current single-instance Postgres permanently doubles instances (a second replica's PVC and DRBD volume). This is legitimate but must be a conscious enablement decision, not a surprise — and it is load-bearing, since at `Σ = 0` the formula yields `desired = 1`.
162162
- **Dependent objects.** Consumers that read `.Values.replicas` (dashboards, some tooling) must switch to the observed count. Note the two are distinct: the **engine CR** carries `.status.instances`; the **`WorkloadMonitor`** carries `availableReplicas`/`observedReplicas`/`operational` — do not read a nonexistent `WorkloadMonitor.status.instances`.
@@ -182,7 +182,7 @@ When `autoscaling.enabled` is false, nothing changes — the chart templates `re
182182

183183
## Testing
184184

185-
- **PoC first — validate the single-value metric (§1) against a real HPA:** confirm the `Σ + target` query with an `AverageValue` threshold drives a KEDA-managed HPA to `1 + ceil(Σ/target)` across the `ceil` boundaries, including `Σ = 0 → 1` clamped up by `minReplicaCount`. This gates everything else.
185+
- **PoC first — validate the single-value metric (§1) against a real HPA:** confirm the `Σ + target` query with an `AverageValue` threshold drives a KEDA-managed HPA to `1 + ceil(Σ/target)` across the `ceil` boundaries, including `Σ = 0 → 1` clamped up by `minReplicaCount`. Also confirm the pinned CloudNativePG version actually exposes `spec.subresources.scale` on `Cluster.spec.instances` (the assumption the whole mechanism rests on — present in the currently vendored CNPG, but version-sensitive). This gates everything else.
186186
- **Chart:** `helm template` with `autoscaling.enabled: true` omits the replica field and renders a well-formed `ScaledObject` (bounds = `max(minReplicas, maxSyncReplicas+1, 2)`, scale-down policy present, query scoped to the app's namespace/labels); with it false, renders `replicas` exactly as today (regression guard).
187187
- **Migration (dev cluster, CNPG):** exercise the two-phase enable on a running multi-instance cluster and assert it does **not** collapse to 1 instance; then drive load and confirm the HPA scales `.spec.instances`, reads route to `<release>-ro`, and Flux does not revert. Exercise the disable path and assert it does **not** shrink the live cluster to the default `replicas`.
188188
- **KEDA integration:** lag above threshold with active writes freezes scaling both ways and releases only past the hysteresis band; raising `maxSyncReplicas` re-renders the floor; scale-down removes one standby per `periodSeconds`.
@@ -198,7 +198,7 @@ When `autoscaling.enabled` is false, nothing changes — the chart templates `re
198198
## Open questions
199199

200200
- Final shape of the lag-clamp query (how `currentInstances` is sourced — pod count vs HPA status series) and the hysteresis recovery band / cooldown — deliberate defaults to be tuned at PoC.
201-
- Exact two-phase enable/disable migration mechanic (chart-templated floor during transition vs a staged handover), to be settled on a dev cluster before MVP.
201+
- The two-phase enable/disable order is committed in §Upgrade; the one detail left for the PoC is whether the phase-2 field handoff rides on Flux/helm-controller SSA field-manager release (preferred) or needs the scale-subresource-pin fallback.
202202
- Default driver metric (read connections vs read QPS vs replica CPU), to be calibrated on real workloads.
203203
- KEDA packaging in cozystack (version, HA, which APIService/metrics-server coexistence concerns) — it is the one new platform singleton and needs an owner.
204204

0 commit comments

Comments
 (0)