You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit de8195a
Browse filesBrowse the repository at this point in the historyBrowse files
docs(dha): retitle the PoC-finding section as a selector gate, not a version floor
The section's own conclusion is that HPA actuation gates on the CNPG
/scale subresource serving a non-empty status.selector, a behavioural
property that 1.29.0/1.29.1 fail despite being numerically above the
1.28.4 line -- so it is explicitly not a version floor. Three spots
still carried the old framing and contradicted that conclusion: the
section heading ("CNPG version floor for HPA actuation"), the "Why this
changed" close ("behind a version floor"), and the blocked-it lead
("why it is version-bound"). Retitle to the selector gate and sync the
three inbound anchor links so the section reads consistently with what
it proves.
The remaining "version floor" mentions are correct and kept: the ones
that assert it is not a version floor, and the KEDA >= 2.18.0 floor in
Open questions, which is a separate, real version floor.
Signed-off-by: Alexey Artamonov <[email protected]>
Copy file name to clipboardExpand all lines: design-proposals/database-horizontal-autoscaling/README.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,27 +4,27 @@
4
4
-**Author(s):**`@scooby87`
5
5
-**Date:**`2026-07-08`; revised `2026-07-24` (mechanism), `2026-07-29` and `2026-07-31` (addressing @lllamnyp and @IvanHunters review on PR #44), with earlier review by @IvanHunters, Gemini, and CodeRabbit
6
6
-**Status:** Draft
7
-
-**PoC:**`2026-08-10` — live validation confirmed the mechanism and pinned a hard precondition: the CNPG `Cluster``/scale` subresource must serve a non-empty `status.selector` (else the HPA rejects the target with `reason=InvalidSelector`) — a behavioural gate, not a version floor (carried by 1.28.4+, 1.29.2+, 1.30.0+ but absent from 1.29.0/1.29.1); see [PoC finding](#poc-finding-2026-08-10--cnpg-version-floor-for-hpa-actuation)
7
+
-**PoC:**`2026-08-10` — live validation confirmed the mechanism and pinned a hard precondition: the CNPG `Cluster``/scale` subresource must serve a non-empty `status.selector` (else the HPA rejects the target with `reason=InvalidSelector`) — a behavioural gate, not a version floor (carried by 1.28.4+, 1.29.2+, 1.30.0+ but absent from 1.29.0/1.29.1); see [PoC finding](#poc-finding-2026-08-10--cnpg-scale-subresource-selector-gate-for-hpa-actuation)
8
8
9
9
## Overview
10
10
11
11
This proposal adds automatic horizontal scaling of a managed database's **read replicas** in response to load. The mechanism is **entirely stock**: the application chart renders a **KEDA `ScaledObject`** next to the database; KEDA queries VictoriaMetrics for the read load, computes the desired count with a plain `HorizontalPodAutoscaler` it manages, and drives the engine operator's **`scale` subresource** (CloudNativePG `Cluster.spec.instances`). There is **no bespoke operator and no new CRD** — the net-new surface of this proposal is a Helm helper, one `autoscaling` values block, one PromQL query, and KEDA added as a platform component.
12
12
13
13
The proposal is deliberately scoped to **horizontal scaling of read replicas**: a stateful primary cannot be scaled horizontally the way a stateless Deployment can. The MVP targets **PostgreSQL (CloudNativePG)**; see [Scope](#scope-and-related-proposals) for the engine ladder.
14
14
15
-
> **Note (2026-08-10):** a live PoC validated this design and pinned a behavioural precondition — a stock HPA can only drive the CNPG `Cluster` scale subresource when that subresource serves a non-empty `status.selector` (otherwise the HPA rejects the target with `reason=InvalidSelector`); this is a served-selector gate, not a version floor (cloudnative-pg#8996). The mechanism is unchanged; see the [PoC finding](#poc-finding-2026-08-10--cnpg-version-floor-for-hpa-actuation) below.
15
+
> **Note (2026-08-10):** a live PoC validated this design and pinned a behavioural precondition — a stock HPA can only drive the CNPG `Cluster` scale subresource when that subresource serves a non-empty `status.selector` (otherwise the HPA rejects the target with `reason=InvalidSelector`); this is a served-selector gate, not a version floor (cloudnative-pg#8996). The mechanism is unchanged; see the [PoC finding](#poc-finding-2026-08-10--cnpg-scale-subresource-selector-gate-for-hpa-actuation) below.
16
16
17
17
### Why this changed
18
18
19
-
This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart ceding effective control of the field to the HPA (rev2/rev3 omitted it; the final design in §3 seeds a constant applied client-side), keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it. The PoC below shows that last step relied on one thing the vendored CNPG version did not yet provide — since fixed upstream, leaving the mechanism unchanged behind a version floor.
19
+
This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart ceding effective control of the field to the HPA (rev2/rev3 omitted it; the final design in §3 seeds a constant applied client-side), keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it. The PoC below shows that last step relied on one thing the vendored CNPG version did not yet provide — since fixed upstream, leaving the mechanism unchanged behind a served-selector gate.
20
20
21
-
## PoC finding (2026-08-10) — CNPG version floor for HPA actuation
A live PoC on a dev cluster (cozystack v45, CloudNativePG 1.27.3, Kubernetes 1.34.3) validated this design and pinned one hard precondition: the actuation path works only where the CNPG `Cluster``/scale` subresource serves a non-empty `status.selector` (without it the HPA rejects the target with `reason=InvalidSelector` before any metric branching) — a behavioural gate, not a version floor (see [§Scope](#scope-and-related-proposals) for the release matrix). This section is a record, not an open decision — the mechanism is unchanged.
24
24
25
25
**What the PoC confirmed.** The single-value metric works on real data: for a live CNPG cluster the query `Σ(active read connections over the standby pods) + target` returned `151` at `target = 150` (one active connection on the replica), so `desired = ceil(151/150) = 2 = 1 primary + 1 read replica` — the §1 arithmetic holds. The required series and labels exist in VictoriaMetrics (`cnpg_backends_total{state="active"}`, and `kube_pod_labels` carrying `label_cnpg_io_cluster` and `label_cnpg_io_instance_role`). The KEDA package installs, its `external.metrics.k8s.io` APIService becomes Available, and KEDA renders the `ScaledObject` into a managed HPA.
26
26
27
-
**What blocked it — and why it is version-bound.** On the CNPG the PoC ran (1.27.x), the HPA never scales: it reports `ScalingActive=False, reason=InvalidSelector` — *"the HPA target's scale is missing a selector"*. That CNPG's `Cluster` `/scale` subresource returned only `status: {replicas: N}` with no `status.selector`, and the CRD declared no `labelSelectorPath`; the Kubernetes HPA controller requires `scale.status.selector` unconditionally, before any metric-type branching, so it fails for every target type (confirmed with both `AverageValue` and `Value`). This is fixed upstream. cloudnative-pg#2574 ("Support the scale sub-resource for Clusters") was implemented by cloudnative-pg#8996 ("feat: add label selector to the Cluster scale subresource"), which populates `Cluster.status.selector` on every status update and wires `subresources.scale.labelSelectorPath: .status.selector` into the CRD, naming HPA/VPA explicitly. It shipped on 2026-06-29 in **CNPG v1.28.4, v1.29.2, and v1.30.0**, and was not backported to the 1.27 line the PoC ran on. (The earlier issue [#7923](https://github.com/cloudnative-pg/cloudnative-pg/issues/7923) asking for the same thing was not declined — it went stale as an unnoticed duplicate; #2574/#8996 is the accepted, completed work.)
27
+
**What blocked it — and why it is a selector gate.** On the CNPG the PoC ran (1.27.x), the HPA never scales: it reports `ScalingActive=False, reason=InvalidSelector` — *"the HPA target's scale is missing a selector"*. That CNPG's `Cluster` `/scale` subresource returned only `status: {replicas: N}` with no `status.selector`, and the CRD declared no `labelSelectorPath`; the Kubernetes HPA controller requires `scale.status.selector` unconditionally, before any metric-type branching, so it fails for every target type (confirmed with both `AverageValue` and `Value`). This is fixed upstream. cloudnative-pg#2574 ("Support the scale sub-resource for Clusters") was implemented by cloudnative-pg#8996 ("feat: add label selector to the Cluster scale subresource"), which populates `Cluster.status.selector` on every status update and wires `subresources.scale.labelSelectorPath: .status.selector` into the CRD, naming HPA/VPA explicitly. It shipped on 2026-06-29 in **CNPG v1.28.4, v1.29.2, and v1.30.0**, and was not backported to the 1.27 line the PoC ran on. (The earlier issue [#7923](https://github.com/cloudnative-pg/cloudnative-pg/issues/7923) asking for the same thing was not declined — it went stale as an unnoticed duplicate; #2574/#8996 is the accepted, completed work.)
28
28
29
29
**Resolution — no design change.** On any CNPG that serves a non-empty `/scale` `status.selector` the approved mechanism (chart conditional + KEDA `ScaledObject` + the `Σ + target` query) works byte-for-byte on stock CNPG against a real operator-maintained selector. The fix was a routine vendored-chart bump, since merged ([cozystack/cozystack#3951](https://github.com/cozystack/cozystack/pull/3951)): `packages/system/postgres-operator` now vendors CNPG 1.30.0, above the floor, so no design change and no pending dependency remain — a re-run of this same PoC on the vendored operator is the verification gate. Two alternatives were considered on the way to this record and are **rejected** as unnecessary once the selector ships: (A) KEDA plus a thin mirror shim behind a proxy object that exposes a selector; (B) a lean actuation controller writing `Cluster.spec.instances` directly. (A) adds a CRD and a controller to work around a gap that no longer exists; (B) re-acquires the hand-rolled decision loop this design spent three revisions deleting. Neither would have re-introduced rev1's ownership machinery — §3 (the chart ceding the field to the HPA) already removed that — but neither is needed.
30
30
@@ -34,7 +34,7 @@ A live PoC on a dev cluster (cozystack v45, CloudNativePG 1.27.3, Kubernetes 1.3
34
34
35
35
This proposal covers **horizontal** autoscaling (read replicas) only. Two sibling axes are deferred to separate proposals: **vertical autoscaling** (stepping the `resourcesPreset` ladder / in-place resize) and **storage autoscaling** (automatic PVC expansion). Write-path scaling that requires data rebalancing (Kafka broker addition, ClickHouse/MongoDB sharding) is out of scope — it is an orchestrated procedure, not a counter change.
36
36
37
-
**Precondition — the CNPG scale subresource must serve a selector.** The `Cluster` `/scale` subresource must serve a non-empty `status.selector` (wired via `labelSelectorPath: .status.selector`), or the Kubernetes HPA rejects the target with `ScalingActive=False, reason=InvalidSelector` before any metric-type branching and never scales (see the [PoC finding](#poc-finding-2026-08-10--cnpg-version-floor-for-hpa-actuation)). The selector shipped in cloudnative-pg#8996 and is carried by CNPG **1.28.4+, 1.29.2+, and 1.30.0+**, but is absent from **1.29.0 and 1.29.1** (cut before the 1.29-line backport) — so the gate is the selector actually being served, not a monotone `≥ 1.28.4` version bound, since 1.29.0 and 1.29.1 are numerically higher yet fail it. cozystack vendors CNPG 1.30.0 since [cozystack/cozystack#3951](https://github.com/cozystack/cozystack/pull/3951), which serves it. Because the gate is behavioural, an operator can verify it on a live cluster before enabling autoscaling — the check is not a version comparison: `kubectl get --raw /apis/postgresql.cnpg.io/v1/namespaces/<ns>/clusters/<name>/scale | jq -r '.status.selector'` must print a non-empty label selector; an empty string or `null` means the running operator does not serve it and the HPA will report `ScalingActive=False, reason=InvalidSelector`. This matters for any cluster on a non-vendored or pinned CNPG (e.g. an external operator, or a downgrade onto the 1.29.0/1.29.1 gap), which a `≥ 1.28.4` reading would wrongly wave through.
37
+
**Precondition — the CNPG scale subresource must serve a selector.** The `Cluster` `/scale` subresource must serve a non-empty `status.selector` (wired via `labelSelectorPath: .status.selector`), or the Kubernetes HPA rejects the target with `ScalingActive=False, reason=InvalidSelector` before any metric-type branching and never scales (see the [PoC finding](#poc-finding-2026-08-10--cnpg-scale-subresource-selector-gate-for-hpa-actuation)). The selector shipped in cloudnative-pg#8996 and is carried by CNPG **1.28.4+, 1.29.2+, and 1.30.0+**, but is absent from **1.29.0 and 1.29.1** (cut before the 1.29-line backport) — so the gate is the selector actually being served, not a monotone `≥ 1.28.4` version bound, since 1.29.0 and 1.29.1 are numerically higher yet fail it. cozystack vendors CNPG 1.30.0 since [cozystack/cozystack#3951](https://github.com/cozystack/cozystack/pull/3951), which serves it. Because the gate is behavioural, an operator can verify it on a live cluster before enabling autoscaling — the check is not a version comparison: `kubectl get --raw /apis/postgresql.cnpg.io/v1/namespaces/<ns>/clusters/<name>/scale | jq -r '.status.selector'` must print a non-empty label selector; an empty string or `null` means the running operator does not serve it and the HPA will report `ScalingActive=False, reason=InvalidSelector`. This matters for any cluster on a non-vendored or pinned CNPG (e.g. an external operator, or a downgrade onto the 1.29.0/1.29.1 gap), which a `≥ 1.28.4` reading would wrongly wave through.
38
38
39
39
**Engine scope of the MVP.** The mechanism applies to engines whose operator CR exposes a `scale` subresource: PostgreSQL (CloudNativePG `Cluster.spec.instances`) and MariaDB (`MariaDB.spec.replicas`). The MVP ships **PostgreSQL**; MariaDB follows once its cozystack chart supports on-the-fly scale-out (today it does not — see [Failure and edge cases](#failure-and-edge-cases)). **Redis (spotahome RedisFailover) and MongoDB (Percona) expose no `scale` subresource**, so a stock HPA cannot drive them; they are deferred to a follow-up that adds a thin actuation shim (see [Alternatives considered](#alternatives-considered)). Note what generalises and what does not: the KEDA wiring, the client-side-apply mechanism ([decision 0001](decisions/0001-client-side-apply-preserves-the-autoscaler-seed.md)), and the KEDA RBAC are engine-general and reviewed once, but the MVP's **alert and dashboard bundle is written against CNPG's labels and metrics** (`label_cnpg_io_cluster`, `cnpg_*`) — it is not automatically shared across engines, so each engine on the ladder needs its own observability bundle when it lands, and the "reviewed once, not per-database" claim in [Security](#security) covers the RBAC, not the monitoring bundle.
0 commit comments