Skip to content

Commit de8195a

Browse files
committed
docs(dha): retitle the PoC-finding section as a selector gate, not a version floor
The section's own conclusion is that HPA actuation gates on the CNPG /scale subresource serving a non-empty status.selector, a behavioural property that 1.29.0/1.29.1 fail despite being numerically above the 1.28.4 line -- so it is explicitly not a version floor. Three spots still carried the old framing and contradicted that conclusion: the section heading ("CNPG version floor for HPA actuation"), the "Why this changed" close ("behind a version floor"), and the blocked-it lead ("why it is version-bound"). Retitle to the selector gate and sync the three inbound anchor links so the section reads consistently with what it proves. The remaining "version floor" mentions are correct and kept: the ones that assert it is not a version floor, and the KEDA >= 2.18.0 floor in Open questions, which is a separate, real version floor. Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 3163b34 commit de8195a

1 file changed

Lines changed: 6 additions & 6 deletions

File tree

  • design-proposals/database-horizontal-autoscaling

‎design-proposals/database-horizontal-autoscaling/README.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,27 +4,27 @@
44
- **Author(s):** `@scooby87`
55
- **Date:** `2026-07-08`; revised `2026-07-24` (mechanism), `2026-07-29` and `2026-07-31` (addressing @lllamnyp and @IvanHunters review on PR #44), with earlier review by @IvanHunters, Gemini, and CodeRabbit
66
- **Status:** Draft
7-
- **PoC:** `2026-08-10` — live validation confirmed the mechanism and pinned a hard precondition: the CNPG `Cluster` `/scale` subresource must serve a non-empty `status.selector` (else the HPA rejects the target with `reason=InvalidSelector`) — a behavioural gate, not a version floor (carried by 1.28.4+, 1.29.2+, 1.30.0+ but absent from 1.29.0/1.29.1); see [PoC finding](#poc-finding-2026-08-10--cnpg-version-floor-for-hpa-actuation)
7+
- **PoC:** `2026-08-10` — live validation confirmed the mechanism and pinned a hard precondition: the CNPG `Cluster` `/scale` subresource must serve a non-empty `status.selector` (else the HPA rejects the target with `reason=InvalidSelector`) — a behavioural gate, not a version floor (carried by 1.28.4+, 1.29.2+, 1.30.0+ but absent from 1.29.0/1.29.1); see [PoC finding](#poc-finding-2026-08-10--cnpg-scale-subresource-selector-gate-for-hpa-actuation)
88

99
## Overview
1010

1111
This proposal adds automatic horizontal scaling of a managed database's **read replicas** in response to load. The mechanism is **entirely stock**: the application chart renders a **KEDA `ScaledObject`** next to the database; KEDA queries VictoriaMetrics for the read load, computes the desired count with a plain `HorizontalPodAutoscaler` it manages, and drives the engine operator's **`scale` subresource** (CloudNativePG `Cluster.spec.instances`). There is **no bespoke operator and no new CRD** — the net-new surface of this proposal is a Helm helper, one `autoscaling` values block, one PromQL query, and KEDA added as a platform component.
1212

1313
The proposal is deliberately scoped to **horizontal scaling of read replicas**: a stateful primary cannot be scaled horizontally the way a stateless Deployment can. The MVP targets **PostgreSQL (CloudNativePG)**; see [Scope](#scope-and-related-proposals) for the engine ladder.
1414

15-
> **Note (2026-08-10):** a live PoC validated this design and pinned a behavioural precondition — a stock HPA can only drive the CNPG `Cluster` scale subresource when that subresource serves a non-empty `status.selector` (otherwise the HPA rejects the target with `reason=InvalidSelector`); this is a served-selector gate, not a version floor (cloudnative-pg#8996). The mechanism is unchanged; see the [PoC finding](#poc-finding-2026-08-10--cnpg-version-floor-for-hpa-actuation) below.
15+
> **Note (2026-08-10):** a live PoC validated this design and pinned a behavioural precondition — a stock HPA can only drive the CNPG `Cluster` scale subresource when that subresource serves a non-empty `status.selector` (otherwise the HPA rejects the target with `reason=InvalidSelector`); this is a served-selector gate, not a version floor (cloudnative-pg#8996). The mechanism is unchanged; see the [PoC finding](#poc-finding-2026-08-10--cnpg-scale-subresource-selector-gate-for-hpa-actuation) below.
1616
1717
### Why this changed
1818

19-
This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart ceding effective control of the field to the HPA (rev2/rev3 omitted it; the final design in §3 seeds a constant applied client-side), keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it. The PoC below shows that last step relied on one thing the vendored CNPG version did not yet provide — since fixed upstream, leaving the mechanism unchanged behind a version floor.
19+
This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart ceding effective control of the field to the HPA (rev2/rev3 omitted it; the final design in §3 seeds a constant applied client-side), keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it. The PoC below shows that last step relied on one thing the vendored CNPG version did not yet provide — since fixed upstream, leaving the mechanism unchanged behind a served-selector gate.
2020

21-
## PoC finding (2026-08-10) — CNPG version floor for HPA actuation
21+
## PoC finding (2026-08-10) — CNPG scale-subresource selector gate for HPA actuation
2222

2323
A live PoC on a dev cluster (cozystack v45, CloudNativePG 1.27.3, Kubernetes 1.34.3) validated this design and pinned one hard precondition: the actuation path works only where the CNPG `Cluster` `/scale` subresource serves a non-empty `status.selector` (without it the HPA rejects the target with `reason=InvalidSelector` before any metric branching) — a behavioural gate, not a version floor (see [§Scope](#scope-and-related-proposals) for the release matrix). This section is a record, not an open decision — the mechanism is unchanged.
2424

2525
**What the PoC confirmed.** The single-value metric works on real data: for a live CNPG cluster the query `Σ(active read connections over the standby pods) + target` returned `151` at `target = 150` (one active connection on the replica), so `desired = ceil(151/150) = 2 = 1 primary + 1 read replica` — the §1 arithmetic holds. The required series and labels exist in VictoriaMetrics (`cnpg_backends_total{state="active"}`, and `kube_pod_labels` carrying `label_cnpg_io_cluster` and `label_cnpg_io_instance_role`). The KEDA package installs, its `external.metrics.k8s.io` APIService becomes Available, and KEDA renders the `ScaledObject` into a managed HPA.
2626

27-
**What blocked it — and why it is version-bound.** On the CNPG the PoC ran (1.27.x), the HPA never scales: it reports `ScalingActive=False, reason=InvalidSelector` — *"the HPA target's scale is missing a selector"*. That CNPG's `Cluster` `/scale` subresource returned only `status: {replicas: N}` with no `status.selector`, and the CRD declared no `labelSelectorPath`; the Kubernetes HPA controller requires `scale.status.selector` unconditionally, before any metric-type branching, so it fails for every target type (confirmed with both `AverageValue` and `Value`). This is fixed upstream. cloudnative-pg#2574 ("Support the scale sub-resource for Clusters") was implemented by cloudnative-pg#8996 ("feat: add label selector to the Cluster scale subresource"), which populates `Cluster.status.selector` on every status update and wires `subresources.scale.labelSelectorPath: .status.selector` into the CRD, naming HPA/VPA explicitly. It shipped on 2026-06-29 in **CNPG v1.28.4, v1.29.2, and v1.30.0**, and was not backported to the 1.27 line the PoC ran on. (The earlier issue [#7923](https://github.com/cloudnative-pg/cloudnative-pg/issues/7923) asking for the same thing was not declined — it went stale as an unnoticed duplicate; #2574/#8996 is the accepted, completed work.)
27+
**What blocked it — and why it is a selector gate.** On the CNPG the PoC ran (1.27.x), the HPA never scales: it reports `ScalingActive=False, reason=InvalidSelector` — *"the HPA target's scale is missing a selector"*. That CNPG's `Cluster` `/scale` subresource returned only `status: {replicas: N}` with no `status.selector`, and the CRD declared no `labelSelectorPath`; the Kubernetes HPA controller requires `scale.status.selector` unconditionally, before any metric-type branching, so it fails for every target type (confirmed with both `AverageValue` and `Value`). This is fixed upstream. cloudnative-pg#2574 ("Support the scale sub-resource for Clusters") was implemented by cloudnative-pg#8996 ("feat: add label selector to the Cluster scale subresource"), which populates `Cluster.status.selector` on every status update and wires `subresources.scale.labelSelectorPath: .status.selector` into the CRD, naming HPA/VPA explicitly. It shipped on 2026-06-29 in **CNPG v1.28.4, v1.29.2, and v1.30.0**, and was not backported to the 1.27 line the PoC ran on. (The earlier issue [#7923](https://github.com/cloudnative-pg/cloudnative-pg/issues/7923) asking for the same thing was not declined — it went stale as an unnoticed duplicate; #2574/#8996 is the accepted, completed work.)
2828

2929
**Resolution — no design change.** On any CNPG that serves a non-empty `/scale` `status.selector` the approved mechanism (chart conditional + KEDA `ScaledObject` + the `Σ + target` query) works byte-for-byte on stock CNPG against a real operator-maintained selector. The fix was a routine vendored-chart bump, since merged ([cozystack/cozystack#3951](https://github.com/cozystack/cozystack/pull/3951)): `packages/system/postgres-operator` now vendors CNPG 1.30.0, above the floor, so no design change and no pending dependency remain — a re-run of this same PoC on the vendored operator is the verification gate. Two alternatives were considered on the way to this record and are **rejected** as unnecessary once the selector ships: (A) KEDA plus a thin mirror shim behind a proxy object that exposes a selector; (B) a lean actuation controller writing `Cluster.spec.instances` directly. (A) adds a CRD and a controller to work around a gap that no longer exists; (B) re-acquires the hand-rolled decision loop this design spent three revisions deleting. Neither would have re-introduced rev1's ownership machinery — §3 (the chart ceding the field to the HPA) already removed that — but neither is needed.
3030

@@ -34,7 +34,7 @@ A live PoC on a dev cluster (cozystack v45, CloudNativePG 1.27.3, Kubernetes 1.3
3434

3535
This proposal covers **horizontal** autoscaling (read replicas) only. Two sibling axes are deferred to separate proposals: **vertical autoscaling** (stepping the `resourcesPreset` ladder / in-place resize) and **storage autoscaling** (automatic PVC expansion). Write-path scaling that requires data rebalancing (Kafka broker addition, ClickHouse/MongoDB sharding) is out of scope — it is an orchestrated procedure, not a counter change.
3636

37-
**Precondition — the CNPG scale subresource must serve a selector.** The `Cluster` `/scale` subresource must serve a non-empty `status.selector` (wired via `labelSelectorPath: .status.selector`), or the Kubernetes HPA rejects the target with `ScalingActive=False, reason=InvalidSelector` before any metric-type branching and never scales (see the [PoC finding](#poc-finding-2026-08-10--cnpg-version-floor-for-hpa-actuation)). The selector shipped in cloudnative-pg#8996 and is carried by CNPG **1.28.4+, 1.29.2+, and 1.30.0+**, but is absent from **1.29.0 and 1.29.1** (cut before the 1.29-line backport) — so the gate is the selector actually being served, not a monotone `≥ 1.28.4` version bound, since 1.29.0 and 1.29.1 are numerically higher yet fail it. cozystack vendors CNPG 1.30.0 since [cozystack/cozystack#3951](https://github.com/cozystack/cozystack/pull/3951), which serves it. Because the gate is behavioural, an operator can verify it on a live cluster before enabling autoscaling — the check is not a version comparison: `kubectl get --raw /apis/postgresql.cnpg.io/v1/namespaces/<ns>/clusters/<name>/scale | jq -r '.status.selector'` must print a non-empty label selector; an empty string or `null` means the running operator does not serve it and the HPA will report `ScalingActive=False, reason=InvalidSelector`. This matters for any cluster on a non-vendored or pinned CNPG (e.g. an external operator, or a downgrade onto the 1.29.0/1.29.1 gap), which a `≥ 1.28.4` reading would wrongly wave through.
37+
**Precondition — the CNPG scale subresource must serve a selector.** The `Cluster` `/scale` subresource must serve a non-empty `status.selector` (wired via `labelSelectorPath: .status.selector`), or the Kubernetes HPA rejects the target with `ScalingActive=False, reason=InvalidSelector` before any metric-type branching and never scales (see the [PoC finding](#poc-finding-2026-08-10--cnpg-scale-subresource-selector-gate-for-hpa-actuation)). The selector shipped in cloudnative-pg#8996 and is carried by CNPG **1.28.4+, 1.29.2+, and 1.30.0+**, but is absent from **1.29.0 and 1.29.1** (cut before the 1.29-line backport) — so the gate is the selector actually being served, not a monotone `≥ 1.28.4` version bound, since 1.29.0 and 1.29.1 are numerically higher yet fail it. cozystack vendors CNPG 1.30.0 since [cozystack/cozystack#3951](https://github.com/cozystack/cozystack/pull/3951), which serves it. Because the gate is behavioural, an operator can verify it on a live cluster before enabling autoscaling — the check is not a version comparison: `kubectl get --raw /apis/postgresql.cnpg.io/v1/namespaces/<ns>/clusters/<name>/scale | jq -r '.status.selector'` must print a non-empty label selector; an empty string or `null` means the running operator does not serve it and the HPA will report `ScalingActive=False, reason=InvalidSelector`. This matters for any cluster on a non-vendored or pinned CNPG (e.g. an external operator, or a downgrade onto the 1.29.0/1.29.1 gap), which a `≥ 1.28.4` reading would wrongly wave through.
3838

3939
**Engine scope of the MVP.** The mechanism applies to engines whose operator CR exposes a `scale` subresource: PostgreSQL (CloudNativePG `Cluster.spec.instances`) and MariaDB (`MariaDB.spec.replicas`). The MVP ships **PostgreSQL**; MariaDB follows once its cozystack chart supports on-the-fly scale-out (today it does not — see [Failure and edge cases](#failure-and-edge-cases)). **Redis (spotahome RedisFailover) and MongoDB (Percona) expose no `scale` subresource**, so a stock HPA cannot drive them; they are deferred to a follow-up that adds a thin actuation shim (see [Alternatives considered](#alternatives-considered)). Note what generalises and what does not: the KEDA wiring, the client-side-apply mechanism ([decision 0001](decisions/0001-client-side-apply-preserves-the-autoscaler-seed.md)), and the KEDA RBAC are engine-general and reviewed once, but the MVP's **alert and dashboard bundle is written against CNPG's labels and metrics** (`label_cnpg_io_cluster`, `cnpg_*`) — it is not automatically shared across engines, so each engine on the ladder needs its own observability bundle when it lands, and the "reviewed once, not per-database" claim in [Security](#security) covers the RBAC, not the monitoring bundle.
4040

0 commit comments

Comments
 (0)