Skip to content

Commit de59928

Browse files
committed
docs(dha): PoC finding — CNPG scale subresource has no selector; reopen mechanism
A live PoC validated the metric encoding (Σ+target drives the right count on real data) but found the load-bearing mechanism — a stock HPA via KEDA driving the CNPG Cluster scale subresource — is not buildable on stock CNPG: the Cluster /scale exposes no status.selector (no labelSelectorPath), so the HPA controller fails with InvalidSelector for every metric type. Upstream CNPG #7923 (add the selector) is closed as not planned; not fixable by a version bump. Record the finding and two resolution options — (A) KEDA + a thin mirror shim behind a proxy object with a selector, or (B) a lean actuation controller (no KEDA) writing Cluster.spec.instances directly. Note that neither returns rev1's ownership machinery, which §3 already removed. This reopens the actuation mechanism decision. Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 1e70234 commit de59928

1 file changed

Lines changed: 22 additions & 2 deletions

File tree

  • design-proposals/database-horizontal-autoscaling

‎design-proposals/database-horizontal-autoscaling/README.md‎

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,17 +3,37 @@
33
- **Title:** `Database Horizontal Autoscaler for Cozystack`
44
- **Author(s):** `@scooby87`
55
- **Date:** `2026-07-08`; revised `2026-07-24` (mechanism), `2026-07-29` and `2026-07-31` (addressing @lllamnyp and @IvanHunters review on PR #44), with earlier review by @IvanHunters, Gemini, and CodeRabbit
6-
- **Status:** Draft
6+
- **Status:** Draft — mechanism reopened by the `2026-08-10` PoC finding below
7+
- **PoC:** `2026-08-10` — live validation surfaced a blocking constraint in CNPG; see [PoC finding](#poc-finding-2026-08-10--blocking-the-cnpg-scale-subresource-exposes-no-selector)
78

89
## Overview
910

1011
This proposal adds automatic horizontal scaling of a managed database's **read replicas** in response to load. The mechanism is **entirely stock**: the application chart renders a **KEDA `ScaledObject`** next to the database; KEDA queries VictoriaMetrics for the read load, computes the desired count with a plain `HorizontalPodAutoscaler` it manages, and drives the engine operator's **`scale` subresource** (CloudNativePG `Cluster.spec.instances`). There is **no bespoke operator and no new CRD** — the net-new surface of this proposal is a Helm helper, one `autoscaling` values block, one PromQL query, and KEDA added as a platform component.
1112

1213
The proposal is deliberately scoped to **horizontal scaling of read replicas**: a stateful primary cannot be scaled horizontally the way a stateless Deployment can. The MVP targets **PostgreSQL (CloudNativePG)**; see [Scope](#scope-and-related-proposals) for the engine ladder.
1314

15+
> **Note (2026-08-10):** a live PoC validated the metric side of this design but found that a stock HPA cannot drive the CNPG `Cluster` scale subresource (it exposes no selector), which reopens the actuation mechanism. See [PoC finding](#poc-finding-2026-08-10--blocking-the-cnpg-scale-subresource-exposes-no-selector) below; the sections after it describe the mechanism as it stood before that finding.
16+
1417
### Why this changed
1518

16-
This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart omitting the field, keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it.
19+
This design converged over three revisions, each removing machinery the previous one thought it needed. Rev1 proposed a bespoke `db-autoscaler` operator that *owned* the application's `replicas` value and enforced that ownership; an implementation spike proved the enforcement premise unbuildable on the aggregated apps API, and showed the whole conflict was self-imposed — it exists only because the chart unconditionally templates the replica field (full findings in the [Appendix](#appendix-findings-from-the-implementation-spike)). Rev2/rev3 therefore moved to a stock HPA on the engine's `scale` subresource with the chart omitting the field, keeping only a thin controller and CRD to render the HPA and drive a synthesized metric. Review then showed even that is unnecessary: the metric can be *queried* into existence rather than emitted per-pod, and once the query exists, KEDA renders and manages everything declaratively — so the controller and CRD are gone too. The guiding principle throughout: reuse the platform Kubernetes ships, do not reimplement it. The PoC below shows that last step reused one thing Kubernetes cannot actually provide here.
20+
21+
## PoC finding (2026-08-10) — blocking: the CNPG scale subresource exposes no selector
22+
23+
A live PoC on a dev cluster (cozystack v45, CloudNativePG 1.27.3, Kubernetes 1.34.3) validated the metric side of this design but uncovered a constraint the mechanism above does not survive as written.
24+
25+
**What the PoC confirmed.** The single-value metric works on real data: for a live CNPG cluster the query `Σ(active read connections over the standby pods) + target` returned `151` at `target = 150` (one active connection on the replica), so `desired = ceil(151/150) = 2 = 1 primary + 1 read replica` — the §1 arithmetic holds. The required series and labels exist in VictoriaMetrics (`cnpg_backends_total{state="active"}`, and `kube_pod_labels` carrying `label_cnpg_io_cluster` and `label_cnpg_io_instance_role`). The KEDA package installs, its `external.metrics.k8s.io` APIService becomes Available, and KEDA renders the `ScaledObject` into a managed HPA.
26+
27+
**The blocker.** That HPA never scales: it reports `ScalingActive=False, reason=InvalidSelector` — *"the HPA target's scale is missing a selector"*. The CNPG `Cluster` `/scale` subresource returns only `status: {replicas: N}` — no `status.selector` — and the CRD declares no `labelSelectorPath`. The Kubernetes HPA controller requires `scale.status.selector` unconditionally, before any metric-type branching, so this fails for **every** target type (confirmed with both `AverageValue` and `Value`). It is not a calibration detail and not fixable by a version bump: upstream CNPG issue [#7923](https://github.com/cloudnative-pg/cloudnative-pg/issues/7923), which requested exactly this selector for HPA/KEDA, is **closed as not planned**, and the CNPG 1.30 docs explicitly recommend against HPA for a `Cluster`.
28+
29+
**Consequence.** The load-bearing mechanism of this revision — a stock HPA (via KEDA) driving the CNPG `Cluster` scale subresource — cannot be built on stock CNPG; an actuation bridge is required after all. Crucially, what returns is **not** the machinery that got rev1 rejected: the ownership/enforcement layer (SSA, marker annotation, HelmRelease webhook, terminal-freeze) existed only because the chart declared `replicas`, and §3 (the chart omitting the field under autoscaling) removes it regardless of mechanism. What returns is only the small write-the-count actuator.
30+
31+
**Two options to resolve (decision needed).**
32+
33+
- **Option A — KEDA + a thin mirror shim.** Keep KEDA's hardened decision loop by pointing its HPA at a proxy object that *does* expose a selector (a small owned CRD, or a placeholder workload), and add a tiny controller that mirrors the proxy's computed count into `Cluster.spec.instances`. Preserves the stock decision loop, but adds a shim, a proxy object, and the platform-wide KEDA dependency for a value KEDA cannot deliver end-to-end on its own.
34+
- **Option B — a lean actuation controller, no KEDA.** A small controller reads the read-load metric from VictoriaMetrics and writes `Cluster.spec.instances` directly, applying the `min`/`max`/quorum-floor bounds and stabilization. This is close to rev1 **minus the ownership machinery** (which §3 already eliminates) and minus the aggregated-API enforcement — a much smaller component than the rejected operator, with no KEDA platform dependency, at the cost of a modest amount of stabilization logic KEDA would otherwise provide.
35+
36+
Both keep §1 (the validated metric encoding) and §3 (chart omits the field). The choice is where the desired-count computation lives — stock KEDA behind a proxy, or a lean purpose-built loop — and whether to take on KEDA as a platform dependency. This reopens the mechanism decision made in the previous revision; the sections below describe the pre-finding mechanism and stand until that decision is made.
1737

1838
## Scope and related proposals
1939

0 commit comments

Comments
 (0)