Skip to content

Commit afc003c

Browse files
authored
feat(postgres): database read-replica autoscaling via KEDA (#3954)
## What this PR does Adds automatic horizontal autoscaling of a PostgreSQL database's **read replicas** in response to load, implementing the merged design proposal [Database Horizontal Autoscaler](https://github.com/cozystack/community/tree/main/design-proposals/database-horizontal-autoscaling) (community#44). The mechanism is entirely stock — no bespoke operator, no new CRD: - **KEDA as an optional platform component** (`packages/system/keda`, wired into `core/platform` as a source + system-bundle entry). It claims the `external.metrics.k8s.io` APIService that nothing served before. - **`cozy-lib.keda.scaledObject` helper** (`packages/library/cozy-lib/templates/_keda.tpl`) renders a KEDA `ScaledObject`, fail-closed on missing/invalid arguments. - **`packages/apps/postgres` autoscaling** — an `autoscaling` values block (`enabled`, `minReplicas`, `maxReplicas`, `metric`, `target`, `maxReplicationLagSeconds`, `dryRun`, `transition`) validated by `values.schema.json`; under active autoscaling the chart renders a `ScaledObject` and seeds `spec.instances` to a constant floor (`max(replicas, effectiveMin)`), while the postgres release is forced to client-side (two-way) Helm apply so the seed is a merge no-op on upgrade and the KEDA-managed HPA — the runtime writer via the CNPG `Cluster` scale subresource — is not reverted. - **Single-value metric (§1)** — one PromQL query returns `Σ(read load over standbys) + target`, so a stock HPA computes `desired = ceil((Σ+target)/target) = 1 + ceil(Σ/target) = primary + desiredRead`, with no per-pod emission. - **The four static brakes (§5)** — quorum floor `max(minReplicas, maxSyncReplicas+1, 2)` (quorum wins over `maxReplicas`), scale-down pacing (`Pods/1/600s`), a replication-lag clamp folded into the query (freeze both directions while lag exceeds the threshold and the primary is writing, with `max_over_time` hysteresis), and dry-run (render paused). - **Two-phase enable/disable migration** via a `transition` sub-flag: phase 1 stages `spec.instances` to the live count and brings KEDA up with its HPA paused in both directions; flipping to phase 2 hands the runtime count to KEDA, and because the seed re-renders the same count the flip is a merge no-op with no dip. - **Observability** — a database-autoscaling Grafana dashboard and PrometheusRule alerts. **Live validation (dev cluster, CNPG 1.30.0).** The full path works end-to-end: the CNPG `Cluster` scale subresource serves `status.selector`, KEDA renders the HPA, the metric reaches it (`ScalingActive=ValidMetricFound`, `TARGETS 151/150`, `desiredReplicas=2 = 1 primary + 1 replica` — the §1 arithmetic), and the lag-clamp query returns the base value on live VictoriaMetrics. Two packaging fixes were needed and are included: KEDA must run on the cozystack `cozy.local` cluster domain (the upstream `cluster.local` default makes the metrics gRPC target NXDOMAIN and every HPA hangs on `FailedGetExternalMetric`), and the operator's client-go throttle is raised so a single-replica operator does not lose its leader lease under scale-subresource polling. Implements #3953. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. Two repositories are affected; both are follow-ups to open once this lands (and after its dependency #3951 merges and the values surface is final), left unticked here for a maintainer to confirm rather than claimed with a speculative link: - **cozystack/terraform-provider-cozystack** — this adds the nested `autoscaling` block to `packages/apps/postgres/values.schema.json`. The provider hand-models each app's fields (schema + model + expand/flatten), so it needs the new block to expose it in Terraform. - **cozystack/website** — KEDA is a new platform component (`guides/platform-stack`), and the postgres reference page gains the `autoscaling` params. The managed-app reference is regenerated from `README.md` by the release bot; the platform-stack component list is hand-maintained. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: to open (new platform component + postgres autoscaling params) - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: to open (new `autoscaling` block in postgres values.schema.json) ### Depends on - #3951 — bump CloudNativePG to 1.30.0. The HPA can only drive the CNPG `Cluster` scale subresource on CNPG ≥ 1.28.4, where it serves `status.selector`; without it the HPA fails with `InvalidSelector`. This PR should not merge before #3951. ### Release note ```release-note feat(postgres): add optional horizontal autoscaling of read replicas via KEDA, configured through the new `autoscaling` values block; KEDA ships as a new optional platform component. Requires CloudNativePG >= 1.28.4 for the Cluster scale subresource selector. Note: every Postgres release now applies client-side (`release.cozystack.io/helm-server-side-apply: "false"`) so the autoscaler's live `/scale` count is not reverted by Helm; a kind-wide side effect is that chart-rendered fields on Postgres releases are no longer re-asserted by server-side-apply drift correction (driftDetection is unset), a documented trade-off with a per-instance apply strategy as follow-up. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable PostgreSQL read-replica autoscaling with KEDA. * Added CPU and read-connection metrics, replica bounds, replication-lag safeguards, dry-run mode, and transition handling. * Added optional KEDA platform support, Grafana autoscaling dashboard, and Prometheus alerts. * **Documentation** * Clarified integer thresholds, prerequisites, configuration, and transition guidance. * **Bug Fixes** * Improved validation and actionable errors for unavailable KEDA or invalid settings. * **Tests** * Expanded coverage for autoscaling, observability, package selection, and validation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 56e10b1 + 9511f56 commit afc003c

96 files changed

Lines changed: 18897 additions & 103 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎api/apps/v1alpha1/postgresql/types.go‎

Lines changed: 38 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/postgresql/zz_generated.deepcopy.go‎

Lines changed: 16 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
{
2+
"title": "Database Autoscaling",
3+
"uid": "cozy-db-autoscaling",
4+
"tags": ["cozystack", "database", "autoscaling"],
5+
"timezone": "browser",
6+
"schemaVersion": 39,
7+
"time": { "from": "now-6h", "to": "now" },
8+
"templating": {
9+
"list": [
10+
{
11+
"current": { "selected": false, "text": "default", "value": "default" },
12+
"hide": 0,
13+
"includeAll": false,
14+
"label": "Datasource",
15+
"multi": false,
16+
"name": "ds_prometheus",
17+
"options": [],
18+
"query": "prometheus",
19+
"queryValue": "",
20+
"refresh": 1,
21+
"regex": "",
22+
"skipUrlSync": false,
23+
"type": "datasource"
24+
},
25+
{
26+
"name": "namespace",
27+
"type": "query",
28+
"datasource": { "type": "prometheus", "uid": "${ds_prometheus}" },
29+
"query": "label_values(kube_horizontalpodautoscaler_status_current_replicas{horizontalpodautoscaler=~\"keda-hpa-.*\"}, namespace)",
30+
"includeAll": true,
31+
"multi": true
32+
}
33+
]
34+
},
35+
"panels": [
36+
{
37+
"id": 1,
38+
"title": "Desired vs current replicas",
39+
"description": "Live HPA actuation, per KEDA HPA (keda-hpa-<cluster>). In dry-run / transition (ScaledObject paused in both directions) the HPA's desiredReplicas is frozen to currentReplicas, so the two lines coincide — read the recommendation from the matching read-load panel (connections or CPU) instead: desired = 1 + ceil(load / target), clamped to [minReplicas..maxReplicas] (floor = max(minReplicas, maxSyncReplicas+1, 2)).",
40+
"type": "timeseries",
41+
"datasource": { "type": "prometheus", "uid": "${ds_prometheus}" },
42+
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
43+
"targets": [
44+
{ "datasource": { "type": "prometheus", "uid": "${ds_prometheus}" }, "expr": "kube_horizontalpodautoscaler_status_current_replicas{horizontalpodautoscaler=~\"keda-hpa-.*\",namespace=~\"$namespace\"}", "legendFormat": "current {{namespace}}/{{horizontalpodautoscaler}}" },
45+
{ "datasource": { "type": "prometheus", "uid": "${ds_prometheus}" }, "expr": "kube_horizontalpodautoscaler_status_desired_replicas{horizontalpodautoscaler=~\"keda-hpa-.*\",namespace=~\"$namespace\"}", "legendFormat": "desired {{namespace}}/{{horizontalpodautoscaler}}" }
46+
]
47+
},
48+
{
49+
"id": 2,
50+
"title": "Active read connections (replicas, per cluster)",
51+
"description": "Replica-scoped active connections per cluster — the ReadConnections autoscaler input. For a cluster on metric: ReadConnections the recommendation is desired = 1 + ceil(this / target), clamped to [minReplicas..maxReplicas] (floor = max(minReplicas, maxSyncReplicas+1, 2)); CPU-mode clusters read the 'Replica CPU' panel instead. Restricted to standby pods via the same replica join as the ScaledObject query and scoped per cluster (label_cnpg_io_cluster), so it also surfaces the recommendation in dry-run, where panel 1's desired line is frozen to current.",
52+
"type": "timeseries",
53+
"datasource": { "type": "prometheus", "uid": "${ds_prometheus}" },
54+
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
55+
"targets": [
56+
{ "datasource": { "type": "prometheus", "uid": "${ds_prometheus}" }, "expr": "sum by (namespace, label_cnpg_io_cluster) (cnpg_backends_total{state=\"active\",namespace=~\"$namespace\"} * on(namespace,pod) group_left(label_cnpg_io_cluster) kube_pod_labels{label_cnpg_io_cluster!=\"\",label_cnpg_io_instance_role=\"replica\",namespace=~\"$namespace\"})", "legendFormat": "{{namespace}}/{{label_cnpg_io_cluster}}" }
57+
]
58+
},
59+
{
60+
"id": 3,
61+
"title": "Replication lag (seconds, per cluster)",
62+
"description": "Max standby replication lag per cluster. Deliberately joins ALL instance pods (no instance-role filter, unlike the read-load panels) because lag is a per-standby signal collapsed by max(); the primary reports no lag. Feeds the opt-in replication-lag brake (autoscaling.maxReplicationLagSeconds).",
63+
"type": "timeseries",
64+
"datasource": { "type": "prometheus", "uid": "${ds_prometheus}" },
65+
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
66+
"targets": [
67+
{ "datasource": { "type": "prometheus", "uid": "${ds_prometheus}" }, "expr": "max by (namespace, label_cnpg_io_cluster) (cnpg_pg_replication_lag{namespace=~\"$namespace\"} * on(namespace,pod) group_left(label_cnpg_io_cluster) kube_pod_labels{label_cnpg_io_cluster!=\"\",namespace=~\"$namespace\"})", "legendFormat": "{{namespace}}/{{label_cnpg_io_cluster}}" }
68+
]
69+
},
70+
{
71+
"id": 4,
72+
"title": "Scaling limited (ScalingLimited=true)",
73+
"description": "Per KEDA HPA: 1 while the HPA reports ScalingLimited=true — set at BOTH the ceiling (maxReplicas) and the floor (an idle cluster's desired clamps UP to minReplicas), so this is 1 on quiet clusters too. Distinguish via the 'Desired vs current replicas' panel; the DatabaseAutoscalerAtMax alert scopes to the ceiling case (desired >= maxReplicas), where raising maxReplicas adds read capacity.",
74+
"type": "timeseries",
75+
"datasource": { "type": "prometheus", "uid": "${ds_prometheus}" },
76+
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
77+
"targets": [
78+
{ "datasource": { "type": "prometheus", "uid": "${ds_prometheus}" }, "expr": "kube_horizontalpodautoscaler_status_condition{horizontalpodautoscaler=~\"keda-hpa-.*\",condition=\"ScalingLimited\",status=\"true\",namespace=~\"$namespace\"}", "legendFormat": "{{namespace}}/{{horizontalpodautoscaler}}" }
79+
]
80+
},
81+
{
82+
"id": 5,
83+
"title": "Replica CPU (millicores, per cluster)",
84+
"description": "Replica-scoped postgres-container CPU per cluster (millicores) — the ReadCPUUtilization autoscaler input. For a cluster on metric: ReadCPUUtilization the recommendation is desired = 1 + ceil(this / target), clamped to [minReplicas..maxReplicas] (floor = max(minReplicas, maxSyncReplicas+1, 2)). Mirrors the ScaledObject's CPU query (rate over 5m x 1000) restricted to standby pods.",
85+
"type": "timeseries",
86+
"datasource": { "type": "prometheus", "uid": "${ds_prometheus}" },
87+
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
88+
"targets": [
89+
{ "datasource": { "type": "prometheus", "uid": "${ds_prometheus}" }, "expr": "sum by (namespace, label_cnpg_io_cluster) (rate(container_cpu_usage_seconds_total{container=\"postgres\",namespace=~\"$namespace\"}[5m]) * 1000 * on(namespace,pod) group_left(label_cnpg_io_cluster) kube_pod_labels{label_cnpg_io_cluster!=\"\",label_cnpg_io_instance_role=\"replica\",namespace=~\"$namespace\"})", "legendFormat": "{{namespace}}/{{label_cnpg_io_cluster}}" }
90+
]
91+
}
92+
]
93+
}

‎internal/operator/package_reconciler_test.go‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -204,6 +204,18 @@ func TestBuildHelmReleaseSpecZeroMaxHistory(t *testing.T) {
204204
}
205205
}
206206

207+
// NOTE: a TestBuildHelmReleaseSpecNoForceOrDriftDetection once lived here, claiming to
208+
// guard the postgres autoscaler's apply invariant. It was removed as misattributed:
209+
// buildHelmReleaseSpec below builds PLATFORM Package HelmReleases, whereas the tenant
210+
// Postgres Application HelmRelease is built in pkg/registry/apps/application/rest.go. The
211+
// autoscaler's real invariant is that the tenant HelmRelease applies CLIENT-SIDE (so
212+
// helm-controller patches the CNPG Cluster from the previous-vs-new rendered manifest
213+
// without consulting live state, the constant spec.instances seed yields no patch, and
214+
// KEDA's /scale value survives) —
215+
// server-side apply IS a HelmRelease knob (Install/Upgrade.ServerSideApply, set from the
216+
// release.cozystack.io/helm-server-side-apply annotation), and it is guarded by
217+
// pkg/registry/apps/application/rest_serversideapply_test.go, not here.
218+
207219
// TestPackageSourceCRDHasUpgradeCRDsEnum guards the generated CRD schema: the
208220
// invalid-value case from the spec is enforced at the API server via a
209221
// kubebuilder enum marker, not in the reconciler. If someone drops the marker

0 commit comments

Comments
 (0)