Skip to content

Commit 16390b9

Browse files
scooby87claude
andcommitted
Add design proposal: Database Horizontal Autoscaler
Co-Authored-By: Claude Opus 4.8 <[email protected]> Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 4ec8567 commit 16390b9

1 file changed

Lines changed: 194 additions & 0 deletions

File tree

  • design-proposals/database-horizontal-autoscaling
Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
# Database Horizontal Autoscaler for Cozystack
2+
3+
- **Title:** `Database Horizontal Autoscaler for Cozystack`
4+
- **Author(s):** `Alexey Artamonov <[email protected]>`
5+
- **Date:** `2026-07-08`
6+
- **Status:** Draft
7+
8+
## Overview
9+
10+
Managed databases in Cozystack (`postgres`, `mariadb`, `redis`, `mongodb`, and others) are scaled only manually today: an operator edits `replicas` in the application values and waits for the underlying operator to converge. This proposal introduces a dedicated operator, `db-autoscaler`, that automatically adjusts the number of **read replicas** of a managed database in response to load, driven by a new HPA-like custom resource `DatabaseHorizontalAutoscaler` (DHA).
11+
12+
The proposal is deliberately scoped to **horizontal scaling of read replicas**, because a stateful database primary cannot be scaled horizontally the way a stateless Deployment can. The autoscaler is topology-aware per engine, respects the synchronous-replica quorum, brakes on replication lag, and applies its decisions by patching the application's `replicas` value — so it never fights Flux/GitOps reconciliation.
13+
14+
## Scope and related proposals
15+
16+
This proposal covers **horizontal** autoscaling (read replicas) only. Two sibling axes are explicitly deferred to separate proposals:
17+
18+
- **Vertical autoscaling** — stepping the `resourcesPreset` ladder / in-place pod resize.
19+
- **Storage autoscaling** — automatic PVC expansion when a volume fills up.
20+
21+
Write-path scaling that requires data rebalancing (Kafka broker addition with partition reassignment, ClickHouse/MongoDB sharding) is out of scope for this proposal — it is an orchestrated procedure, not a counter change.
22+
23+
## Context
24+
25+
A managed database in Cozystack is an `Application` in the aggregated `apps.cozystack.io` API. The REST layer (`pkg/registry/apps/application/rest.go`, `ConvertApplicationToHelmRelease`) stores the application values in a Flux `HelmRelease.Spec.Values`, which Flux reconciles into the engine operator's custom resource (for example a CloudNativePG `Cluster`). Every managed database already exposes a horizontal knob in its values — `replicas` (or `kafka.replicas`, etc.) — and Cozystack already runs the observability the autoscaler needs:
26+
27+
- A per-database `WorkloadMonitor` (`cozystack.io/v1alpha1`, reconciled by `internal/controller/workloadmonitor_controller.go`) reports `status.availableReplicas`, `status.observedReplicas`, and `status.operational`, and already queries VictoriaMetrics over the vmselect Prometheus API.
28+
- Managed-app pods are labeled by the lineage webhook (`internal/lineagecontrollerwebhook/webhook.go`) with `apps.cozystack.io/application.{group,kind,name}` and `internal.cozystack.io/managed-by-cozystack: "true"`.
29+
- VictoriaMetrics (`packages/system/monitoring`) scrapes per-database metrics via `PodMonitor` (for PostgreSQL, `enablePodMonitor: true` on the CNPG `Cluster`).
30+
31+
### The problem
32+
33+
> "My database is saturated with read traffic during business hours and idle at night, but I have to notice it, hand-edit `replicas`, and hope I picked the right number — and undo it later."
34+
35+
There is no automated way to add or remove read replicas under load. Manually reusing a stock `HorizontalPodAutoscaler` does not work here: HPA scales via the scale subresource on the operator CR, which **fights Flux** (reconciliation restores `replicas` from the application values), and HPA has no notion of database topology — it would happily remove a synchronous standby or scale while replication lag is unbounded.
36+
37+
## Goals
38+
39+
- Automatically scale the number of read replicas for primary-replica engines: PostgreSQL (CNPG), MariaDB, Redis, MongoDB (replica set).
40+
- Apply all decisions by patching the `Application` values (Flux-compatible write path), never the operator CR directly.
41+
- Reuse existing telemetry (VictoriaMetrics + `WorkloadMonitor`); introduce no new exporters.
42+
- Be safe for stateful workloads: respect the replica quorum, brake on replication lag, drain read connections on scale-down, use long stabilization windows, and honor tenant quotas.
43+
- Provide HPA-like observability: status conditions, events, and a `dryRun` mode.
44+
45+
### Non-goals
46+
47+
- Vertical scaling (resources / presets).
48+
- Autoscaling the write path / the primary.
49+
- Engines that require data rebalancing (Kafka brokers, ClickHouse/MongoDB shards).
50+
- Cluster-node autoscaling (that is cluster-autoscaler's job).
51+
52+
## Design
53+
54+
### 1. Data flow
55+
56+
```mermaid
57+
flowchart LR
58+
DHA[DatabaseHorizontalAutoscaler CR] -- watch --> OP[db-autoscaler]
59+
OP -- HTTP /api/v1/query --> VM[(VictoriaMetrics<br/>vmselect-shortterm:8481)]
60+
WM[WorkloadMonitor status] -- operational / availableReplicas --> OP
61+
OP -- PATCH spec.replicas --> APP[Application<br/>apps.cozystack.io]
62+
APP -- REST layer --> HR[HelmRelease]
63+
HR -- Flux --> CR[Engine CR<br/>e.g. CNPG Cluster]
64+
CR --> PODS[(replica pods)]
65+
```
66+
67+
### 2. Topology adapters
68+
69+
Engine topology differs, so per-`kind` logic is isolated behind an adapter interface. Only primary-replica engines are scalable; sharded modes return `Scalable=false` with a reason.
70+
71+
```go
72+
type TopologyAdapter interface {
73+
ReplicasPath() string // "replicas" for pg/mariadb/redis/mongo
74+
QuorumFloor(appValues map[string]any) int32 // CNPG: quorum.minSyncReplicas + 1
75+
DriverQuery(app types.NamespacedName, k DriverKind) string // PromQL for read load
76+
ReplicationLagQuery(app types.NamespacedName) string // PromQL for lag (brake)
77+
ReadReplicaSelector(app types.NamespacedName) labels.Selector // for connection draining
78+
Scalable(appValues map[string]any) (bool, reason string) // false for sharded modes
79+
}
80+
```
81+
82+
MVP ships the `postgres` (CNPG) adapter. Follow-ups: `mariadb`, `redis`, `mongodb` (only when `sharding: false`). `clickhouse`, `kafka`, and sharded `mongodb` report `Scalable=false`.
83+
84+
### 3. Reconcile loop
85+
86+
1. Resolve `targetRef` → load the `Application` values and the linked `WorkloadMonitor`.
87+
2. Ask the adapter `Scalable`? If not → set condition `ScalingActive=False(reason)` and stop.
88+
3. If `operational=false` **or** a scale is still in flight (`availableReplicas != spec.replicas`) → freeze (single-flight) and requeue.
89+
4. Query VictoriaMetrics for the driver metric and the replication lag.
90+
5. Compute `desired = ceil(currentReplicas * currentMetric / targetMetric)` with a tolerance dead-zone.
91+
6. Apply guardrails (see below): clamp to `[min,max]`, quorum floor, lag brake, stabilization windows, step limit, tenant quota.
92+
7. If `desired != current` and the decision passes → PATCH `spec.replicas` on the `Application` (with `RetryOnConflict`); on scale-down, drain read connections first.
93+
8. Update `status`, set `lastScaleTime`, emit an Event.
94+
95+
### 4. Guardrails (normative)
96+
97+
- `min ≤ desired ≤ max`; at most `behavior.*.step` replicas per decision.
98+
- `desired ≥ QuorumFloor(app)` when `respectQuorum` is set (CNPG: `quorum.minSyncReplicas + 1`).
99+
- **Lag brake:** replication lag above `maxReplicationLagSeconds` forbids both scale-down and scale-up; condition `AbleToScale=False`.
100+
- **Cooldown / stabilization:** separate windows for scale-up and (longer) scale-down; scale-down only when the signal held for the whole window.
101+
- **Single-flight:** one change at a time; the next decision only after `operational=true && availableReplicas == spec.replicas`.
102+
- **Tenant quota:** the new replica count × preset resources must fit the tenant quota; otherwise `ScalingLimited=True`.
103+
- **Fail-safe freeze:** if vmselect is unreachable or the metric is missing, do not scale (never scale blind); alert.
104+
- **dryRun:** decisions are written to status/events but no patch is applied.
105+
106+
## User-facing changes
107+
108+
A new namespaced CRD, `DatabaseHorizontalAutoscaler` (group `autoscaling.cozystack.io/v1alpha1`), created by a tenant next to their database application:
109+
110+
```yaml
111+
apiVersion: autoscaling.cozystack.io/v1alpha1
112+
kind: DatabaseHorizontalAutoscaler
113+
metadata: { name: db, namespace: tenant-acme }
114+
spec:
115+
targetRef: { kind: Postgres, name: db } # apiGroup defaults to apps.cozystack.io
116+
minReplicas: 2
117+
maxReplicas: 6
118+
metrics:
119+
- type: ReadConnections # | ReadCPUUtilization | CustomQuery
120+
target: { averageValue: "150" } # per replica
121+
behavior:
122+
scaleUp: { stabilizationWindowSeconds: 300, step: 1 }
123+
scaleDown: { stabilizationWindowSeconds: 1800, step: 1 }
124+
constraints:
125+
respectQuorum: true
126+
maxReplicationLagSeconds: 30
127+
drainReadConnections: true
128+
dryRun: false
129+
status:
130+
currentReplicas: 3
131+
desiredReplicas: 4
132+
lastScaleTime: "..."
133+
currentMetrics: [ { type: ReadConnections, averageValue: "210" } ]
134+
conditions: [ ScalingActive, AbleToScale, ScalingLimited ]
135+
```
136+
137+
The dashboard can surface the DHA status and scaling events like it does for other application sub-resources. When no DHA references an application, nothing changes.
138+
139+
## Upgrade and rollback compatibility
140+
141+
- **Opt-in and off by default.** The operator ships as an optional platform package, enabled via `bundles.enabledPackages`. Existing clusters, manifests, and APIs are unaffected until a tenant creates a DHA.
142+
- **Ownership.** While a DHA is active, `replicas` is considered owned by the autoscaler. This is mutually exclusive with managing `replicas` from a tenant GitOps repo for the same application; documented explicitly.
143+
- **Rollback.** Deleting the DHA stops all autoscaling immediately and leaves the application at its current `replicas`. Disabling the package removes the operator; no data migration is involved and the change is fully reversible.
144+
145+
## Security
146+
147+
- **RBAC.** The operator needs: read DHA and read/patch `applications.apps.cozystack.io`, read `workloadmonitors.cozystack.io`, read `pods`, and read-only HTTP to vmselect. It has no direct access to engine operator CRs or Flux `HelmRelease` objects — only the aggregated apps API.
148+
- **Multi-tenancy.** DHA is namespaced and lives in the tenant namespace. The `databasehorizontalautoscalers.autoscaling.cozystack.io` resource is granted in the tenant ClusterRole tiers exactly like `apps.cozystack.io` (ServiceAccount full, human `view` read-only, `admin`/`super-admin` write). A tenant can only autoscale its own applications, and scale-up is validated against the tenant quota.
149+
- **New inputs.** The only tenant-supplied inputs are the DHA fields, all bounded and schema-validated (`minReplicas`/`maxReplicas`, metric targets, windows). No new secrets are stored or transmitted.
150+
151+
## Failure and edge cases
152+
153+
- vmselect unreachable or metric missing → the autoscaler freezes (no scaling) and surfaces `AbleToScale=False`; alert fires.
154+
- Replication lag above the configured threshold → no scale-up and no scale-down until lag recovers.
155+
- Scale still in flight (`availableReplicas != spec.replicas`) → single-flight; the next decision waits for convergence, preventing thrashing.
156+
- Target is a sharded engine (e.g. ClickHouse, or MongoDB with `sharding: true`) → `ScalingActive=False` with a clear reason; no action.
157+
- Desired count would drop below the quorum floor → clamped to the floor; `ScalingLimited=True`.
158+
- Tenant quota exceeded on scale-up → clamped; `ScalingLimited=True`.
159+
160+
## Testing
161+
162+
- **Unit:** reconcile decisions and each `TopologyAdapter` with mocked VictoriaMetrics and a mocked Application client (`go test ./internal/controller/...`).
163+
- **Codegen:** `make generate` produces the CRD and deepcopy without errors.
164+
- **Manual (dev cluster, CNPG postgres):** create a DHA with `dryRun: true` → decisions appear in `status`/Events, replicas unchanged. Then disable `dryRun` under read load → the `Application`'s `spec.replicas` grows, CNPG adds a read replica, Flux does not revert; on load decrease and after the window, scale-down drains connections and never drops below quorum.
165+
- **Negative:** vmselect down → freeze; lag above threshold → no scaling; DHA targeting a sharded ClickHouse → `ScalingActive=False`.
166+
167+
## Rollout
168+
169+
1. **PoC** — CNPG PostgreSQL on a dev cluster: DHA + `replicas` patch driven by `ReadConnections`; confirm Flux does not revert and reads route to `<release>-ro`.
170+
2. **MVP** — the operator plus the `postgres` adapter, full guardrails (quorum, lag, cooldown, quota), `dryRun`, dashboard surface and events. Shipped as an optional `paas`-bundle package.
171+
3. **Adapter expansion** — `mariadb` → `redis` → `mongodb` (replica set).
172+
4. **Observability & policy** — Grafana dashboard of scaling decisions, alerts for "limit reached / freeze".
173+
174+
## Open questions
175+
176+
- Adapter order after MVP: `mariadb` → `redis` → `mongodb`?
177+
- Default driver metric: read connections, read QPS, or replica CPU (to be calibrated on real workloads)?
178+
- Is scale-down enabled by default, or scale-up only (down conservative/manual)?
179+
- Ownership policy for `replicas` when the application is also managed from a tenant GitOps repo.
180+
181+
## Alternatives considered
182+
183+
- **A controller inside `cozystack-controller`** instead of a standalone operator. It would reuse the existing binary, RBAC, and VictoriaMetrics helper, at the cost of coupling the autoscaler's lifecycle to the platform controller. Rejected in favor of a standalone operator for isolation and an independent release cadence; the logic can be moved later if desired.
184+
- **Patching the `HelmRelease` (or engine CR) directly.** Rejected: a direct patch to the operator CR is reverted by Flux, and a `HelmRelease.Spec.Values` patch can be clobbered when the Application REST layer regenerates the release. Patching the `Application` is the canonical, reconciliation-safe path.
185+
- **Stock HPA + KEDA.** Rejected as the primary mechanism: it fights Flux, is topology-unaware, and offers no safe scale-down (draining, quorum, lag). A KEDA/PromQL trigger can still be reused *as a metric source inside* the operator (`CustomQuery`).
186+
- **Scaling the write path via sharding.** Out of scope: it requires data rebalancing (Cruise Control for Kafka, resharding for ClickHouse/MongoDB), which is an orchestrated procedure rather than a replica-count change.
187+
188+
---
189+
190+
<!--
191+
Inspired by KubeVirt enhancement proposals
192+
(https://github.com/kubevirt/enhancements) and Kubernetes Enhancement
193+
Proposals (KEPs).
194+
-->

0 commit comments

Comments
 (0)