|
| 1 | +# Database Horizontal Autoscaler for Cozystack |
| 2 | + |
| 3 | +- **Title:** `Database Horizontal Autoscaler for Cozystack` |
| 4 | +- **Author(s): ** `Alexey Artamonov <[email protected]>` |
| 5 | +- **Date:** `2026-07-08` |
| 6 | +- **Status:** Draft |
| 7 | + |
| 8 | +## Overview |
| 9 | + |
| 10 | +Managed databases in Cozystack (`postgres`, `mariadb`, `redis`, `mongodb`, and others) are scaled only manually today: an operator edits `replicas` in the application values and waits for the underlying operator to converge. This proposal introduces a dedicated operator, `db-autoscaler`, that automatically adjusts the number of **read replicas** of a managed database in response to load, driven by a new HPA-like custom resource `DatabaseHorizontalAutoscaler` (DHA). |
| 11 | + |
| 12 | +The proposal is deliberately scoped to **horizontal scaling of read replicas**, because a stateful database primary cannot be scaled horizontally the way a stateless Deployment can. The autoscaler is topology-aware per engine, respects the synchronous-replica quorum, brakes on replication lag, and applies its decisions by patching the application's `replicas` value — so it never fights Flux/GitOps reconciliation. |
| 13 | + |
| 14 | +## Scope and related proposals |
| 15 | + |
| 16 | +This proposal covers **horizontal** autoscaling (read replicas) only. Two sibling axes are explicitly deferred to separate proposals: |
| 17 | + |
| 18 | +- **Vertical autoscaling** — stepping the `resourcesPreset` ladder / in-place pod resize. |
| 19 | +- **Storage autoscaling** — automatic PVC expansion when a volume fills up. |
| 20 | + |
| 21 | +Write-path scaling that requires data rebalancing (Kafka broker addition with partition reassignment, ClickHouse/MongoDB sharding) is out of scope for this proposal — it is an orchestrated procedure, not a counter change. |
| 22 | + |
| 23 | +## Context |
| 24 | + |
| 25 | +A managed database in Cozystack is an `Application` in the aggregated `apps.cozystack.io` API. The REST layer (`pkg/registry/apps/application/rest.go`, `ConvertApplicationToHelmRelease`) stores the application values in a Flux `HelmRelease.Spec.Values`, which Flux reconciles into the engine operator's custom resource (for example a CloudNativePG `Cluster`). Every managed database already exposes a horizontal knob in its values — `replicas` (or `kafka.replicas`, etc.) — and Cozystack already runs the observability the autoscaler needs: |
| 26 | + |
| 27 | +- A per-database `WorkloadMonitor` (`cozystack.io/v1alpha1`, reconciled by `internal/controller/workloadmonitor_controller.go`) reports `status.availableReplicas`, `status.observedReplicas`, and `status.operational`, and already queries VictoriaMetrics over the vmselect Prometheus API. |
| 28 | +- Managed-app pods are labeled by the lineage webhook (`internal/lineagecontrollerwebhook/webhook.go`) with `apps.cozystack.io/application.{group,kind,name}` and `internal.cozystack.io/managed-by-cozystack: "true"`. |
| 29 | +- VictoriaMetrics (`packages/system/monitoring`) scrapes per-database metrics via `PodMonitor` (for PostgreSQL, `enablePodMonitor: true` on the CNPG `Cluster`). |
| 30 | + |
| 31 | +### The problem |
| 32 | + |
| 33 | +> "My database is saturated with read traffic during business hours and idle at night, but I have to notice it, hand-edit `replicas`, and hope I picked the right number — and undo it later." |
| 34 | +
|
| 35 | +There is no automated way to add or remove read replicas under load. Manually reusing a stock `HorizontalPodAutoscaler` does not work here: HPA scales via the scale subresource on the operator CR, which **fights Flux** (reconciliation restores `replicas` from the application values), and HPA has no notion of database topology — it would happily remove a synchronous standby or scale while replication lag is unbounded. |
| 36 | + |
| 37 | +## Goals |
| 38 | + |
| 39 | +- Automatically scale the number of read replicas for primary-replica engines: PostgreSQL (CNPG), MariaDB, Redis, MongoDB (replica set). |
| 40 | +- Apply all decisions by patching the `Application` values (Flux-compatible write path), never the operator CR directly. |
| 41 | +- Reuse existing telemetry (VictoriaMetrics + `WorkloadMonitor`); introduce no new exporters. |
| 42 | +- Be safe for stateful workloads: respect the replica quorum, brake on replication lag, drain read connections on scale-down, use long stabilization windows, and honor tenant quotas. |
| 43 | +- Provide HPA-like observability: status conditions, events, and a `dryRun` mode. |
| 44 | + |
| 45 | +### Non-goals |
| 46 | + |
| 47 | +- Vertical scaling (resources / presets). |
| 48 | +- Autoscaling the write path / the primary. |
| 49 | +- Engines that require data rebalancing (Kafka brokers, ClickHouse/MongoDB shards). |
| 50 | +- Cluster-node autoscaling (that is cluster-autoscaler's job). |
| 51 | + |
| 52 | +## Design |
| 53 | + |
| 54 | +### 1. Data flow |
| 55 | + |
| 56 | +```mermaid |
| 57 | +flowchart LR |
| 58 | + DHA[DatabaseHorizontalAutoscaler CR] -- watch --> OP[db-autoscaler] |
| 59 | + OP -- HTTP /api/v1/query --> VM[(VictoriaMetrics<br/>vmselect-shortterm:8481)] |
| 60 | + WM[WorkloadMonitor status] -- operational / availableReplicas --> OP |
| 61 | + OP -- PATCH spec.replicas --> APP[Application<br/>apps.cozystack.io] |
| 62 | + APP -- REST layer --> HR[HelmRelease] |
| 63 | + HR -- Flux --> CR[Engine CR<br/>e.g. CNPG Cluster] |
| 64 | + CR --> PODS[(replica pods)] |
| 65 | +``` |
| 66 | + |
| 67 | +### 2. Topology adapters |
| 68 | + |
| 69 | +Engine topology differs, so per-`kind` logic is isolated behind an adapter interface. Only primary-replica engines are scalable; sharded modes return `Scalable=false` with a reason. |
| 70 | + |
| 71 | +```go |
| 72 | +type TopologyAdapter interface { |
| 73 | + ReplicasPath() string // "replicas" for pg/mariadb/redis/mongo |
| 74 | + QuorumFloor(appValues map[string]any) int32 // CNPG: quorum.minSyncReplicas + 1 |
| 75 | + DriverQuery(app types.NamespacedName, k DriverKind) string // PromQL for read load |
| 76 | + ReplicationLagQuery(app types.NamespacedName) string // PromQL for lag (brake) |
| 77 | + ReadReplicaSelector(app types.NamespacedName) labels.Selector // for connection draining |
| 78 | + Scalable(appValues map[string]any) (bool, reason string) // false for sharded modes |
| 79 | +} |
| 80 | +``` |
| 81 | + |
| 82 | +MVP ships the `postgres` (CNPG) adapter. Follow-ups: `mariadb`, `redis`, `mongodb` (only when `sharding: false`). `clickhouse`, `kafka`, and sharded `mongodb` report `Scalable=false`. |
| 83 | + |
| 84 | +### 3. Reconcile loop |
| 85 | + |
| 86 | +1. Resolve `targetRef` → load the `Application` values and the linked `WorkloadMonitor`. |
| 87 | +2. Ask the adapter `Scalable`? If not → set condition `ScalingActive=False(reason)` and stop. |
| 88 | +3. If `operational=false` **or** a scale is still in flight (`availableReplicas != spec.replicas`) → freeze (single-flight) and requeue. |
| 89 | +4. Query VictoriaMetrics for the driver metric and the replication lag. |
| 90 | +5. Compute `desired = ceil(currentReplicas * currentMetric / targetMetric)` with a tolerance dead-zone. |
| 91 | +6. Apply guardrails (see below): clamp to `[min,max]`, quorum floor, lag brake, stabilization windows, step limit, tenant quota. |
| 92 | +7. If `desired != current` and the decision passes → PATCH `spec.replicas` on the `Application` (with `RetryOnConflict`); on scale-down, drain read connections first. |
| 93 | +8. Update `status`, set `lastScaleTime`, emit an Event. |
| 94 | + |
| 95 | +### 4. Guardrails (normative) |
| 96 | + |
| 97 | +- `min ≤ desired ≤ max`; at most `behavior.*.step` replicas per decision. |
| 98 | +- `desired ≥ QuorumFloor(app)` when `respectQuorum` is set (CNPG: `quorum.minSyncReplicas + 1`). |
| 99 | +- **Lag brake:** replication lag above `maxReplicationLagSeconds` forbids both scale-down and scale-up; condition `AbleToScale=False`. |
| 100 | +- **Cooldown / stabilization:** separate windows for scale-up and (longer) scale-down; scale-down only when the signal held for the whole window. |
| 101 | +- **Single-flight:** one change at a time; the next decision only after `operational=true && availableReplicas == spec.replicas`. |
| 102 | +- **Tenant quota:** the new replica count × preset resources must fit the tenant quota; otherwise `ScalingLimited=True`. |
| 103 | +- **Fail-safe freeze:** if vmselect is unreachable or the metric is missing, do not scale (never scale blind); alert. |
| 104 | +- **dryRun:** decisions are written to status/events but no patch is applied. |
| 105 | + |
| 106 | +## User-facing changes |
| 107 | + |
| 108 | +A new namespaced CRD, `DatabaseHorizontalAutoscaler` (group `autoscaling.cozystack.io/v1alpha1`), created by a tenant next to their database application: |
| 109 | + |
| 110 | +```yaml |
| 111 | +apiVersion: autoscaling.cozystack.io/v1alpha1 |
| 112 | +kind: DatabaseHorizontalAutoscaler |
| 113 | +metadata: { name: db, namespace: tenant-acme } |
| 114 | +spec: |
| 115 | + targetRef: { kind: Postgres, name: db } # apiGroup defaults to apps.cozystack.io |
| 116 | + minReplicas: 2 |
| 117 | + maxReplicas: 6 |
| 118 | + metrics: |
| 119 | + - type: ReadConnections # | ReadCPUUtilization | CustomQuery |
| 120 | + target: { averageValue: "150" } # per replica |
| 121 | + behavior: |
| 122 | + scaleUp: { stabilizationWindowSeconds: 300, step: 1 } |
| 123 | + scaleDown: { stabilizationWindowSeconds: 1800, step: 1 } |
| 124 | + constraints: |
| 125 | + respectQuorum: true |
| 126 | + maxReplicationLagSeconds: 30 |
| 127 | + drainReadConnections: true |
| 128 | + dryRun: false |
| 129 | +status: |
| 130 | + currentReplicas: 3 |
| 131 | + desiredReplicas: 4 |
| 132 | + lastScaleTime: "..." |
| 133 | + currentMetrics: [ { type: ReadConnections, averageValue: "210" } ] |
| 134 | + conditions: [ ScalingActive, AbleToScale, ScalingLimited ] |
| 135 | +``` |
| 136 | +
|
| 137 | +The dashboard can surface the DHA status and scaling events like it does for other application sub-resources. When no DHA references an application, nothing changes. |
| 138 | +
|
| 139 | +## Upgrade and rollback compatibility |
| 140 | +
|
| 141 | +- **Opt-in and off by default.** The operator ships as an optional platform package, enabled via `bundles.enabledPackages`. Existing clusters, manifests, and APIs are unaffected until a tenant creates a DHA. |
| 142 | +- **Ownership.** While a DHA is active, `replicas` is considered owned by the autoscaler. This is mutually exclusive with managing `replicas` from a tenant GitOps repo for the same application; documented explicitly. |
| 143 | +- **Rollback.** Deleting the DHA stops all autoscaling immediately and leaves the application at its current `replicas`. Disabling the package removes the operator; no data migration is involved and the change is fully reversible. |
| 144 | + |
| 145 | +## Security |
| 146 | + |
| 147 | +- **RBAC.** The operator needs: read DHA and read/patch `applications.apps.cozystack.io`, read `workloadmonitors.cozystack.io`, read `pods`, and read-only HTTP to vmselect. It has no direct access to engine operator CRs or Flux `HelmRelease` objects — only the aggregated apps API. |
| 148 | +- **Multi-tenancy.** DHA is namespaced and lives in the tenant namespace. The `databasehorizontalautoscalers.autoscaling.cozystack.io` resource is granted in the tenant ClusterRole tiers exactly like `apps.cozystack.io` (ServiceAccount full, human `view` read-only, `admin`/`super-admin` write). A tenant can only autoscale its own applications, and scale-up is validated against the tenant quota. |
| 149 | +- **New inputs.** The only tenant-supplied inputs are the DHA fields, all bounded and schema-validated (`minReplicas`/`maxReplicas`, metric targets, windows). No new secrets are stored or transmitted. |
| 150 | + |
| 151 | +## Failure and edge cases |
| 152 | + |
| 153 | +- vmselect unreachable or metric missing → the autoscaler freezes (no scaling) and surfaces `AbleToScale=False`; alert fires. |
| 154 | +- Replication lag above the configured threshold → no scale-up and no scale-down until lag recovers. |
| 155 | +- Scale still in flight (`availableReplicas != spec.replicas`) → single-flight; the next decision waits for convergence, preventing thrashing. |
| 156 | +- Target is a sharded engine (e.g. ClickHouse, or MongoDB with `sharding: true`) → `ScalingActive=False` with a clear reason; no action. |
| 157 | +- Desired count would drop below the quorum floor → clamped to the floor; `ScalingLimited=True`. |
| 158 | +- Tenant quota exceeded on scale-up → clamped; `ScalingLimited=True`. |
| 159 | + |
| 160 | +## Testing |
| 161 | + |
| 162 | +- **Unit:** reconcile decisions and each `TopologyAdapter` with mocked VictoriaMetrics and a mocked Application client (`go test ./internal/controller/...`). |
| 163 | +- **Codegen:** `make generate` produces the CRD and deepcopy without errors. |
| 164 | +- **Manual (dev cluster, CNPG postgres):** create a DHA with `dryRun: true` → decisions appear in `status`/Events, replicas unchanged. Then disable `dryRun` under read load → the `Application`'s `spec.replicas` grows, CNPG adds a read replica, Flux does not revert; on load decrease and after the window, scale-down drains connections and never drops below quorum. |
| 165 | +- **Negative:** vmselect down → freeze; lag above threshold → no scaling; DHA targeting a sharded ClickHouse → `ScalingActive=False`. |
| 166 | + |
| 167 | +## Rollout |
| 168 | + |
| 169 | +1. **PoC** — CNPG PostgreSQL on a dev cluster: DHA + `replicas` patch driven by `ReadConnections`; confirm Flux does not revert and reads route to `<release>-ro`. |
| 170 | +2. **MVP** — the operator plus the `postgres` adapter, full guardrails (quorum, lag, cooldown, quota), `dryRun`, dashboard surface and events. Shipped as an optional `paas`-bundle package. |
| 171 | +3. **Adapter expansion** — `mariadb` → `redis` → `mongodb` (replica set). |
| 172 | +4. **Observability & policy** — Grafana dashboard of scaling decisions, alerts for "limit reached / freeze". |
| 173 | + |
| 174 | +## Open questions |
| 175 | + |
| 176 | +- Adapter order after MVP: `mariadb` → `redis` → `mongodb`? |
| 177 | +- Default driver metric: read connections, read QPS, or replica CPU (to be calibrated on real workloads)? |
| 178 | +- Is scale-down enabled by default, or scale-up only (down conservative/manual)? |
| 179 | +- Ownership policy for `replicas` when the application is also managed from a tenant GitOps repo. |
| 180 | + |
| 181 | +## Alternatives considered |
| 182 | + |
| 183 | +- **A controller inside `cozystack-controller`** instead of a standalone operator. It would reuse the existing binary, RBAC, and VictoriaMetrics helper, at the cost of coupling the autoscaler's lifecycle to the platform controller. Rejected in favor of a standalone operator for isolation and an independent release cadence; the logic can be moved later if desired. |
| 184 | +- **Patching the `HelmRelease` (or engine CR) directly.** Rejected: a direct patch to the operator CR is reverted by Flux, and a `HelmRelease.Spec.Values` patch can be clobbered when the Application REST layer regenerates the release. Patching the `Application` is the canonical, reconciliation-safe path. |
| 185 | +- **Stock HPA + KEDA.** Rejected as the primary mechanism: it fights Flux, is topology-unaware, and offers no safe scale-down (draining, quorum, lag). A KEDA/PromQL trigger can still be reused *as a metric source inside* the operator (`CustomQuery`). |
| 186 | +- **Scaling the write path via sharding.** Out of scope: it requires data rebalancing (Cruise Control for Kafka, resharding for ClickHouse/MongoDB), which is an orchestrated procedure rather than a replica-count change. |
| 187 | + |
| 188 | +--- |
| 189 | + |
| 190 | +<!-- |
| 191 | +Inspired by KubeVirt enhancement proposals |
| 192 | +(https://github.com/kubevirt/enhancements) and Kubernetes Enhancement |
| 193 | +Proposals (KEPs). |
| 194 | +--> |
0 commit comments