Skip to content

Commit da90eca

Browse files
committed
Address 2026-07-16 review: SSA-on-aggregated-API, HA, lag metric, RBAC, deps
From @IvanHunters CHANGES_REQUESTED (2026-07-16): - Ownership/Testing/Open questions: flag that field-level SSA is unconfirmed for the hand-written aggregated apps API (rest.Patcher, not a CRD); require an apiserver spike; webhook becomes mandatory if SSA field-tracking absent. - Security/Rollout: operator is a single active reconciler (replicas: 1 + leader-election), no active/active race on marker/single-flight/rollback. - Guardrails/adapter: use CNPG gauge cnpg_pg_replication_lag; fix replay_lsn; drop PromQL-calls-SQL error. - Security: DHA ships package-owned ClusterRoles with rbac.cozystack.io/aggregate-to-tenant* labels (not editing cozystack-basics). - Rollout: declare hard PackageSource.dependsOn on the monitoring stack. - Metadata: Author(s) -> @scooby87 (repo handle convention). Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 6d99743 commit da90eca

1 file changed

Lines changed: 9 additions & 8 deletions

File tree

  • design-proposals/database-horizontal-autoscaling

‎design-proposals/database-horizontal-autoscaling/README.md‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Database Horizontal Autoscaler for Cozystack
22

33
- **Title:** `Database Horizontal Autoscaler for Cozystack`
4-
- **Author(s):** `Alexey Artamonov <[email protected]>`
4+
- **Author(s):** `@scooby87`
55
- **Date:** `2026-07-08` (addressing review by `@IvanHunters`, Gemini, and CodeRabbit)
66
- **Status:** Draft
77

@@ -84,7 +84,7 @@ type TopologyAdapter interface {
8484
PrimaryCount() int32 // CNPG: 1 (non-read-serving instances)
8585
QuorumFloor(appValues map[string]any) int32 // CNPG: quorum.maxSyncReplicas + 1
8686
DriverQuery(app types.NamespacedName, k DriverKind) string // PromQL for read load (per read replica)
87-
ReplicationLagQuery(app types.NamespacedName) string // PromQL for lag (brake), write-activity gated
87+
ReplicationLagQuery(app types.NamespacedName) string // e.g. cnpg_pg_replication_lag gauge; write-activity gated
8888
Scalable(appValues map[string]any) (bool, reason string) // false for sharded modes
8989
}
9090
```
@@ -107,7 +107,7 @@ MVP ships the `postgres` (CNPG) adapter. Follow-ups: `mariadb`, `redis`, `mongod
107107
- `min ≤ desired ≤ max`; at most `behavior.*.step` replicas per decision.
108108
- `desired ≥ QuorumFloor(app)`. For CNPG the floor is `maxSyncReplicas + 1`: the chart documents `maxSyncReplicas` as "must be less than total replicas", and dropping to/below it makes CNPG cap/reject the change and can starve synchronous commits (writes stall). The floor also never leaves fewer than `minSyncReplicas` standbys available. Pin this to the CNPG version cozystack ships, since the sync-replica API changed across versions.
109109
- **Precedence — quota > quorum floor > min/max.** `maxSyncReplicas` is tenant-mutable after the DHA is created, so at runtime `QuorumFloor` (`maxSyncReplicas + 1`) may exceed `minReplicas`/`maxReplicas`, and may even exceed what the tenant quota permits. The resolution order is fixed and unambiguous: (1) the **tenant quota is a hard ceiling and is never exceeded**; (2) subject to that, the **quorum floor wins** over `minReplicas`/`maxReplicas` — `desired` is clamped *up* to the floor (even above `maxReplicas`), never letting `min`/`max` push the cluster below a safe quorum. When these two rules collide irreconcilably — the quorum floor does not fit the quota (raised `maxSyncReplicas` + tight quota) — the operator does **not** patch and freezes with `ScalingLimited=True` reason `QuorumExceedsQuota` (alert), rather than exceeding quota (which would only hit the StuckScaling path) or scaling below a safe quorum.
110-
- **Lag brake:** replication lag above `maxReplicationLagSeconds` forbids both scale-down and scale-up (`AbleToScale=False`). The lag signal is **write-activity gated** — it uses byte-based lag (`pg_wal_lsn_diff(pg_current_wal_lsn(), replayed_lsn)`) and/or only trusts the seconds-based value while the primary LSN is advancing, so an idle primary does not produce a false freeze during the low-load windows scale-down targets.
110+
- **Lag brake:** replication lag above `maxReplicationLagSeconds` forbids both scale-down and scale-up (`AbleToScale=False`). The signal is the CNPG-exported gauge **`cnpg_pg_replication_lag`** (seconds), already scraped into VictoriaMetrics and used by cozystack's own CNPG dashboards and alerts (`dashboards/db/cloudnativepg.json`, `packages/system/postgres-operator/alerts/`), so no custom query is added. Because that seconds value keeps climbing on a write-idle primary, the brake is **write-activity gated**: it is honoured only while the primary's WAL position is advancing (from CNPG's exported current-vs-`replay_lsn` LSN metrics), so an idle primary does not produce a false freeze during the low-load windows scale-down targets.
111111
- **Cooldown / stabilization:** separate windows for scale-up and (longer) scale-down; scale-down only when the signal held for the whole window.
112112
- **Single-flight with convergence deadline:** one change at a time; the next decision only after `operational=true && availableReplicas == replicas`. Because that gate can never clear if a scale-up cannot converge — a new standby rejected by ResourceQuota admission, an unbindable PVC, or an unschedulable pod — a patched change must reach convergence within `behavior.convergenceDeadlineSeconds` (default a small multiple of the scale-up window). On timeout the operator surfaces `AbleToScale=False` with reason `StuckScaling`, alerts, and **rolls `replicas` back to `status.lastConvergedReplicas`**, releasing single-flight so a subsequent scale-down (which may itself relieve the pressure) is not blocked. `status.lastConvergedReplicas` records the last count that reached `availableReplicas == replicas`. Note the tenant-quota pre-check is advisory (a concurrent allocation can consume quota between check and pod creation), so this stuck path is reachable in practice, not just in theory.
113113
- **Tenant quota:** the new replica count × preset resources must fit the tenant quota; otherwise `ScalingLimited=True`.
@@ -152,13 +152,14 @@ When several `metrics` are set, the desired count is the **maximum** of the per-
152152
## Upgrade and rollback compatibility
153153

154154
- **Opt-in and off by default.** The operator ships as an optional platform package, enabled via `bundles.enabledPackages`. Existing clusters, manifests, and APIs are unaffected until a tenant creates a DHA.
155-
- **Ownership (enforced, not advisory).** While a DHA is active, the autoscaler is the **single explicit owner** of the application's `replicas` value. Enforcement: the operator writes `replicas` via **server-side apply with a dedicated field manager (`db-autoscaler`)** and stamps a marker annotation `autoscaling.cozystack.io/managed-by: <dha-name>` on the `Application`. A competing declarative writer (Flux from a tenant GitOps repo, a human edit) that also claims `replicas` produces an SSA field-manager conflict, which the operator surfaces as a `ScalingLimited`/conflict condition rather than silently fighting. `RetryOnConflict` handles only API-level write races on a single writer — it is **not** the ownership mechanism. **This SSA guarantee holds only against writers that do not force-apply:** a tenant GitOps Flux `Kustomization` with `spec.force: true` (a common setting) takes over the `replicas` managed-fields entry, so the autoscaler's next (non-force) apply is the one that hits the conflict — i.e. the autoscaler loses ownership rather than the competitor. Because SSA alone cannot win against a force-applier, a **validating admission webhook** that rejects conflicting `replicas` writes for DHA-managed applications is **recommended** to close this case deterministically (tracked in Open questions), not merely optional.
155+
- **Ownership (enforced, not advisory).** While a DHA is active, the autoscaler is the **single explicit owner** of the application's `replicas` value. Enforcement: the operator writes `replicas` via **server-side apply with a dedicated field manager (`db-autoscaler`)** and stamps a marker annotation `autoscaling.cozystack.io/managed-by: <dha-name>` on the `Application`. A competing declarative writer (Flux from a tenant GitOps repo, a human edit) that also claims `replicas` produces an SSA field-manager conflict, which the operator surfaces as a `ScalingLimited`/conflict condition rather than silently fighting. `RetryOnConflict` handles only API-level write races on a single writer — it is **not** the ownership mechanism. **This SSA guarantee holds only against writers that do not force-apply:** a tenant GitOps Flux `Kustomization` with `spec.force: true` (a common setting) takes over the `replicas` managed-fields entry, so the autoscaler's next (non-force) apply is the one that hits the conflict — i.e. the autoscaler loses ownership rather than the competitor. Because SSA alone cannot win against a force-applier, a **validating admission webhook** that rejects conflicting `replicas` writes for DHA-managed applications is **recommended** to close this case deterministically (tracked in Open questions), not merely optional. **Caveat — field-level SSA is not yet confirmed for this API:** `Application` is served by a hand-written `rest.Patcher` (`pkg/registry/apps/application/rest.go`), not a CRD, and its existing conflict test (`rest_conflict_test.go`) covers only `RetryOnConflict` on the backing `HelmRelease` resourceVersion — not per-field managed-fields SSA. If the aggregated Patch handler does not track `.spec.replicas` at field granularity, this ownership model and the `lastConvergedReplicas` rollback silently degrade to advisory. A spike against a real API server (see Testing) must confirm field-level SSA before MVP; **if it does not hold, the admission webhook is mandatory, not merely recommended.**
156156
- **Rollback.** Deleting the DHA stops all autoscaling immediately (and clears the marker), leaving the application at its current `replicas`. Disabling the package removes the operator; no data migration is involved and the change is fully reversible.
157157

158158
## Security
159159

160160
- **RBAC.** The operator needs: read DHA and read/patch `applications.apps.cozystack.io`; read `workloadmonitors.cozystack.io`; read `pods`; read `resourcequotas` (core) for the tenant-quota guardrail; and read-only HTTP to vmselect. The `resourcesPreset → resources` mapping is a static table compiled into the operator from the published cozy-lib preset ladder, so no cluster read of preset definitions is required. The operator has **no** write access to Pods, Services, or Endpoints, **no** exec, and **no** direct access to engine operator CRs or Flux `HelmRelease` objects — only the aggregated apps API.
161-
- **Multi-tenancy.** DHA is namespaced and lives in the tenant namespace. The `databasehorizontalautoscalers.autoscaling.cozystack.io` resource is granted in the tenant ClusterRole tiers exactly like `apps.cozystack.io` (ServiceAccount full, human `view` read-only, `admin`/`super-admin` write). A tenant can only autoscale its own applications, and scale-up is validated against the tenant quota.
161+
- **Multi-tenancy.** DHA is namespaced and lives in the tenant namespace. Tenant access is granted through the platform's RBAC-aggregation mechanism: the DHA package **ships its own self-contained ClusterRoles** labelled `rbac.cozystack.io/aggregate-to-tenant[-view|-admin|-super-admin]: "true"` (per `packages/system/cozystack-basics/templates/clusterroles.yaml`) — it does **not** edit the shared `cozystack-basics` file, whose write tiers grant apps via a hard-coded per-kind allowlist rather than a wildcard. This gives the tenant ServiceAccount full access, human `view` read-only, and `admin`/`super-admin` write on `databasehorizontalautoscalers.autoscaling.cozystack.io`. A tenant can only autoscale its own applications, and scale-up is validated against the tenant quota.
162+
- **Single active reconciler (HA).** The operator runs as `replicas: 1` with controller-runtime leader-election enabled; per-target reconcile state (the `managed-by` marker, single-flight, and the convergence rollback to `lastConvergedReplicas`) assumes exactly one active writer, so availability is active/standby via leader-election, never active/active — two live reconcilers would race on the annotation write and the rollback decision.
162163
- **Bounded inputs.** All tenant-supplied DHA fields are enumerable and schema-validated: `minReplicas`/`maxReplicas` (integers), a fixed `metrics[].type` enum (`ReadConnections`, `ReadCPUUtilization`), numeric targets, and windows. Arbitrary tenant-supplied PromQL is **not** accepted (see Alternatives), so there is no path for a tenant query to read another tenant's series from shared vmselect. No new secrets are stored or transmitted.
163164

164165
## Failure and edge cases
@@ -177,14 +178,14 @@ When several `metrics` are set, the desired count is the **maximum** of the per-
177178

178179
- **Unit:** reconcile decisions and each `TopologyAdapter` (including the `PrimaryCount`/`replicas − 1` math and `QuorumFloor = maxSyncReplicas + 1`) with mocked VictoriaMetrics and a mocked Application client (`go test ./internal/controller/...`).
179180
- **Codegen:** `make generate` produces the CRD and deepcopy without errors.
180-
- **envtest (apiserver-backed):** the ownership path is exercised against a real API server, since server-side-apply managed-fields conflict semantics do not exist with a mocked client. A competing writer claims `replicas` both without force and with `force: true`, and the test asserts the `autoscaling.cozystack.io/managed-by` marker, the surfaced conflict condition, and the absence of scaling thrash. This is the only layer that actually verifies the ownership guarantee — the mocked unit tests above cannot.
181+
- **envtest (apiserver-backed):** the ownership path is exercised against a real API server, since server-side-apply managed-fields conflict semantics do not exist with a mocked client. A competing writer claims `replicas` both without force and with `force: true`, and the test asserts the `autoscaling.cozystack.io/managed-by` marker, the surfaced conflict condition, and the absence of scaling thrash. This is the only layer that actually verifies the ownership guarantee — the mocked unit tests above cannot. The spike also confirms whether the aggregated `apps.cozystack.io` Patch handler tracks `.spec.replicas` managed-fields **at all** (see the Ownership caveat); if it does not, the ownership guarantee falls back to the admission webhook.
181182
- **Manual (dev cluster, CNPG postgres):** create a DHA with `dryRun: true` → decisions appear in `status`/Events, replicas unchanged. Then disable `dryRun` under read load → the `Application`'s `replicas` grows, CNPG adds a standby, reads route to `<release>-ro`, and Flux does not revert; on load decrease and after the window, scale-down removes a standby gracefully and never drops to/below `maxSyncReplicas + 1`.
182183
- **Negative:** vmselect down → freeze; lag above threshold with active writes → no scaling; idle primary with high lag-seconds → no false freeze; DHA targeting a sharded ClickHouse → `ScalingActive=False`; concurrent GitOps write to `replicas` → SSA conflict condition, no thrash (covered by the envtest above).
183184

184185
## Rollout
185186

186187
1. **PoC** — CNPG PostgreSQL on a dev cluster: DHA + `replicas` patch driven by `ReadConnections`; confirm Flux does not revert and reads route to `<release>-ro`.
187-
2. **MVP** — the operator plus the `postgres` adapter, full guardrails (quorum, lag, cooldown, quota), `dryRun`, dashboard surface and events. Shipped as an optional `paas`-bundle package.
188+
2. **MVP** — the operator plus the `postgres` adapter, full guardrails (quorum, lag, cooldown, quota), `dryRun`, dashboard surface and events. Shipped as an optional `paas`-bundle package that declares a hard `PackageSource.dependsOn`-class dependency on the monitoring stack (VictoriaMetrics/vmselect + `WorkloadMonitor`) — the decision loop cannot function without it, the same cold-install ordering the platform already applies to cert-manager-dependent charts. The operator Deployment runs a **single active instance** (`replicas: 1` with controller-runtime **leader-election**), so two replicas never race on the `managed-by` annotation, single-flight, or the `lastConvergedReplicas` rollback.
188189
3. **Adapter expansion** — `mariadb` → `redis` → `mongodb` (replica set).
189190
4. **Observability & policy** — Grafana dashboard of scaling decisions, alerts for "limit reached / freeze".
190191

@@ -193,7 +194,7 @@ When several `metrics` are set, the desired count is the **maximum** of the per-
193194
- Adapter order after MVP: `mariadb` → `redis` → `mongodb`?
194195
- Default driver metric: read connections, read QPS, or replica CPU (to be calibrated on real workloads)?
195196
- Is scale-down enabled by default, or scale-up only (down conservative/manual)?
196-
- Ownership enforcement: is the SSA field-manager + marker sufficient, or do we add a validating admission webhook that hard-rejects conflicting `replicas` writes for DHA-managed applications?
197+
- Ownership enforcement: **does the aggregated `apps.cozystack.io` Patch handler support per-field managed-fields SSA at all** (spike required — see Testing)? If not, the SSA field-manager + marker is insufficient and the validating admission webhook that hard-rejects conflicting `replicas` writes for DHA-managed applications becomes mandatory (it is also the only thing that beats a `spec.force: true` writer).
197198

198199
## Alternatives considered
199200

0 commit comments

Comments
 (0)