You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 55aeda2
Browse filesBrowse the repository at this point in the historyBrowse files
Replace the one-listener-per-database model with one shared passthrough
listener per engine type, with per-release routing on standard TLSRoute
objects keyed by SNI hostname. Listener consumption drops from O(database
instances) to O(engine types), so the 64-listener Gateway cap stops being
a per-database ceiling. Rename the API field to TLSPassthroughListeners,
declare the route backend as a standard Gateway API BackendObjectReference
instead of a placeholder, and rework the diagram, budget, failure,
testing, and open-questions sections to match.
Reject an explicit tls.enabled=false combined with external=true at
admission via a ValidatingAdmissionPolicy rather than silently overriding
it, so a tenant cannot stand up a plaintext external endpoint; an unset
tri-state still auto-enables TLS together with external.
Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
A Gateway listener is keyed by the tuple (port, protocol, hostname/SNI). SNI-based routing therefore works on any TCP port, not only 443 — a `TLSRoute` selects its backend by hostname regardless of the listener's port. "Share one IP via SNI" and "use the native port" are orthogonal, so the design takes both: one passthrough listener per engine on its native port — `tls-<release>` on 5432 for postgres, 6379 for redis, 27017 for sharded mongo, 3306 for mariadb — each `mode: Passthrough`, hostname `<release>.<apex>`, SNI-routed via a per-release `TLSRoute`. All of a tenant's database listeners live on the one tenant Gateway and therefore share its single IP.
106
+
A Gateway listener is keyed by the tuple (port, protocol, hostname/SNI), and **multiple `TLSRoute` objects can attach to one listener**, each selecting its backend by its own `spec.hostnames`. SNI-based routing therefore works on any TCP port, not only 443, and — crucially — it does not need a listener per database. The design uses **one passthrough listener per engine type**, on that engine's native port, with a wildcard hostname: `tls-postgres` on 5432, `tls-redis` on 6379, `tls-mongos` on 27017, `tls-mariadb` on 3306 — each `mode: Passthrough`, hostname `*.<apex>` (or unrestricted), `AllowedRoutes` limited to `TLSRoute`. Every database release of that engine then attaches a per-release `TLSRoute` carrying `spec.hostnames: ["<release>.<apex>"]`, and the Gateway SNI-routes each connection to the right backend. All of a tenant's database listeners live on the one tenant Gateway and therefore share its single IP.
107
+
108
+
This keeps listener consumption at **O(engine types)** — at most four or five — rather than O(database instances). A tenant running thirty Postgres releases spends one `tls-postgres` slot, not thirty, so the 64-listener budget (§3) stops being a per-database ceiling and the consolidation goal scales with instance count, not against it.
96
109
97
110
Native ports are chosen over forcing everything onto 443 because the latter buys nothing: it does not improve IP consolidation (SNI already does that on any port) and it breaks client ergonomics and tooling defaults (`psql -h host` assumes 5432) while still demanding direct-TLS. The all-on-443 variant is retained as a documented opt-in for operators who want a single-port firewall surface (see Alternatives).
98
111
@@ -119,7 +132,7 @@ The headline payoff is IPv4 scarcity relief. Today ten external databases cost t
119
132
120
133
The ceiling, stated honestly:
121
134
122
-
-`Gateway.spec.listeners` is capped at 64. The tenant Gateway already spends slots on the http/https/https-apex listeners, the per-child-apex wildcard listeners, and the three default passthrough services. Database listeners draw from the same budget; an operator near the cap should split a high-fanout subtree onto its own Gateway via `tenant.spec.gateway=true`, as the controller already advises.
135
+
-`Gateway.spec.listeners` is capped at 64. Because database listeners are **one per engine type** (§1), not one per release, their contribution to that budget is bounded by the four or five supported engines no matter how many instances a tenant runs; the per-release fan-out lands on `TLSRoute` objects, which are not capped. The remaining slots go to the http/https/https-apex listeners, the per-child-apex wildcard listeners, and the three default passthrough services. A tenant therefore approaches the cap through child-apex fan-out, not database fan-out; the controller's existing advice to split a high-fanout subtree onto its own Gateway via `tenant.spec.gateway=true` still applies there.
123
136
- Multi-Gateway IP sharing is not possible today (the one-IP-per-tenant blocker above). The win is one IP per tenant Gateway. Lifting it to share an IP across Gateways depends on Cilium implementing ListenerSet (experimental in Gateway API v1.5.1, not implemented by Cilium 1.19.3).
124
137
125
138
### 4. Per-engine treatment
@@ -139,26 +152,30 @@ Kafka is deferred because its wire protocol redirects: the client connects to a
139
152
140
153
The trigger and listener synthesis stay in the controller; database charts do not render Gateway plumbing. Database charts already receive the `_cluster` values channel and read parts of it (for example `packages/apps/postgres/templates/db.yaml` reads `_cluster.scheduling`, and mongodb reads `_cluster["cluster-domain"]`), but they do not read the gateway-discovery keys that `cozystack-api` consumes (the gateway-enabled flag and the gateway name) and have no logic to locate the tenant Gateway. Teaching every database chart that discovery dance would duplicate it across five charts and couple application charts to networking topology. The controller already owns listener synthesis; keep it there.
141
154
142
-
The existing `TLSPassthroughServices []string` field (`api/gateway/v1alpha1/tenantgateway_types.go`) is too weak for databases — a bare service name hardcodes the layer-7 convention of port 443 and hostname `<name>.<apex>`. Add one structured field alongside it, leaving the existing field untouched for backward compatibility:
155
+
Two objects implement the model, at different cardinalities: one shared listener per engine type, and one `TLSRoute` per database release.
156
+
157
+
The **listener** is the new API surface. The existing `TLSPassthroughServices []string` field (`api/gateway/v1alpha1/tenantgateway_types.go`) is too weak — a bare service name hardcodes the layer-7 convention of port 443 and hostname `<name>.<apex>`. Add one structured field alongside it, leaving the existing field untouched for backward compatibility:
143
158
144
159
```go
145
-
//TLSPassthroughBackend declares a per-engine passthrough listener on a
The controller change is to extend the existing passthrough-listener loop to also iterate `TLSPassthroughBackends`, rendering each listener with the supplied `Port` instead of the hardcoded 443. The `TLSRoute` attaches by `sectionName: tls-<name>`, identical to the existing `api-tlsroute.yaml` pattern. The field is populated by the Tenant / HelmRelease orchestration layer that already knows both the tenant Gateway and the database release — not the database chart and not the human — so a database's surface stays a single `external`-adjacent toggle.
176
+
The **route** is a standard Gateway API `TLSRoute` (no new type), rendered once per exposed database release: `spec.parentRefs` attaches to the shared `tls-<engine>` listener by `sectionName`, `spec.hostnames: ["<release>.<apex>"]` carries the SNI match, and `spec.rules[].backendRefs` is a standard `gwapiv1alpha2.BackendRef` (whose embedded `BackendObjectReference` points at the database Service on its native port, cross-namespace via `ReferenceGrant`).
177
+
178
+
The controller change is to render, per `TLSPassthroughListeners` entry, one Passthrough listener on the supplied `Port` instead of the hardcoded 443, and to attach each per-release `TLSRoute` by `sectionName: tls-<name>` — the same attachment pattern as the existing `api-tlsroute.yaml`, except that many routes share one listener and the Gateway disambiguates them by SNI hostname. Both are populated by the Tenant / HelmRelease orchestration layer that already knows the tenant Gateway and the database release — not the database chart and not the human — so a database's surface stays a single `external`-adjacent toggle. The engine-type listener is created on first exposure of that engine and removed once its last release is gone; per-release add/remove only touches the route, never the shared listener.
162
179
163
180
### 6. Trust-anchor and SAN flow
164
181
@@ -170,24 +187,25 @@ A database gains an `external`-adjacent toggle to select passthrough/SNI mode. P
170
187
171
188
## Upgrade and rollback compatibility
172
189
173
-
The default stays today's per-database LoadBalancer, so no existing external database changes its IP on upgrade. Passthrough/SNI mode is opt-in. Migrating an existing external database to SNI mode is a breaking, opt-in change — the IP changes and the client must be reconfigured (new host, direct-TLS for Postgres); it is not automatic. The existing `TLSPassthroughServices` field continues to work unchanged, and reverting the feature removes the new listener and field without touching the database's own PKI.
190
+
The default stays today's per-database LoadBalancer, so no existing external database changes its IP on upgrade. Passthrough/SNI mode is opt-in. Migrating an existing external database to SNI mode is a breaking, opt-in change — the IP changes and the client must be reconfigured (new host, direct-TLS for Postgres); it is not automatic. The existing `TLSPassthroughServices` field continues to work unchanged, and reverting the feature removes the per-release routes and the engine-type listeners without touching the database's own PKI.
174
191
175
192
## Security
176
193
177
-
The edge never holds the database's private key — the central strength of passthrough over termination. SNI is sent in cleartext on TLS 1.3 except under ECH, which database clients do not use, so the external hostname `<release>.<apex>` is observable on the wire; this is no worse than DNS or SNI exposure for any TLS service, and the payload stays encrypted end-to-end. A missing or mis-emitted SNI is a hard connection failure (a passthrough listener has no certificate to fall back to) — fail-closed, which is the secure default. Cross-namespace `backendRef` requires a `ReferenceGrant`, consistent with the existing attached-namespaces model. Because trust rides the `ca.crt`-only object, clients never receive private key material.
194
+
The edge never holds the database's private key — the central strength of passthrough over termination. SNI is sent in cleartext on TLS 1.3 except under ECH, which database clients do not use, so the external hostname `<release>.<apex>` is observable on the wire; this is no worse than DNS or SNI exposure for any TLS service, and the payload stays encrypted end-to-end. A missing or mis-emitted SNI is a hard connection failure (a passthrough listener has no certificate to fall back to) — fail-closed, which is the secure default. Exposing a database externally with TLS explicitly off is not silently corrected: when `tls.enabled` is left unset the tri-state turns TLS on together with `external`, but an explicit `tls.enabled: false` combined with `external: true` is **rejected by a ValidatingAdmissionPolicy** — the same admission mechanism that already enforces tenant hostname/IP isolation — so a tenant cannot stand up a plaintext external endpoint by accident or by overriding the default, and the conflict surfaces as a clear admission error rather than a silent override the operator never sees. Cross-namespace `backendRef` requires a `ReferenceGrant`, consistent with the existing attached-namespaces model. Because trust rides the `ca.crt`-only object, clients never receive private key material.
178
195
179
196
## Failure and edge cases
180
197
181
198
- A pre-PG17 or non-direct-TLS Postgres client sends no SNI → no route → connection reset/timeout (document the symptom).
182
-
- The 64-listener budget is exceeded → the listener is rejected; mitigation is to split the high-fanout subtree onto its own Gateway.
183
-
-`tls.enabled=false`while exposing externally → certificate SAN mismatch; the tri-state should auto-enable TLS with `external`.
199
+
- The 64-listener budget is exceeded → the listener is rejected; because listeners are one per engine type this is reached only through child-apex fan-out, and the mitigation is to split that subtree onto its own Gateway.
200
+
-An explicit `tls.enabled: false`together with `external: true` → rejected at admission by a ValidatingAdmissionPolicy, not silently overridden; an unset `tls`tri-state auto-enables TLS with `external`.
184
201
- An operator expects multi-Gateway IP sharing → each Gateway still gets its own IP (expected under the Cilium constraint).
185
-
- A database is deleted → its `TLSPassthroughBackends` entry and listener must be cleaned up so they do not orphan against the 64-listener cap.
202
+
- A database is deleted → its per-release `TLSRoute` is removed; the shared engine-type listener is removed only when its last release is gone, so a single deletion never orphans a listener.
186
203
187
204
## Testing
188
205
189
206
- Helm-template assertions that the certificate SAN includes `<release>.<apex>` per engine, mirroring the existing TLS test fixtures.
190
-
- A controller unit test that a `TLSPassthroughBackends` entry renders a listener on the native port with `mode: Passthrough` and `AllowedRoutes` restricted to `TLSRoute`.
207
+
- A controller unit test that a `TLSPassthroughListeners` entry renders a shared listener on the native port with `mode: Passthrough` and `AllowedRoutes` restricted to `TLSRoute`, and that two per-release `TLSRoute` objects on that one listener SNI-route to their respective backends.
208
+
- An admission test that `tls.enabled: false` with `external: true` is rejected by the ValidatingAdmissionPolicy, while an unset `tls` is admitted and auto-enables.
191
209
- An end-to-end test per fitting engine: connect from outside the cluster with SNI and `ca.crt`, and assert that the serial of the presented certificate equals the operator-issued internal certificate — proving reuse, not re-issuance.
192
210
- A negative test: a client without SNI fails closed.
193
211
@@ -205,9 +223,9 @@ Kafka and non-sharded MongoDB are explicitly out of this rollout. Each phase shi
205
223
-**MariaDB / MySQL connector conformance** — do the dominant connectors (libmysqlclient, MariaDB Connector/C, JDBC, Go drivers) emit SNI and do TLS-first such that passthrough routes correctly, or does any do a server-greeting-then-STARTTLS dance like pre-direct-TLS Postgres? Default-on versus opt-in for MariaDB hinges on this.
206
224
-**Direct-TLS client floor for Postgres** — what fraction of the tenant base predates `sslnegotiation=direct`? Is a per-release attestation gate enough, given there is no graceful downgrade on a passthrough listener?
207
225
-**Cilium passthrough on a non-443 port** — the existing passthrough listeners all run on 443; whether Cilium 1.19.3's Gateway implementation honors a `mode: Passthrough` listener on a native database port (5432/6379/27017/3306) and SNI-routes it is unverified and is the single biggest implementation risk. It is the Phase 1 validation gate in Rollout.
208
-
-**TLSRoute ownership** — does the controller synthesize the `TLSRoute` alongside the listener (single source of truth, but the controller must know the backend Service), or does the database chart render it (the chart owns the backend selector, but must learn a gateway name/namespace it does not have today)? This decides whether database charts gain gateway-discovery logic.
209
-
-**Who writes `TLSPassthroughBackends`** — the orchestration layer auto-appending on `external: true`plus passthrough mode, an operator-managed explicit list, or both; and which reconciliation owns add/remove and orphaned-listener cleanup on database deletion.
210
-
-**64-listener budget accounting** — with the parent listeners, per-child-apex wildcards, default passthrough services, and N database listeners, what is the realistic per-tenant ceiling, and should the controller surface a status condition as the budget nears 64?
226
+
-**TLSRoute ownership** — does the controller synthesize the per-release `TLSRoute` alongside the shared listener (single source of truth, but the controller must know the backend Service), or does the database chart render it (the chart owns the backend selector, but must learn a gateway name/namespace it does not have today)? This decides whether database charts gain gateway-discovery logic.
227
+
-**Who writes the listener/route set** — the orchestration layer auto-creating the engine-type listener on first exposure and a per-release route on each `external: true`+ passthrough, an operator-managed explicit list, or both; and which reconciliation owns route add/remove plus reference-counting the shared listener down to zero on the last database deletion.
228
+
-**64-listener budget accounting** — with the parent listeners, per-child-apex wildcards, and default passthrough services now joined by one listener per database engine type (not per instance), what is the realistic per-tenant ceiling, and should the controller still surface a status condition as the budget nears 64?
211
229
-**ListenerSet timeline** — multi-Gateway IP sharing is blocked on Cilium implementing ListenerSet. Do we design the field and status to be ListenerSet-ready now, or revisit when Cilium ships it? This determines whether one-IP-per-tenant is a permanent or temporary ceiling.
0 commit comments