Skip to content

Commit 52f09d1

Browse files
lexfreilexfrei
committed
fix(gateway)!: render passthrough services only where platform routes attach
The chart wrote the tlsPassthroughServices default onto every non-edge tenant's TenantGateway, so every tenant Gateway carried tls-api, tls-vm-exportproxy and tls-cdi-uploadproxy. The TLSRoutes that serve those names are platform routes: the packages owning the services render one per name in publishing.exposedServices while gateway.enabled is on, parented to the publishing tenant's Gateway alone. Everywhere else no shipped route attached to the listeners, and a passthrough listener holds its name whether or not anything is routed through it. The chart now writes the field only on the publishing tenant, with the same gateway.enabled switch and the same exposedServices filter those routes use, so a listener exists exactly where a platform route can attach. A TLSRoute attached by hand to one of those listeners on another tenant loses its parent. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent bdca165 commit 52f09d1

5 files changed

Lines changed: 120 additions & 26 deletions

File tree

‎packages/extra/gateway/README.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ The Secret must exist in the `TenantGateway`'s own namespace, be of type `kubern
6666
TLS termination is a property of the GatewayClass, so it is declared once per class rather than per tenant: list a class in the platform's `gateway.edgeTerminatedClasses` when its provider terminates TLS upstream of the Gateway — a Cloudflare Tunnel class ends TLS at the Cloudflare edge and reaches the Gateway over the tunnel. Every tenant whose Gateway lands on such a class gets `certMode: edge` on its `TenantGateway`, whether it arrived there through the platform default (`gateway.className`) or by choosing the class itself (`tenant.spec.gatewayClass`, bounded by `gateway.tenantSelectableClasses`). The mode wins over `wildcardSecretName` and over `solver`; the solver, provider, issuer and wildcard-secret inputs are not read. The controller then:
6767

6868
- Mints no `Issuer`, no `Certificate` and no http-to-https redirect route (the edge redirects itself). Switching into this mode garbage-collects the ACME objects the controller previously owned — the `Issuer` and the `Certificate` objects, not the TLS Secrets they issued: cert-manager runs with `enableCertificateOwnerRef: false`, so each Secret stays in the tenant namespace, unreferenced, until an operator removes it. That is how every mode transition already behaves, not something edge introduces.
69-
- Renders the apex, its wildcard and every inheriting child apex as plain HTTP listeners (`edge`, `edge-apex`, `edge-child-*`) with the same label-based `allowedRoutes` as the HTTPS listeners of the other modes, so app HTTPRoutes attach by hostname unchanged. `TLSPassthroughServices` are not rendered: an HTTP-only edge cannot carry a TLS-passthrough listener, so the tenant Kubernetes API, VM export and CDI upload endpoints are not published through such a Gateway. `TLSPassthroughListeners` is refused rather than dropped, and the difference is which side sets the field: `tlsPassthroughServices` carries a chart default on every tenant, so dropping it is the only way an edge tenant renders at all, while a native-port listener is set on the CR by hand and a tenant that asked for one is better told it cannot have it here than left reading `Ready=True` over a port nothing answers. Where the packages that own them create those `TLSRoute` objects at all — each renders only when `gateway.enabled` is on and its name is still in `publishing.exposedServices` — the routes then match no listener on the Gateway, so those endpoints are unpublished rather than undeclared. The controller says so on the routes themselves rather than leaving them to be read as healthy: every attached `TLSRoute` that declares hostnames gets `Accepted=False` with reason `NoMatchingParent` under this controller's name, which is Gateway API's reason for a `parentRef` whose port or section matches no listener on the Gateway. A `TLSRoute` that declares none has this controller's entry removed instead, because the hostname it would be judged on is lent by a listener this mode does not render, and every attached `HTTPRoute` has the entry removed too, because whatever it carried was written by the HTTP-01 pass about listeners this mode does not render; `dns01` and `existingSecret` remove both the same way. The class controller writes its own entry under its own name and this does not touch it. Moving back off the mode clears the condition again, either by judging the route afresh under `http01` or, under `dns01` and `existingSecret` where the controller collects no hostname claims, by dropping the entry instead of holding an opinion it cannot support.
69+
- Renders the apex, its wildcard and every inheriting child apex as plain HTTP listeners (`edge`, `edge-apex`, `edge-child-*`) with the same label-based `allowedRoutes` as the HTTPS listeners of the other modes, so app HTTPRoutes attach by hostname unchanged. `TLSPassthroughServices` are not rendered: an HTTP-only edge cannot carry a TLS-passthrough listener, so the tenant Kubernetes API, VM export and CDI upload endpoints are not published through such a Gateway. `TLSPassthroughListeners` is refused rather than dropped, and the difference is which side sets the field: `tlsPassthroughServices` carries a chart default on the publishing tenant, so dropping it is the only way an edge tenant renders at all, while a native-port listener is set on the CR by hand and a tenant that asked for one is better told it cannot have it here than left reading `Ready=True` over a port nothing answers. Where the packages that own them create those `TLSRoute` objects at all — each renders only when `gateway.enabled` is on and its name is still in `publishing.exposedServices` — the routes then match no listener on the Gateway, so those endpoints are unpublished rather than undeclared. The controller says so on the routes themselves rather than leaving them to be read as healthy: every attached `TLSRoute` that declares hostnames gets `Accepted=False` with reason `NoMatchingParent` under this controller's name, which is Gateway API's reason for a `parentRef` whose port or section matches no listener on the Gateway. A `TLSRoute` that declares none has this controller's entry removed instead, because the hostname it would be judged on is lent by a listener this mode does not render, and every attached `HTTPRoute` has the entry removed too, because whatever it carried was written by the HTTP-01 pass about listeners this mode does not render; `dns01` and `existingSecret` remove both the same way. The class controller writes its own entry under its own name and this does not touch it. Moving back off the mode clears the condition again, either by judging the route afresh under `http01` or, under `dns01` and `existingSecret` where the controller collects no hostname claims, by dropping the entry instead of holding an opinion it cannot support.
7070

7171
Listing a class here is a claim about that class and nothing validates it: name the bundled `cilium` and its Gateways serve every app hostname over plain HTTP on a public address, with no redirect and no certificate — exactly the cleartext-credentials case the narrow port-80 selector exists to prevent in the other modes. Only list a class whose provider terminates TLS in front of the Gateway.
7272

@@ -114,10 +114,10 @@ Schema constraints tightened alongside this field reach objects that already exi
114114

115115
### Common parameters
116116

117-
| Name | Description | Type | Value |
118-
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---------------------------------------- |
119-
| `gatewayClassName` | GatewayClass for this tenant's Gateway, set from `tenant.spec.gatewayClass`. Empty means the platform default (`_cluster.gateway-class-name`, itself defaulting to the bundled `cilium` class). Whether the class terminates TLS itself or at its provider's edge comes from the platform's `gateway.edgeTerminatedClasses`, so the two cannot get out of step. | `string` | `""` |
120-
| `tlsPassthroughServices` | Names (from publishing.exposedServices) whose traffic is TLS-passthrough rather than TLS-terminate. For each such service a dedicated HTTPS listener with tls.mode=Passthrough is rendered on the Gateway, and the service is expected to attach a TLSRoute instead of an HTTPRoute. Ignored when this tenant's Gateway is on a class the platform lists in `gateway.edgeTerminatedClasses`: an HTTP-only edge cannot carry a TLS listener, so those services are not published through the tenant Gateway at all. | `[]string` | `[api, vm-exportproxy, cdi-uploadproxy]` |
117+
| Name | Description | Type | Value |
118+
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ---------------------------------------- |
119+
| `gatewayClassName` | GatewayClass for this tenant's Gateway, set from `tenant.spec.gatewayClass`. Empty means the platform default (`_cluster.gateway-class-name`, itself defaulting to the bundled `cilium` class). Whether the class terminates TLS itself or at its provider's edge comes from the platform's `gateway.edgeTerminatedClasses`, so the two cannot get out of step. | `string` | `""` |
120+
| `tlsPassthroughServices` | Names (from publishing.exposedServices) whose traffic is TLS-passthrough rather than TLS-terminate. For each such service a dedicated HTTPS listener with tls.mode=Passthrough is rendered on the Gateway, and the service is expected to attach a TLSRoute instead of an HTTPRoute. Rendered only on the publishing tenant, the namespace `publishing.ingressName` names, while `gateway.enabled` is on, and only for names `publishing.exposedServices` lists: the platform TLSRoutes for these services attach to that tenant's Gateway alone, and a passthrough listener anywhere else would reserve its name with nothing behind it. Ignored when this tenant's Gateway is on a class the platform lists in `gateway.edgeTerminatedClasses`: an HTTP-only edge cannot carry a TLS listener, so those services are not published through the tenant Gateway at all. | `[]string` | `[api, vm-exportproxy, cdi-uploadproxy]` |
121121

122122

123123
## Security model

‎packages/extra/gateway/templates/tenantgateway.yaml‎

Lines changed: 23 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,13 @@
4848
class. The platform default is always allowed: every tenant that chooses
4949
nothing already lands on it.
5050
*/}}
51+
{{- /* expose-ingress is overloaded: it is both an ingressClassName and
52+
the publishing namespace, and the two coincide only in the default
53+
deployment (see the NOTE in the wildcardsecret controller's
54+
readConfig). A deployment that splits them makes the checks below
55+
compare against the wrong namespace, so they stop protecting the
56+
tenant they are meant to. */}}
57+
{{- $publishingNS := (index .Values._cluster "expose-ingress") | default "tenant-root" }}
5158
{{- if .Values.gatewayClassName }}
5259
{{- $allowed := list $defaultClass }}
5360
{{- range splitList "," ((index .Values._cluster "gateway-tenant-classes") | default "") }}
@@ -70,13 +77,6 @@
7077
nothing in any status saying why. Refuse it here, where the operator
7178
is looking, rather than in the certificate that silently never renews.
7279
*/}}
73-
{{- /* expose-ingress is overloaded: it is both an ingressClassName and
74-
the publishing namespace, and the two coincide only in the default
75-
deployment (see the NOTE in the wildcardsecret controller's
76-
readConfig). A deployment that splits them makes this guard compare
77-
against the wrong namespace, so it stops protecting the tenant it
78-
is meant to. */}}
79-
{{- $publishingNS := (index .Values._cluster "expose-ingress") | default "tenant-root" }}
8080
{{- if eq .Release.Namespace $publishingNS }}
8181
{{- $defaultAtEdge := has $defaultClass $edgeClasses }}
8282
{{- $selectedAtEdge := has $className $edgeClasses }}
@@ -190,10 +190,22 @@ spec:
190190
- {{ . | quote }}
191191
{{- end }}
192192
{{- end }}
193-
{{- /* Not written at all in edge mode: the controller renders no TLS
194-
listener for these, so an object advertising them would send an
195-
operator looking for endpoints that are not published. */}}
196-
{{- with (ternary (list) .Values.tlsPassthroughServices (eq $tlsTermination "edge")) }}
193+
{{- /* Only where a platform TLSRoute can attach: the packages owning
194+
these services render one per name in expose-services while
195+
gateway-enabled is on, always parented to the publishing tenant's
196+
Gateway. Anywhere else the listener would be routeless and would
197+
still reserve its name, taking it from the tenant's own HTTPRoutes.
198+
None in edge mode: the controller renders no TLS listener there. */}}
199+
{{- $exposed := splitList "," ((index .Values._cluster "expose-services") | default "") }}
200+
{{- $passthrough := list }}
201+
{{- if and (ne $tlsTermination "edge") (eq .Release.Namespace $publishingNS) (eq (toString ((index .Values._cluster "gateway-enabled") | default "false")) "true") }}
202+
{{- range .Values.tlsPassthroughServices }}
203+
{{- if has . $exposed }}
204+
{{- $passthrough = append $passthrough . }}
205+
{{- end }}
206+
{{- end }}
207+
{{- end }}
208+
{{- with $passthrough }}
197209
tlsPassthroughServices:
198210
{{- range . }}
199211
- {{ . | quote }}

‎packages/extra/gateway/tests/tenantgateway_test.yaml‎

Lines changed: 90 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -259,11 +259,13 @@ tests:
259259
- cozy-dashboard
260260
- cozy-keycloak
261261

262-
- it: tlsPassthroughServices defaults to api,vm-exportproxy,cdi-uploadproxy
262+
- it: the publishing tenant renders the default tlsPassthroughServices the platform exposes
263263
set:
264264
_cluster:
265265
solver: http01
266266
expose-ingress: tenant-root
267+
expose-services: api,dashboard,vm-exportproxy,cdi-uploadproxy
268+
gateway-enabled: "true"
267269
issuer-name: letsencrypt-prod
268270
_namespace:
269271
host: example.org
@@ -275,6 +277,86 @@ tests:
275277
- vm-exportproxy
276278
- cdi-uploadproxy
277279

280+
- it: the publishing tenant renders only the passthrough services the platform exposes
281+
set:
282+
_cluster:
283+
solver: http01
284+
expose-ingress: tenant-root
285+
expose-services: dashboard,api
286+
gateway-enabled: "true"
287+
issuer-name: letsencrypt-prod
288+
_namespace:
289+
host: example.org
290+
asserts:
291+
- equal:
292+
path: spec.tlsPassthroughServices
293+
value:
294+
- api
295+
296+
- it: the publishing tenant is the one expose-ingress names
297+
release:
298+
namespace: tenant-pub
299+
set:
300+
_cluster:
301+
solver: http01
302+
expose-ingress: tenant-pub
303+
expose-services: api,dashboard,vm-exportproxy,cdi-uploadproxy
304+
gateway-enabled: "true"
305+
issuer-name: letsencrypt-prod
306+
_namespace:
307+
host: pub.example.org
308+
asserts:
309+
- equal:
310+
path: spec.tlsPassthroughServices
311+
value:
312+
- api
313+
- vm-exportproxy
314+
- cdi-uploadproxy
315+
316+
- it: a tenant that does not publish the platform services renders no tlsPassthroughServices
317+
release:
318+
namespace: tenant-alice
319+
set:
320+
_cluster:
321+
solver: http01
322+
expose-ingress: tenant-root
323+
expose-services: api,dashboard,vm-exportproxy,cdi-uploadproxy
324+
gateway-enabled: "true"
325+
issuer-name: letsencrypt-prod
326+
_namespace:
327+
host: alice.example.org
328+
asserts:
329+
- notExists:
330+
path: spec.tlsPassthroughServices
331+
332+
- it: the publishing tenant renders no tlsPassthroughServices while the platform gateway is off
333+
set:
334+
_cluster:
335+
solver: http01
336+
expose-ingress: tenant-root
337+
expose-services: api,dashboard,vm-exportproxy,cdi-uploadproxy
338+
gateway-enabled: "false"
339+
issuer-name: letsencrypt-prod
340+
_namespace:
341+
host: example.org
342+
asserts:
343+
- notExists:
344+
path: spec.tlsPassthroughServices
345+
346+
- it: the publishing tenant renders no tlsPassthroughServices when the platform exposes none of them
347+
set:
348+
_cluster:
349+
solver: http01
350+
expose-ingress: tenant-root
351+
expose-services: dashboard
352+
gateway-enabled: "true"
353+
issuer-name: letsencrypt-prod
354+
_namespace:
355+
host: example.org
356+
asserts:
357+
- notExists:
358+
path: spec.tlsPassthroughServices
359+
278360
- it: wildcard-secret-name switches to certMode=existingSecret with wildcardSecretRef
279361
set:
280362
_cluster:
@@ -337,6 +419,8 @@ tests:
337419
_cluster:
338420
solver: http01
339421
expose-ingress: tenant-root
422+
expose-services: api,dashboard,vm-exportproxy,cdi-uploadproxy
423+
gateway-enabled: "true"
340424
issuer-name: letsencrypt-prod
341425
wildcard-secret-name: wildcard-tls
342426
_namespace:
@@ -679,20 +763,18 @@ tests:
679763
path: spec.certMode
680764
value: edge
681765

682-
- it: an edge tenant advertises no tlsPassthroughServices in its CR
683-
release:
684-
namespace: tenant-alice
766+
- it: an edge publishing tenant advertises no tlsPassthroughServices in its CR
685767
set:
686768
_cluster:
687769
solver: http01
688770
expose-ingress: tenant-root
771+
expose-services: api,dashboard,vm-exportproxy,cdi-uploadproxy
772+
gateway-enabled: "true"
689773
issuer-name: letsencrypt-prod
690-
gateway-class-name: cilium
774+
gateway-class-name: cloudflare-tunnel
691775
gateway-edge-classes: cloudflare-tunnel
692-
gateway-tenant-classes: cloudflare-tunnel
693776
_namespace:
694-
host: alice.example.org
695-
gatewayClassName: cloudflare-tunnel
777+
host: example.org
696778
asserts:
697779
- equal:
698780
path: spec.certMode

0 commit comments

Comments
 (0)