Skip to content

Commit 891e57a

Browse files
committed
docs(gateway): document GatewayClass selection and edge TLS termination
The per-tenant Gateway chart README covers the four certificate modes as chart reference, but an operator setting up an edge-terminated class has no page answering their questions: which platform values bound the choice, what a tenant writes to make it, what a move between classes does to certificates and to the replicated wildcard Secret, and how the mechanism fails when a configured class name matches nothing installed. Add docs/gateway-classes.md covering that surface, and record in the chart README the one constraint it did not state: the publishing tenant cannot select a class whose TLS termination differs from the platform default, because the cluster-wide ACME issuers derive their solver pin from that default alone. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent bb08ae0 commit 891e57a

2 files changed

Lines changed: 121 additions & 0 deletions

File tree

‎docs/gateway-classes.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# GatewayClass selection and TLS termination for tenant Gateways
2+
3+
A tenant that owns a Gateway (`tenant.spec.gateway: true`) gets one `Gateway` object bound to one `GatewayClass`. Which class that is decides who programs the dataplane and, with it, whether TLS is terminated by the Gateway itself or by something in front of it. This page covers the operator-facing surface: the platform values that bound the choice, the tenant field that makes it, what each resulting certificate mode implies, what a move between classes does to certificates and Secrets, and how the mechanism fails when the configured names and the installed classes disagree.
4+
5+
The chart-level reference for the per-tenant Gateway — inheritance, listener shapes, the security model, ACME rate limits — is [`packages/extra/gateway/README.md`](../packages/extra/gateway/README.md).
6+
7+
## Platform values
8+
9+
Three keys under `gateway` in the platform values bound which class a tenant Gateway can land on:
10+
11+
```yaml
12+
gateway:
13+
enabled: true
14+
className: cilium
15+
tenantSelectableClasses: []
16+
edgeTerminatedClasses: []
17+
```
18+
19+
- `className` — the class every tenant Gateway uses unless the tenant names another one. Defaults to the Cilium class the platform ships.
20+
- `tenantSelectableClasses` — the additional class names a tenant may name for itself. Empty (the default) means tenants cannot choose at all.
21+
- `edgeTerminatedClasses` — the class names whose provider terminates TLS upstream of the Gateway. Membership in this list is the only thing that puts a tenant into edge mode.
22+
23+
All three reach the per-tenant gateway chart through the platform values channel, as `_cluster.gateway-class-name`, `_cluster.gateway-tenant-classes` and `_cluster.gateway-edge-classes`. The two lists are joined with commas on the way, so a name containing a comma is rejected at platform render time rather than silently split into two names.
24+
25+
`gateway.enabled` is a separate switch and not a prerequisite for any of this: it moves the platform's own published services from Ingress onto Gateway API and points the ACME `ClusterIssuer`s' HTTP-01 solver at the publishing tenant's Gateway. A tenant Gateway is created from `tenant.spec.gateway` alone.
26+
27+
## How a tenant picks a class
28+
29+
```yaml
30+
apiVersion: apps.cozystack.io/v1alpha1
31+
kind: Tenant
32+
metadata:
33+
name: acme
34+
namespace: tenant-root
35+
spec:
36+
gateway: true
37+
gatewayClass: edge-tunnel
38+
```
39+
40+
`edge-tunnel` throughout this page stands for the name of a GatewayClass whose controller terminates TLS outside the cluster and forwards cleartext to the Gateway. Substitute whatever your own edge-terminating controller registers; the platform ships no such class.
41+
42+
The field is read only for a tenant that owns a Gateway. A tenant that inherits — `gateway` left unset or set to `false` — publishes through the nearest ancestor that owns one and runs on that ancestor's class, so its own `gatewayClass` has no effect. Leaving `gatewayClass` empty selects `gateway.className`.
43+
44+
The set a tenant may name is `gateway.tenantSelectableClasses` plus the current `gateway.className`, so a tenant may always name the default explicitly. Anything else fails that tenant's own gateway release at render time, with a message naming the class and the allowed set, and reaches no other tenant. The allowlist exists because `Tenant` is a tenant-writable object while a `GatewayClass` is cluster-scoped: without it a tenant could attach its Gateway to any class the cluster registers, including one an admin keeps for internal-only traffic.
45+
46+
## What the class decides: the certificate mode
47+
48+
`TenantGateway.spec.certMode` is derived, never written by the tenant. The gateway chart resolves it in this order, first match winning:
49+
50+
| Condition | `certMode` | What the controller issues |
51+
| --- | --- | --- |
52+
| the resolved class is in `gateway.edgeTerminatedClasses` | `edge` | nothing |
53+
| `publishing.certificates.wildcardSecretName` is set | `existingSecret` | nothing; the operator's Secret is referenced |
54+
| `publishing.certificates.solver: dns01` | `dns01` | one wildcard `Certificate` per tenant |
55+
| otherwise | `http01` | one `Certificate` per published hostname |
56+
57+
Edge therefore wins over an operator-supplied wildcard Secret, which wins over the solver. In edge mode the solver, DNS-01 provider, issuer name and wildcard-secret inputs are not read at all, so an unsupported value in any of them does not fail the tenant's release the way it would in the other modes.
58+
59+
## What edge mode changes on the Gateway
60+
61+
- The Gateway serves port 80 only. Its listeners are `edge` for `*.<apex>`, `edge-apex` for `<apex>`, and one `edge-child-*` per inheriting child tenant carrying that child's `*.<child-apex>`. There is no port-443 listener, and no hostname-less port-80 listener either.
62+
- No per-tenant ACME `Issuer` and no `Certificate` are created.
63+
- No http-to-https redirect `HTTPRoute` is created. The edge is expected to perform that redirect.
64+
- `tlsPassthroughServices` is not written onto the `TenantGateway` and no passthrough listener is rendered, so the tenant Kubernetes API, VM export and CDI upload endpoints are not published through such a Gateway. The packages that own those endpoints still create their `TLSRoute` objects; the routes match no listener.
65+
- Application `HTTPRoute`s attach by hostname exactly as in the other modes — the edge listeners carry the same namespace selector the HTTPS listeners carry elsewhere.
66+
67+
Listing a class in `gateway.edgeTerminatedClasses` is an assertion about that class, and nothing verifies it. Put the bundled `cilium` class in the list and its Gateways serve every application hostname over plain HTTP on a public address, with no redirect and no certificate. Only list a class whose provider really does terminate TLS in front of the Gateway.
68+
69+
## Moving a tenant between classes
70+
71+
Onto an edge-terminated class, on the next reconcile:
72+
73+
| Object | What happens |
74+
| --- | --- |
75+
| the per-tenant ACME `Issuer` | deleted |
76+
| per-listener and wildcard `Certificate` objects | deleted |
77+
| the TLS Secrets those `Certificate`s issued | left in the namespace, unreferenced |
78+
| the http-to-https redirect `HTTPRoute` | deleted |
79+
| the replica of `publishing.certificates.wildcardSecretName` | deleted from that tenant's namespace |
80+
| route status conditions the controller wrote earlier | left as they are |
81+
82+
Every deletion is guarded by ownership: an object of the same name that the controller did not create is left in place.
83+
84+
The TLS Secrets survive because cert-manager ships here with `enableCertificateOwnerRef: false`, so a `Certificate` does not own the Secret it issued into. That is how every mode transition behaves and is not something edge introduces. Removing the leftovers is an operator action.
85+
86+
The wildcard replica is withdrawn because the namespace stops counting as a TLS termination point. Three things keep it anyway: a namespace that also owns an ingress controller still terminates locally; a namespace running more than one Gateway keeps the replica while any of them is not on an edge-terminated class; and a namespace whose `TenantGateway` objects cannot be listed, or which carries the Gateway owner label but has no `TenantGateway` yet, keeps it as well, so that a transient read failure never withdraws a key a Gateway may still be serving.
87+
88+
Moving back off an edge-terminated class reverses each row: the `Issuer` is recreated, certificates are requested for the hostnames the Gateway serves, the redirect route returns, and the wildcard replica is restored if `publishing.certificates.wildcardSecretName` is set.
89+
90+
The switch is not atomic in either direction. The wildcard-secret controller watches the `TenantGateway` and prunes as soon as `certMode` changes, while the `Gateway` itself is re-rendered when the tenant's HelmRelease reconciles, and the class controller reprograms after that. A tenant can therefore be briefly serving HTTPS listeners whose Secret has already been removed.
91+
92+
## When the configured names and the installed classes disagree
93+
94+
Nothing in this mechanism checks that a class name corresponds to a `GatewayClass` the cluster actually has. Run `kubectl get gatewayclass` before putting a name into any of the three values.
95+
96+
- **A name in `className` or `tenantSelectableClasses` that matches no installed class.** The tenant's release renders and the `Gateway` object is created; nothing programs it, so it never gets an address and its listeners are never accepted. There is no error at render time.
97+
- **A misspelled name in `edgeTerminatedClasses`.** It matches no class, so edge mode never engages for it and nothing says so. A tenant whose TLS really is terminated upstream stays in a certificate-issuing mode, and its Gateway keeps the port-443 listeners that go with it.
98+
- **A class removed from `tenantSelectableClasses` while a tenant still names it.** That tenant's gateway release starts failing to render. Other tenants are unaffected.
99+
- **`className` changed while a tenant pins the outgoing name.** The allowed set is the *current* default plus `tenantSelectableClasses`, so the pin stops being valid and that tenant's gateway release fails. Add the outgoing class to `tenantSelectableClasses` before changing the default, or update the tenant first.
100+
- **The publishing tenant selecting a class that terminates TLS differently from `className`.** Refused at render time. The cluster-wide ACME `ClusterIssuer`s point their HTTP-01 solver at the publishing tenant's Gateway and are rendered from `gateway.className` alone — a tenant's own choice does not travel on that channel. A publishing Gateway whose termination disagreed with the default would leave the solver pinned to a listener that Gateway does not have, and every HTTP-01 certificate in the cluster would stop issuing with nothing in any status explaining why. Change `gateway.className` instead. The publishing tenant is the namespace named by `publishing.ingressName`, `tenant-root` by default.
101+
- **A comma or a non-string entry in either list.** The platform chart fails to render, naming the entry.
102+
103+
## Checking what a tenant landed on
104+
105+
```bash
106+
kubectl get gatewayclass
107+
108+
kubectl --namespace tenant-acme get tenantgateway cozystack \
109+
--output jsonpath='{.spec.gatewayClassName}{"\t"}{.spec.certMode}{"\n"}'
110+
111+
kubectl --namespace tenant-acme get gateway cozystack \
112+
--output jsonpath='{range .spec.listeners[*]}{.name}{"\t"}{.port}{"\t"}{.hostname}{"\n"}{end}'
113+
114+
kubectl --namespace tenant-acme get certificate,issuer
115+
```
116+
117+
A tenant on an edge-terminated class reports `certMode: edge`, shows port-80 listeners only, and has no `Certificate` or `Issuer` left from an earlier mode. Leftover TLS Secrets from that earlier mode are not shown by the last command — list Secrets of type `kubernetes.io/tls` in the namespace to find them.

‎packages/extra/gateway/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ Owning a separate Gateway makes sense for: a tenant that needs its own LB IP (DN
1616

1717
The platform-wide `publishing.certificates.solver` value selects how the controller sources TLS certificates for the tenant Gateway. Setting `publishing.certificates.wildcardSecretName` switches to a third mode (existing Secret) that overrides the solver entirely, and putting the tenant's GatewayClass in `gateway.edgeTerminatedClasses` switches to a fourth that issues nothing at all — see below.
1818

19+
The sections below are the chart's own reference. For the operator's view of the same surface — the three platform values that bound which class a tenant lands on, what a tenant writes to pick one, what moving between classes does to certificates and to the wildcard replica, and how the mechanism fails when the configured names and the installed classes disagree — see [`docs/gateway-classes.md`](../../../docs/gateway-classes.md).
20+
1921
### Default — HTTP-01
2022

2123
Out of the box, no extra config required. The controller:
@@ -74,6 +76,8 @@ Moving an existing tenant onto such a class also withdraws the operator wildcard
7476

7577
Route hostnames still stay inside the tenant apex; on a class that shares one edge address between all Gateways the class controller has to enforce that itself, since the platform's route-hostname policy gates HTTPRoute and TLSRoute in tenant namespaces but not GRPCRoute, and admits routes that declare no hostnames at all.
7678

79+
The publishing tenant is the one tenant that cannot cross this boundary by itself: the chart refuses to render when it selects a class whose termination differs from `gateway.className`, edge on either side. The cluster-wide ACME `ClusterIssuer`s point their HTTP-01 solver at that tenant's Gateway and are rendered from the platform default class alone, so a mismatch would pin the solver to a listener the Gateway does not have — or leave it unpinned when it does — and every HTTP-01 certificate in the cluster would stop issuing with nothing in any status explaining why. Move `gateway.className` instead. Which namespace counts as the publishing tenant comes from `publishing.ingressName`.
80+
7781
## External IP allocation
7882

7983
The per-tenant Gateway's auto-created `LoadBalancer` Service draws its IP from whatever LB allocator the cluster admin has configured at the platform layer — same shape as ingress-nginx today. Cozystack itself ships MetalLB installed but does not render any `IPAddressPool` / `L2Advertisement` / `BGPAdvertisement` from this chart; admins set up the allocator that suits their environment (MetalLB pool with L2 / BGP, Cilium LB-IPAM with announcer, robotlb against a cloud provider, or `Service.spec.externalIPs` pinning).

0 commit comments

Comments
 (0)