Skip to content

Commit 0e2b3d0

Browse files
committed
docs(cloudflare-tunnel): clarify resource ownership and removal
Tenant class selection does not grant raw Gateway API access, and a Ready tenant HelmRelease no longer establishes listener health. Explain those boundaries and the expected platform failure during removal so operators do not wait for a condition that Package deletion must clear. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 72fb1d5 commit 0e2b3d0

1 file changed

Lines changed: 7 additions & 7 deletions

File tree

‎docs/cloudflare-tunnel-gateway.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -158,7 +158,7 @@ gateway:
158158
- cloudflare-tunnel
159159
```
160160

161-
`tenantSelectableClasses` is the allowlist. A tenant naming a class outside it plus `gateway.className` fails its own `gateway` release with a message naming the allowed set, and no other tenant is affected. `edgeTerminatedClasses` declares that the class's provider ends TLS upstream of the Gateway, which is what Cloudflare's edge does here. Leave it out and the tenant Gateway is rendered for in-cluster termination: cert-manager certificates, `mode: Terminate` listeners, and the TLS-passthrough listeners this class has nothing to map onto.
161+
`tenantSelectableClasses` is the allowlist. A tenant naming a class outside it plus `gateway.className` fails its own `gateway` release with a message naming the allowed set, and no other tenant is affected. `edgeTerminatedClasses` declares that the class's provider ends TLS upstream of the Gateway, which is what Cloudflare's edge does here. Leave it out and the tenant Gateway is rendered for in-cluster TLS termination, with certificates and `mode: Terminate` listeners instead of the edge-mode HTTP listeners.
162162

163163
On the Tenant:
164164

@@ -184,7 +184,7 @@ An inheriting tenant does not choose: it publishes through its ancestor's Gatewa
184184

185185
## Hostname ownership between tenants
186186

187-
By default every Gateway of this class shares one tunnel and one proxy pool, so the data plane picks a backend by hostname across all of them — a route claiming another tenant's hostname would be answered rather than ignored, which is not the case behind a per-tenant Cilium address.
187+
By default every Gateway of this class shares one tunnel and one proxy pool, so routes attached to different Gateways still share one hostname-routing space. The tenant roles in `cozystack-basics` do not grant writes to Gateway API or `cf.k8s.lex.la` resources: platform controllers and application releases create these resources, and the standalone examples above require an administrator. Selecting a class through the Tenant does not grant raw Gateway or route access. Hostname checks protect the resources these writers produce and any administrator-created routes.
188188

189189
Cozystack's own admission policy covers most of that. `cozystack-route-hostname-policy` requires every `HTTPRoute`, `TLSRoute` and `GRPCRoute` to declare its hostnames and binds each one to the namespace's `namespace.cozystack.io/host` label, failing closed when the label is missing. It matches only namespaces whose name starts with `tenant-`, though, and it decides once, at admission.
190190

@@ -199,7 +199,7 @@ Keep it an operator resource. A `GatewayConfig` also names the image for the Dep
199199
## Limitations
200200

201201
- **HTTP and gRPC only.** A Cloudflare Tunnel carries HTTP to the origin and this controller implements `HTTPRoute` and `GRPCRoute`. There is no TLS passthrough and no raw TCP or UDP.
202-
- **A tenant Gateway on this class carries HTTPRoute only.** The edge-mode listeners cozystack-controller renders admit `HTTPRoute` alone and no TLS-passthrough listener, so the tenant Kubernetes API, VM export and CDI upload endpoints, and every `GRPCRoute`, need another path. A tenant Gateway reaches this class through the platform allowlist and edge declaration described under [Putting a tenant's own Gateway on the class](#putting-a-tenants-own-gateway-on-the-class). Allowlisted without the edge declaration, it is rendered with `mode: Terminate` and passthrough listeners, and one listener the controller will not accept holds the `TenantGateway` at `Ready=False`, because cozystack-controller marks it Ready only when every listener reports both `Accepted` and `Programmed`.
202+
- **A tenant Gateway on this class carries HTTPRoute only.** The edge-mode listeners cozystack-controller renders admit `HTTPRoute` alone and no TLS-passthrough listener, so the tenant Kubernetes API, VM export and CDI upload endpoints, and every `GRPCRoute`, need another path. A tenant Gateway reaches this class through the platform allowlist and edge declaration described under [Putting a tenant's own Gateway on the class](#putting-a-tenants-own-gateway-on-the-class). Without the edge declaration it instead gets in-cluster TLS termination. Inspect `TenantGateway` listener status when diagnosing that mismatch: the tenant `gateway` HelmRelease waits for the controller to observe its generation, not for every listener to become Ready, so a Ready HelmRelease does not prove the Gateway is serving.
203203
- **No HTTP-to-HTTPS redirect in the cluster.** cozystack-controller renders no redirect route for an edge-terminated `TenantGateway`, and a standalone Gateway on this class gets none either. The redirect route it renders in the other modes carries no hostnames, which the hostname-ownership layer would reject on this class anyway. Enforce HTTPS at the Cloudflare zone level.
204204
- **Certificates are Cloudflare's.** The edge terminates TLS, so the certificate a client sees is the one Cloudflare serves for the zone, not one cert-manager issued in the cluster. Cloudflare's Universal SSL covers the zone apex and one label below it, so a two-label hostname such as `app.tenant.example.com` needs Advanced Certificate Manager or a separate zone per tenant apex.
205205
- **The controller holds a cluster-wide write grant.** Its ClusterRole is attached by a ClusterRoleBinding, so every rule in it applies in every namespace. It reads Secrets everywhere (`get`, `list`, `watch`, plus `create` for the config-API token it generates), because it resolves credential, TLS and CA-bundle references wherever a Gateway or route points at them. It can also write Deployments, Services, NetworkPolicies and HorizontalPodAutoscalers in any namespace — `create`, `update` and `delete` on all four, plus `patch` on Deployments — which is how it materialises a dedicated data plane next to a Gateway that asks for one. And it holds `update` and `patch` on `gateways` and `gatewayclasses` themselves, not only on their `/status` subresources — it writes a finalizer onto a class in use — which reaches the spec of any Gateway in the cluster, including ones belonging to the Cilium class. Cluster-wide Deployment creation is the strongest of these — it places a pod in any namespace, under any ServiceAccount that already exists there — so weigh that, not only the Secret read, when deciding to enable the package.
@@ -211,7 +211,7 @@ Keep it an operator resource. A `GatewayConfig` also names the image for the Dep
211211
- **Leave the controller's own NetworkPolicy off.** `networkPolicy.enabled` is `false` in the chart and the package does not change it. Turning it on confines controller egress to DNS, TCP 443 and 6443, and the Cloudflare address ranges — none of which covers the proxy's config API port, `8081` by default. Where the CNI enforces NetworkPolicy, the controller can then no longer push configuration to its data plane. The proxy's own policy is a separate value, on by default, and is not affected.
212212
- **The Gateway API bundle has to be 1.6.x.** The controller reads the `gateway.networking.k8s.io/bundle-version` annotation off the installed `gatewayclasses` CRD and requires an exact major-and-minor match with the bundle it was built against: a patch difference inside 1.6 is accepted, any other minor is not. That bundle is supplied cluster-wide by the `gateway-api-crds` package, so it is a platform-wide version rather than anything this package can carry on its own: this package works only where `gateway-api-crds` ships 1.6.x, and nothing in the package's own wiring enforces that. Its `dependsOn: cozystack.gateway-api-crds` orders the install and carries no version constraint, so a satisfied dependency is not evidence of a satisfied requirement. Read the bundle off the cluster before enabling. On a cluster whose bundle is a different minor the class comes up `Accepted=True` next to `SupportedVersion=False`, reason `UnsupportedVersion`, with a message naming the minor it wants. Nothing gates serving on that condition, so such an install reads as usable and is not: the Gateway API CRDs carry structural schemas, so any field the controller writes that exists only in the newer bundle is pruned on write with no error recorded anywhere. Once bundle and controller agree the condition turns `True`, but only on that GatewayClass's next reconcile — the controller deliberately does not watch the CRDs, so a spec change, a periodic resync or a controller restart is what recomputes it.
213213
- **Images are not mirrored into the Cozystack registry.** The controller and proxy images come from `ghcr.io/lexfrei`, pinned by tag and digest. Air-gapped installs mirror them; the chart itself is vendored into the repository, so nothing fetches it at runtime.
214-
- **Coverage stops at the chart contract.** The package ships chart tests that pin what these values do to the vendored chart, and no end-to-end test here exercises a live tunnel. The hostname-ownership decision in particular cannot be pinned in this repository: it lives in an `internal/` package of the upstream module, which Go will not let another module import, and the controller resolves its Cloudflare account before it evaluates any route, so there is no credential-free path that reaches a verdict. Upstream drives that decision from one shared vector table through both of its enforcement layers; what the tests here pin is that these values are what select the layer, and that they still select it after a re-vendor.
214+
- **Coverage stops at the chart contract.** The package tests verify rendered configuration, not live tunnel traffic or hostname-ownership decisions. The ownership logic lives in the upstream module's `internal/` packages and is tested there. A supported-configuration check still needs a real tunnel, Gateway API 1.6.x, `GatewayClass` reporting `SupportedVersion=True`, and traffic and hostname-ownership probes; rendering the flags is not a substitute for that check.
215215

216216
## Upgrades
217217

@@ -227,9 +227,9 @@ Deleting the Package while the name is still listed does not stick either. `keep
227227

228228
So, in this order:
229229

230-
1. Delete every Gateway on the `cloudflare-tunnel` class, and the routes attached to them. A tenant Gateway on the class goes by clearing `gatewayClass` on that Tenant (or setting `gateway: false`); then take `cloudflare-tunnel` out of `gateway.tenantSelectableClasses` and `gateway.edgeTerminatedClasses`, so no tenant can select it again.
231-
2. Remove `cozystack.cloudflare-tunnel-gateway-controller` from `bundles.enabledPackages`. The platform HelmRelease now fails on the guard described above, and that is expected: the controller is still running and nothing has been deleted.
232-
3. Delete the Package: `kubectl delete package.cozystack.io cozystack.cloudflare-tunnel-gateway-controller`. The HelmRelease carries an ownerReference back to the Package, so this garbage-collects the release and with it both Deployments, the Services, the RBAC, the GatewayClass and the cluster-scoped `GatewayClassConfig` the chart renders. The next platform reconcile finds no Package and renders cleanly. Deleting the Package before step 2, while the platform still emits it, lands in the state described above.
230+
1. Stop selecting this class on every Tenant, either by clearing `gatewayClass` to use the platform default or by setting `gateway: false`. Remove administrator-created Gateways on this class and their routes. Confirm that no Gateway still names `cloudflare-tunnel` before proceeding; changing a Tenant field alone is not evidence that its Gateway has moved. Then take the class out of `gateway.tenantSelectableClasses` and `gateway.edgeTerminatedClasses`.
231+
2. Remove `cozystack.cloudflare-tunnel-gateway-controller` from `bundles.enabledPackages` and let the platform attempt reconciliation. The HelmRelease now fails on the guard while the Package still exists; this is expected, including during an upgrade. Do not wait for platform readiness before the next step: the guard cannot clear until the Package is removed.
232+
3. Delete the Package: `kubectl --context "$KUBE_CONTEXT" delete package.cozystack.io cozystack.cloudflare-tunnel-gateway-controller`. The HelmRelease carries an ownerReference back to the Package, so this garbage-collects the release and with it both Deployments, the Services, the RBAC, the GatewayClass and the cluster-scoped `GatewayClassConfig` the chart renders. The next platform reconcile finds no Package and renders cleanly. Deleting the Package before step 2, while the platform still emits it, recreates it without the operator-set tunnel ID.
233233

234234
Step 1 has to precede step 3 for a second, unrelated reason. While any Gateway uses the class the controller keeps the `gateway-exists-finalizer.gateway.networking.k8s.io` finalizer on the GatewayClass, and the controller is the only thing that removes it. Delete the Package first and the GatewayClass is left in `Terminating` behind a finalizer nothing will clear — recovering from that means editing the finalizer off by hand.
235235

0 commit comments

Comments
 (0)