Skip to content

Commit d360c28

Browse files
committed
docs(cloudflare-tunnel): add the operator guide for the tunnel class
Covers what the package buys over the Cilium GatewayClass, the Cloudflare-side prerequisites, the single credentials Secret and its key names, the two-step enable path through bundles.enabledPackages and the tunnel ID on the emitted Package, how a Gateway joins the cloudflare-tunnel class, and the limits an operator hits: HTTP and gRPC only, TLS terminated at the edge, the passthrough listeners a tenant Gateway has to give up, cluster-wide Secret read on the controller's ClusterRole, and unmirrored images. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 6a64567 commit d360c28

2 files changed

Lines changed: 162 additions & 0 deletions

File tree

‎docs/cloudflare-tunnel-gateway.md‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
# Publishing through a Cloudflare Tunnel
2+
3+
`cloudflare-tunnel-gateway-controller` is an optional system package that registers a second Gateway API implementation alongside the Cilium one. A Gateway on its class is published by Cloudflare: clients reach Cloudflare's edge, and the edge carries the request into the cluster over a Cloudflare Tunnel that the in-cluster data plane opens outbound. Every Service the chart renders is `ClusterIP`, and it renders no `LoadBalancer` at all, so nothing on this path asks the cluster for an address reachable from outside.
4+
5+
## When to reach for it
6+
7+
The Gateway a Cozystack tenant publishes through is backed by Cilium: `packages/extra/gateway` renders a `TenantGateway` whose `gatewayClassName` value defaults to `cilium`, and the cozystack-controller turns that into a `Gateway` plus a cert-manager `Issuer` and `Certificate` per listener. That path wants two things from the outside world — an address clients can open a connection to, and an ACME challenge that completes against the cluster.
8+
9+
The tunnel class drops both for the hostnames it serves. Inbound connections are replaced by an outbound connection from the proxy pods to Cloudflare's edge, and TLS is terminated at the edge rather than on a Gateway listener. It fits a cluster behind NAT or CGNAT, a lab or home cluster with no routable prefix, a site whose firewall will not forward ports, or a deployment that wants Cloudflare's edge in front of a subset of hostnames.
10+
11+
It is the wrong tool when you need TLS passthrough or any non-HTTP protocol, when the certificate a client sees has to be one the cluster controls, or when a third party in the request path is unacceptable. GatewayClasses are cluster-scoped and each Gateway names the one it wants, so enabling this package does not move anything that is already published through Cilium.
12+
13+
## What has to exist on the Cloudflare side
14+
15+
1. A zone whose DNS is served by Cloudflare, covering the hostnames you intend to publish.
16+
2. A Tunnel, created under **Zero Trust → Networks → Tunnels** with the `cloudflared` connector type. Keep both its **Tunnel ID** and its **tunnel token**.
17+
3. An API token with the **Account → Cloudflare Tunnel → Edit** permission.
18+
19+
Give the controller a tunnel of its own. Its upstream documentation states that it assumes exclusive ownership of the tunnel configuration, performs a full synchronization on startup, and removes ingress rules that do not come from routes it manages — so a tunnel that also carries hand-written public hostnames, or one shared with another system, loses them.
20+
21+
The Cloudflare account ID is auto-detected when the API token has access to a single account. When it does not, supply it explicitly, either as an `account-id` key in the Secret below or through the chart's `gatewayClassConfig.accountId` value.
22+
23+
## The credentials Secret
24+
25+
Both planes read one Secret, named `cloudflare-tunnel-credentials`, in the package namespace `cozy-cloudflare-tunnel-gateway-controller`. The package pins that name and those keys, and a chart test in the package holds them in place across re-vendoring:
26+
27+
| Key | Read by | Contents |
28+
| --- | --- | --- |
29+
| `api-token` | the controller, through `GatewayClassConfig.spec.cloudflareCredentialsSecretRef` | the Cloudflare API token |
30+
| `tunnel-token` | the proxy, as the `TUNNEL_TOKEN` environment variable | the tunnel's connector token |
31+
| `account-id` | the controller, optional | account ID, when auto-detection cannot pick one |
32+
33+
Create it before or shortly after enabling the package — the proxy pod cannot start without it:
34+
35+
```bash
36+
kubectl create namespace cozy-cloudflare-tunnel-gateway-controller \
37+
--dry-run=client --output yaml | kubectl apply --filename -
38+
39+
kubectl --namespace cozy-cloudflare-tunnel-gateway-controller \
40+
create secret generic cloudflare-tunnel-credentials \
41+
--from-literal=api-token="$CF_API_TOKEN" \
42+
--from-literal=tunnel-token="$CF_TUNNEL_TOKEN"
43+
```
44+
45+
## Enabling the package
46+
47+
The package is not in any bundle by default, because it cannot become Ready without operator input. Enabling it takes two edits, and the second one lands on an object that exists only after the first.
48+
49+
First, list it in `bundles.enabledPackages` on the platform values — the `platform` component of the `cozystack.cozystack-platform` Package CR:
50+
51+
```yaml
52+
apiVersion: cozystack.io/v1alpha1
53+
kind: Package
54+
metadata:
55+
name: cozystack.cozystack-platform
56+
spec:
57+
components:
58+
platform:
59+
values:
60+
bundles:
61+
enabledPackages:
62+
- cozystack.cloudflare-tunnel-gateway-controller
63+
```
64+
65+
The name also has to stay out of `bundles.disabledPackages`: the platform helper that emits every package checks that list first, so a name present in both is not emitted.
66+
67+
That render creates a `Package` named `cozystack.cloudflare-tunnel-gateway-controller`. Second, set the tunnel ID on it:
68+
69+
```yaml
70+
apiVersion: cozystack.io/v1alpha1
71+
kind: Package
72+
metadata:
73+
name: cozystack.cloudflare-tunnel-gateway-controller
74+
spec:
75+
components:
76+
cloudflare-tunnel-gateway-controller:
77+
values:
78+
cloudflare-tunnel-gateway-controller:
79+
gatewayClassConfig:
80+
tunnelID: "00000000-0000-4000-8000-000000000000"
81+
```
82+
83+
Until that value is set the chart refuses to render, and the HelmRelease reports `gatewayClassConfig.tunnelID is required`. Because the Package appears only once the name is in `enabledPackages`, the first reconcile after enabling always fails this way; setting the tunnel ID changes the values and the install is retried.
84+
85+
The platform writes exactly one key of its own inside that same component values block — `controller.clusterDomain`, taken from `networking.clusterDomain` — because the vendored chart bakes the cluster domain into the proxy's config-endpoint URL and falls back to `cluster.local` when it is empty, which is wrong on a Cozystack default of `cozy.local`. The `gatewayClassConfig` block you add is a different key from the one the platform renders. If a future reconcile ever did drop it, the failure is the loud one above rather than a silently mis-rendered release.
86+
87+
## The GatewayClass, and attaching a Gateway
88+
89+
The package creates one `GatewayClass` named `cloudflare-tunnel`, with `spec.controllerName: cf.k8s.lex.la/tunnel-controller` and a `parametersRef` to the `GatewayClassConfig` named `cloudflare-tunnel-gateway-controller` in the package namespace. The class name is cosmetic; the controller binds by `controllerName`.
90+
91+
A Gateway joins the class by naming it, and routes attach to that Gateway the usual way:
92+
93+
```yaml
94+
apiVersion: gateway.networking.k8s.io/v1
95+
kind: Gateway
96+
metadata:
97+
name: tunnel
98+
namespace: tenant-example
99+
spec:
100+
gatewayClassName: cloudflare-tunnel
101+
listeners:
102+
- name: https
103+
protocol: HTTPS
104+
port: 443
105+
---
106+
apiVersion: gateway.networking.k8s.io/v1
107+
kind: HTTPRoute
108+
metadata:
109+
name: app
110+
namespace: tenant-example
111+
spec:
112+
parentRefs:
113+
- name: tunnel
114+
hostnames:
115+
- app.example.com
116+
rules:
117+
- backendRefs:
118+
- name: app
119+
port: 80
120+
```
121+
122+
The listener carries no `tls` block: TLS ends at Cloudflare's edge, so there is no certificate for the Gateway to present. This is the listener shape the chart's own installation notes use.
123+
124+
The controller writes the tunnel's CNAME target, `<tunnel-id>.cfargotunnel.com`, into the Gateway's `status.addresses`. external-dns reads that as the target for records it manages; without external-dns, create the CNAME in the Cloudflare zone yourself, proxied.
125+
126+
Routes are `HTTPRoute` and `GRPCRoute`. A backend is normally a Service; the chart also installs an `ExternalBackend` CRD for pointing a route at an out-of-cluster HTTP(S) origin, which a Service of type `ExternalName` cannot express because it carries no scheme.
127+
128+
## Hostname ownership between tenants
129+
130+
By default every Gateway of this class shares one tunnel and one proxy pool, so the data plane picks a backend by hostname across all of them — a route claiming another tenant's hostname would be answered rather than ignored, which is not the case behind a per-tenant Cilium address.
131+
132+
Cozystack's own admission policy does not close that on its own. `cozystack-route-hostname-policy` binds route hostnames to the namespace's `namespace.cozystack.io/host` label, but it matches only `HTTPRoute` and `TLSRoute` in namespaces whose name starts with `tenant-`, it does not see `GRPCRoute`, and it admits a route that declares no hostnames at all.
133+
134+
So the package turns on the chart's own hostname-ownership layer, keyed on the same `namespace.cozystack.io/host` label and scoped to every namespace that carries it. On this class, a route in such a namespace is rejected by the controller unless it declares hostnames explicitly and each one equals the label value or is a subdomain of it. The chart's fail-fast admission half of that feature stays off on purpose: admission cannot resolve `parentRefs`, so it would also police routes bound to the Cilium Gateway in those same namespaces.
135+
136+
## Giving one Gateway its own tunnel
137+
138+
A Gateway can opt out of the shared data plane with a `GatewayConfig` (`cf.k8s.lex.la/v1alpha1`) referenced from its `spec.infrastructure.parametersRef`, in the same namespace. The controller then runs a proxy Deployment and a Cloudflare Tunnel dedicated to that Gateway. `spec.tunnelTokenSecretRef` is the only required field — the tunnel ID and account are parsed out of the connector token rather than configured separately. Reach for it when two sets of hostnames must not share a connector, or when one Gateway needs its own data-plane sizing.
139+
140+
## Limitations
141+
142+
- **HTTP and gRPC only.** A Cloudflare Tunnel carries HTTP to the origin and this controller implements `HTTPRoute` and `GRPCRoute`. There is no TLS passthrough and no raw TCP or UDP.
143+
- **The tenant publishing Gateway needs its passthrough listeners removed.** `packages/extra/gateway` defaults `tlsPassthroughServices` to `api`, `vm-exportproxy` and `cdi-uploadproxy`, each of which becomes a `protocol: TLS`, `mode: Passthrough` listener on the rendered Gateway. A listener this controller does not accept holds the whole TenantGateway at `Ready=False`, because cozystack-controller marks it Ready only when every listener reports both `Accepted` and `Programmed`. Emptying the list clears that, at the price of those three services — the Kubernetes API endpoint among them — not being published through that Gateway.
144+
- **The HTTP-to-HTTPS redirect does not survive the ownership layer.** The redirect `HTTPRoute` that cozystack-controller renders next to a TenantGateway deliberately carries no hostnames, so it matches every host on the port-80 listener. Routes without explicit hostnames are exactly what the hostname-ownership layer rejects. Enforce HTTPS at the Cloudflare zone level instead.
145+
- **Certificates are Cloudflare's.** The edge terminates TLS, so the certificate a client sees is the one Cloudflare serves for the zone, not one cert-manager issued in the cluster. Cloudflare's Universal SSL covers the zone apex and one label below it, so a two-label hostname such as `app.tenant.example.com` needs Advanced Certificate Manager or a separate zone per tenant apex.
146+
- **The controller can read every Secret in the cluster.** Its ClusterRole grants `get`, `list`, `watch` and `create` on `secrets` cluster-wide — it resolves credential, TLS and CA-bundle references in arbitrary namespaces and creates the proxy's config-API token.
147+
- **One controller replica, no leader election.** The chart defaults are `replicaCount: 1` with leader election disabled and the package does not override them, so configuration pushes pause while the controller pod restarts. The proxy runs two replicas.
148+
- **Tunnel dial is deferred on `protocol: auto`.** The proxy defaults to negotiating QUIC with an HTTP/2 fallback and waits for the controller's first config push before dialing, bounded at roughly 30 seconds — so a proxy on a route-less cluster can take that long to connect on each start. gRPC needs HTTP/2, since cloudflared drops HTTP trailers over QUIC and `grpc-status` is lost with them; `auto` upgrades at startup when a GRPCRoute already exists, so a GRPCRoute added later needs a proxy restart.
149+
- **Images are not mirrored into the Cozystack registry.** The controller and proxy images come from `ghcr.io/lexfrei`, pinned by tag and digest. Air-gapped installs mirror them; the chart itself is vendored into the repository, so nothing fetches it at runtime.
150+
- **Coverage is template-level.** The package ships chart tests that pin its wiring; there is no end-to-end test in this repository that exercises a live tunnel.
151+
152+
## Upgrades
153+
154+
The vendored chart ships its CRDs in `crds/`, which Helm installs once and never touches again on upgrade. The PackageSource sets `upgradeCRDs: CreateReplace` for that reason: the controller has added kinds across minor releases, and without it a chart bump would land a new controller binary against the old CRD set.
155+
156+
## See also
157+
158+
- [`packages/extra/gateway/README.md`](../packages/extra/gateway/README.md) — the Cilium-backed per-tenant Gateway, its cert modes, and the layered hostname security model.
159+
- [Cloudflare Tunnel documentation](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/) — creating a tunnel, connector tokens, and edge behaviour.
160+
- [Gateway API](https://gateway-api.sigs.k8s.io/) — GatewayClass, Gateway and route semantics.

‎packages/system/cloudflare-tunnel-gateway-controller/values.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# Operator guide: docs/cloudflare-tunnel-gateway.md
2+
#
13
# Cloudflare Tunnel Gateway API controller: publishes HTTPRoutes/GRPCRoutes
24
# through a Cloudflare Tunnel instead of a public LoadBalancer. Opt-in
35
# (bundles.enabledPackages), and needs two operator-supplied inputs before

0 commit comments

Comments
 (0)