|
| 1 | +# Publishing through a Cloudflare Tunnel |
| 2 | + |
| 3 | +`cloudflare-tunnel-gateway-controller` is an optional system package that registers a second Gateway API implementation alongside the Cilium one. A Gateway on its class is published by Cloudflare: clients reach Cloudflare's edge, and the edge carries the request into the cluster over a Cloudflare Tunnel that the in-cluster data plane opens outbound. The Services the chart renders are `ClusterIP`, so nothing on this path asks the cluster for an address reachable from outside. |
| 4 | + |
| 5 | +## When to reach for it |
| 6 | + |
| 7 | +The Gateway a Cozystack tenant publishes through is backed by Cilium: `packages/extra/gateway` renders a `TenantGateway` whose `gatewayClassName` value defaults to `cilium`, and the cozystack-controller turns that into a `Gateway` plus whatever cert-manager `Issuer` and `Certificate` objects its cert mode calls for. That path wants two things from the outside world — an address clients can open a connection to, and an ACME challenge that completes against the cluster. |
| 8 | + |
| 9 | +The tunnel class drops both for the hostnames it serves. Inbound connections are replaced by an outbound connection from the proxy pods to Cloudflare's edge, and TLS is terminated at the edge rather than on a Gateway listener. It fits a cluster behind NAT or CGNAT, a lab or home cluster with no routable prefix, a site whose firewall will not forward ports, or a deployment that wants Cloudflare's edge in front of a subset of hostnames. |
| 10 | + |
| 11 | +It is the wrong tool when you need TLS passthrough or any non-HTTP protocol, when the certificate a client sees has to be one the cluster controls, or when a third party in the request path is unacceptable. GatewayClasses are cluster-scoped and each Gateway names the one it wants, so enabling this package does not move anything that is already published through Cilium. |
| 12 | + |
| 13 | +## What has to exist on the Cloudflare side |
| 14 | + |
| 15 | +1. A zone whose DNS is served by Cloudflare, covering the hostnames you intend to publish. |
| 16 | +2. A Tunnel, created under **Zero Trust → Networks → Tunnels** with the `cloudflared` connector type. Keep both its **Tunnel ID** and its **tunnel token**. |
| 17 | +3. An API token with the **Account → Cloudflare Tunnel → Edit** permission. |
| 18 | + |
| 19 | +Give the controller a tunnel of its own. Its upstream documentation states that it assumes exclusive ownership of the tunnel configuration, performs a full synchronization on startup, and removes ingress rules that do not come from routes it manages — so a tunnel that also carries hand-written public hostnames, or one shared with another system, loses them. |
| 20 | + |
| 21 | +The Cloudflare account ID is auto-detected when the API token has access to a single account. When it does not, supply it explicitly, either as an `account-id` key in the Secret below or through the chart's `gatewayClassConfig.accountId` value. |
| 22 | + |
| 23 | +## The credentials Secret |
| 24 | + |
| 25 | +Both planes read one Secret, named `cloudflare-tunnel-credentials`, in the package namespace `cozy-cloudflare-tunnel-gateway-controller`. The package pins the Secret name and both required keys, and a chart test holds them in place across re-vendoring: |
| 26 | + |
| 27 | +| Key | Read by | Contents | |
| 28 | +| --- | --- | --- | |
| 29 | +| `api-token` | the controller, through `GatewayClassConfig.spec.cloudflareCredentialsSecretRef` | the Cloudflare API token | |
| 30 | +| `tunnel-token` | the proxy, as the `TUNNEL_TOKEN` environment variable | the tunnel's connector token | |
| 31 | +| `account-id` | the controller, optional | account ID, when auto-detection cannot pick one | |
| 32 | + |
| 33 | +Create it before or shortly after enabling the package — the proxy pod cannot start without it: |
| 34 | + |
| 35 | +```bash |
| 36 | +kubectl create namespace cozy-cloudflare-tunnel-gateway-controller \ |
| 37 | + --dry-run=client --output yaml | kubectl apply --filename - |
| 38 | + |
| 39 | +kubectl --namespace cozy-cloudflare-tunnel-gateway-controller \ |
| 40 | + create secret generic cloudflare-tunnel-credentials \ |
| 41 | + --from-literal=api-token="$CF_API_TOKEN" \ |
| 42 | + --from-literal=tunnel-token="$CF_TUNNEL_TOKEN" |
| 43 | +``` |
| 44 | + |
| 45 | +## Enabling the package |
| 46 | + |
| 47 | +The package is not in any bundle by default, because it cannot become Ready without operator input. Enabling it takes two edits, and the second one lands on an object that exists only after the first. |
| 48 | + |
| 49 | +First, list it in `bundles.enabledPackages` on the platform values — the `platform` component of the `cozystack.cozystack-platform` Package CR: |
| 50 | + |
| 51 | +```yaml |
| 52 | +apiVersion: cozystack.io/v1alpha1 |
| 53 | +kind: Package |
| 54 | +metadata: |
| 55 | + name: cozystack.cozystack-platform |
| 56 | +spec: |
| 57 | + components: |
| 58 | + platform: |
| 59 | + values: |
| 60 | + bundles: |
| 61 | + enabledPackages: |
| 62 | + - cozystack.cloudflare-tunnel-gateway-controller |
| 63 | +``` |
| 64 | +
|
| 65 | +The name also has to stay out of `bundles.disabledPackages`: the platform helper that emits it checks that list first, so a name present in both is not emitted. |
| 66 | + |
| 67 | +That render creates a `Package` named `cozystack.cloudflare-tunnel-gateway-controller`. Second, set the tunnel ID on it: |
| 68 | + |
| 69 | +```yaml |
| 70 | +apiVersion: cozystack.io/v1alpha1 |
| 71 | +kind: Package |
| 72 | +metadata: |
| 73 | + name: cozystack.cloudflare-tunnel-gateway-controller |
| 74 | +spec: |
| 75 | + components: |
| 76 | + cloudflare-tunnel-gateway-controller: |
| 77 | + values: |
| 78 | + cloudflare-tunnel-gateway-controller: |
| 79 | + gatewayClassConfig: |
| 80 | + tunnelID: "00000000-0000-4000-8000-000000000000" |
| 81 | +``` |
| 82 | + |
| 83 | +Until that value is set the chart refuses to render, and the HelmRelease reports `gatewayClassConfig.tunnelID is required`. Because the Package appears only once the name is in `enabledPackages`, the first reconcile after enabling fails this way; setting the tunnel ID changes the values and the install is retried. |
| 84 | + |
| 85 | +The platform writes exactly one key of its own inside that same component values block — `controller.clusterDomain`, taken from `networking.clusterDomain` — because the vendored chart bakes the cluster domain into the proxy's config-endpoint URL and falls back to `cluster.local` when it is empty, which is wrong on a Cozystack default of `cozy.local`. The `gatewayClassConfig` block you add is a different key from the one the platform renders. If a future reconcile ever did drop it, the failure is the loud one above rather than a silently mis-rendered release. |
| 86 | + |
| 87 | +## The GatewayClass, and attaching a Gateway |
| 88 | + |
| 89 | +The package creates one `GatewayClass` named `cloudflare-tunnel`, with `spec.controllerName: cf.k8s.lex.la/tunnel-controller` and a `parametersRef` to the cluster-scoped `GatewayClassConfig` named `cloudflare-tunnel-gateway-controller`, which is where the tunnel ID and the credentials reference land. The class name is cosmetic; the controller binds its GatewayClasses by `controllerName`. |
| 90 | + |
| 91 | +A Gateway joins the class by naming it, and routes attach to that Gateway the usual way: |
| 92 | + |
| 93 | +```yaml |
| 94 | +apiVersion: gateway.networking.k8s.io/v1 |
| 95 | +kind: Gateway |
| 96 | +metadata: |
| 97 | + name: tunnel |
| 98 | + namespace: tenant-example |
| 99 | +spec: |
| 100 | + gatewayClassName: cloudflare-tunnel |
| 101 | + listeners: |
| 102 | + - name: https |
| 103 | + protocol: HTTPS |
| 104 | + port: 443 |
| 105 | +--- |
| 106 | +apiVersion: gateway.networking.k8s.io/v1 |
| 107 | +kind: HTTPRoute |
| 108 | +metadata: |
| 109 | + name: app |
| 110 | + namespace: tenant-example |
| 111 | +spec: |
| 112 | + parentRefs: |
| 113 | + - name: tunnel |
| 114 | + hostnames: |
| 115 | + - app.example.com |
| 116 | + rules: |
| 117 | + - backendRefs: |
| 118 | + - name: app |
| 119 | + port: 80 |
| 120 | +``` |
| 121 | + |
| 122 | +The listener carries no `tls` block: TLS ends at Cloudflare's edge, so there is no certificate for the Gateway to present. This is the listener shape the chart's own installation notes use. |
| 123 | + |
| 124 | +The controller writes the tunnel's CNAME target, `<tunnel-id>.cfargotunnel.com`, into the Gateway's `status.addresses`. external-dns reads that as the target for records it manages; without external-dns, create the CNAME in the Cloudflare zone yourself, proxied. |
| 125 | + |
| 126 | +Routes are `HTTPRoute` and `GRPCRoute`. A backend is normally a Service; the chart also installs an `ExternalBackend` CRD for pointing a route at an out-of-cluster HTTP(S) origin, which a Service of type `ExternalName` cannot express because it carries no scheme. |
| 127 | + |
| 128 | +## Hostname ownership between tenants |
| 129 | + |
| 130 | +By default every Gateway of this class shares one tunnel and one proxy pool, so the data plane picks a backend by hostname across all of them — a route claiming another tenant's hostname would be answered rather than ignored, which is not the case behind a per-tenant Cilium address. |
| 131 | + |
| 132 | +Cozystack's own admission policy does not close that on its own. `cozystack-route-hostname-policy` binds route hostnames to the namespace's `namespace.cozystack.io/host` label, but it matches only `HTTPRoute` and `TLSRoute` in namespaces whose name starts with `tenant-`, it does not see `GRPCRoute`, and it admits a route that declares no hostnames at all. |
| 133 | + |
| 134 | +So the package turns on the chart's own hostname-ownership layer, keyed on the same `namespace.cozystack.io/host` label and scoped to every namespace that carries it. On this class, a route in such a namespace is rejected by the controller unless it declares hostnames explicitly and each one equals the label value or is a subdomain of it. The chart's fail-fast admission half of that feature stays off on purpose: admission cannot resolve `parentRefs`, so it would also police routes bound to the Cilium Gateway in those same namespaces. |
| 135 | + |
| 136 | +## Giving one Gateway its own tunnel |
| 137 | + |
| 138 | +A Gateway can opt out of the shared data plane with a `GatewayConfig` (`cf.k8s.lex.la/v1alpha1`) referenced from its `spec.infrastructure.parametersRef`, in the same namespace. The controller then runs a proxy Deployment and a Cloudflare Tunnel dedicated to that Gateway. `spec.tunnelTokenSecretRef` is the only required field — the tunnel ID and account are parsed out of the connector token rather than configured separately. Reach for it when two sets of hostnames must not share a connector, or when one Gateway needs its own data-plane sizing. |
| 139 | + |
| 140 | +Keep it an operator resource. A `GatewayConfig` also names the image for the Deployment the controller creates on its behalf, and the controller creates that Deployment with the cluster-wide grant described under Limitations. The tenant roles in `cozystack-basics` carry no access to the `cf.k8s.lex.la` group today, which is what keeps that out of tenant reach — widening a tenant's Gateway API access later should not quietly include this group. |
| 141 | + |
| 142 | +## Limitations |
| 143 | + |
| 144 | +- **HTTP and gRPC only.** A Cloudflare Tunnel carries HTTP to the origin and this controller implements `HTTPRoute` and `GRPCRoute`. There is no TLS passthrough and no raw TCP or UDP. |
| 145 | +- **The tenant publishing Gateway needs its passthrough listeners removed.** `packages/extra/gateway` defaults `tlsPassthroughServices` to `api`, `vm-exportproxy` and `cdi-uploadproxy`, each of which becomes a `protocol: TLS`, `mode: Passthrough` listener on the rendered Gateway. A listener this controller does not accept holds the whole TenantGateway at `Ready=False`, because cozystack-controller marks it Ready only when every listener reports both `Accepted` and `Programmed`. Emptying the list clears that, at the price of those three services — the Kubernetes API endpoint among them — not being published through that Gateway. |
| 146 | +- **The HTTP-to-HTTPS redirect does not survive the ownership layer.** The redirect `HTTPRoute` that cozystack-controller renders next to a TenantGateway deliberately carries no hostnames, so it matches every host on the port-80 listener. Routes without explicit hostnames are exactly what the hostname-ownership layer rejects. Enforce HTTPS at the Cloudflare zone level instead. |
| 147 | +- **Certificates are Cloudflare's.** The edge terminates TLS, so the certificate a client sees is the one Cloudflare serves for the zone, not one cert-manager issued in the cluster. Cloudflare's Universal SSL covers the zone apex and one label below it, so a two-label hostname such as `app.tenant.example.com` needs Advanced Certificate Manager or a separate zone per tenant apex. |
| 148 | +- **The controller holds a cluster-wide write grant.** Its ClusterRole is attached by a ClusterRoleBinding, so every rule in it applies in every namespace. It reads Secrets everywhere (`get`, `list`, `watch`, plus `create` for the config-API token it generates), because it resolves credential, TLS and CA-bundle references wherever a Gateway or route points at them. It can also write Deployments, Services, NetworkPolicies and HorizontalPodAutoscalers in any namespace — `create`, `update` and `delete` on all four, plus `patch` on Deployments — which is how it materialises a dedicated data plane next to a Gateway that asks for one. Cluster-wide Deployment creation is the stronger of the two privileges — it places a pod in any namespace, under any ServiceAccount that already exists there — so weigh that, not only the Secret read, when deciding to enable the package. |
| 149 | +- **One controller replica, no leader election.** The chart defaults are `replicaCount: 1` with leader election disabled and the package does not override them, so configuration pushes pause while the controller pod restarts. The proxy runs two replicas. |
| 150 | +- **Tunnel dial is deferred on `protocol: auto`.** The proxy defaults to negotiating QUIC with an HTTP/2 fallback and waits for the controller's first config push before dialing, bounded at roughly 30 seconds — so a proxy on a route-less cluster can take that long to connect on each start. gRPC needs HTTP/2, since cloudflared drops HTTP trailers over QUIC and `grpc-status` is lost with them; `auto` upgrades at startup when a GRPCRoute already exists, so a GRPCRoute added later needs a proxy restart. |
| 151 | +- **A namespace without the ownership label is out of scope, not denied.** The layer is scoped by `namespace.cozystack.io/host` existing, so a namespace that does not carry it is not policed rather than refused. Cozystack's own policy still covers `HTTPRoute` and `TLSRoute` in `tenant-`-prefixed namespaces and is fail-closed on a missing label there, but it does not see `GRPCRoute` — so a tenant namespace caught without its label, mid-creation or after the label is scrubbed, leaves a GRPCRoute free to claim any hostname on the shared tunnel. |
| 152 | +- **Nothing is scraped by default.** Both ServiceMonitors the chart carries are gated on `serviceMonitor.enabled`, which is `false` and which the package does not change, so neither plane reaches the platform's Prometheus. Turning it on is not sufficient on its own: the proxy serves `/metrics` on its config API port, and the proxy's NetworkPolicy — which is on by default — admits that port only from the package namespace, so the monitoring namespace has to be added to `proxy.networkPolicy.ingress.from` in the same change. |
| 153 | +- **Leave the controller's own NetworkPolicy off.** `networkPolicy.enabled` is `false` in the chart and the package does not change it. Turning it on confines controller egress to DNS, TCP 443 and 6443, and the Cloudflare address ranges — none of which covers the proxy's config API port, `8081` by default. Where the CNI enforces NetworkPolicy, the controller can then no longer push configuration to its data plane. The proxy's own policy is a separate value, on by default, and is not affected. |
| 154 | +- **Gateway API version skew.** Cozystack ships the Gateway API CRD bundle at v1.5.1, while the controller's own chart documents installing v1.6.1. A GatewayClass advertises the bundle it was built against through a `SupportedVersion` condition, so that condition reading `False` on a current cluster is expected rather than a fault — read `Accepted` for whether the class is usable. |
| 155 | +- **Images are not mirrored into the Cozystack registry.** The controller and proxy images come from `ghcr.io/lexfrei`, pinned by tag and digest. Air-gapped installs mirror them; the chart itself is vendored into the repository, so nothing fetches it at runtime. |
| 156 | +- **Coverage is template-level.** The package ships chart tests that pin its wiring; there is no end-to-end test in this repository that exercises a live tunnel. |
| 157 | + |
| 158 | +## Upgrades |
| 159 | + |
| 160 | +The vendored chart ships its CRDs in `crds/`, which Helm installs once and never touches again on upgrade. The PackageSource sets `upgradeCRDs: CreateReplace` for that reason: the controller has added kinds across minor releases, and without it a chart bump would land a new controller binary against the old CRD set. |
| 161 | + |
| 162 | +## See also |
| 163 | + |
| 164 | +- [`packages/extra/gateway/README.md`](../packages/extra/gateway/README.md) — the Cilium-backed per-tenant Gateway, its cert modes, and the layered hostname security model. |
| 165 | +- [Cloudflare Tunnel documentation](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/) — creating a tunnel, connector tokens, and edge behaviour. |
| 166 | +- [Gateway API](https://gateway-api.sigs.k8s.io/) — GatewayClass, Gateway and route semantics. |
0 commit comments