Skip to content

Commit a2cb47c

Browse files
authored
feat(gateway): update the CRD bundle to Gateway API v1.6.1 and gate GRPCRoute hostnames (#3872)
## What this PR does Bumps the vendored Gateway API CRD bundle in `packages/system/gateway-api-crds` from v1.5.1 to v1.6.1. The package Makefile pins the upstream ref, so the change is that pin plus the re-rendered `templates/crds-experimental.yaml`. The bundle version is not only bookkeeping: it is a compatibility signal consumers read at runtime. Every CRD in the bundle carries a `gateway.networking.k8s.io/bundle-version` annotation, and a Gateway API controller can read it off the `gatewayclasses` CRD to decide whether the installed bundle matches the one it was built against. Controllers that do this typically accept any patch release but require an exact major and minor match, and report `SupportedVersion=False` with reason `UnsupportedVersion` on their GatewayClass when the minor differs, a status condition that surfaces the skew rather than a functional break. The Cloudflare Tunnel Gateway API controller proposed for the platform in #3858 is built against Gateway API v1.6.1 and performs exactly that comparison, so against the v1.5.1 bundle shipped today its GatewayClass reports the bundle unsupported, and with this bump it reports supported. Neither change depends on the other landing first: this PR only moves the bundle, and the check lives entirely on the consumer side. One consequence worth stating plainly, since it will outlive this PR: an exact-minor check makes the shipped bundle version a coupled pin for any such controller. A future bump to v1.7.x flips the same condition back to `UnsupportedVersion` until that controller is rebuilt against the newer bundle, so the two want to move together. The bundle still emits the same twelve CustomResourceDefinitions and the same two `safe-upgrades.gateway.networking.k8s.io` admission objects, and every CRD still carries the `helm.sh/resource-policy: keep` stamp the Makefile's post-processing step adds. One of the policy's two CEL rules is rewritten, though, and it changes what a cluster accepts. That has its own section below. Two kinds change their version surface: TCPRoute and UDPRoute gain a served `v1` that takes over as the storage version, and their `v1alpha2` stays served with a deprecation marker. Every other kind keeps exactly the versions it served at v1.5.1. That matters because the Cilium operator resolves GatewayClass, Gateway, HTTPRoute and GRPCRoute in `v1`, ReferenceGrant in `v1beta1` and TLSRoute in `v1alpha2` before it enables its Gateway API controller; all six are still served by the v1.6.1 bundle. TLSRoute in particular keeps `v1alpha2`, which the tenant gateway controller, the platform TLSRoute templates and the tenant hostname admission policy all address. Schema changes are small and land on fields nothing in this repository sets. `sessionPersistence.idleTimeout` is gone from HTTPRoute, GRPCRoute and XBackendTrafficPolicy, removed upstream under GEP-1619 because no implementation ever implemented it and HTTP cookies have no native idle-timeout mechanism, so no working behaviour is lost even though the field disappears from the schema; HTTPRoute gains a lower bound on `retry.attempts` and now rejects duplicate `retry.codes`; ReferenceGrant requires `spec` at the root. In the other direction, Gateway infrastructure annotations and frontend CA certificate references go from a limit of 8 to 16, and TLSRoute hostnames from 16 to 1024. No CRD schema loses or changes a CEL validation rule, and the six added belong exclusively to the new TCPRoute and UDPRoute `v1` schemas. ### The safe-upgrades admission policy accepts a different set of bundles The experimental kustomization ships a `ValidatingAdmissionPolicy` and its binding, both named `safe-upgrades.gateway.networking.k8s.io`, next to the twelve CRDs. The binding is `validationActions: [Deny]` with `failurePolicy: Fail`, matching CREATE and UPDATE on `apiextensions.k8s.io/v1 customresourcedefinitions` cluster-wide, and Cozystack installs it in two places: on the host through `cozystack.gateway-api-crds`, and inside every guest cluster that sets `addons.gatewayAPI.enabled`. Its second rule reads the `gateway.networking.k8s.io/bundle-version` annotation off an incoming CRD. At v1.5.1 that rule was a blacklist: reject anything matching `v1.[0-4].\d+` or containing `v0`, allow the rest. At v1.6.1 it is a whitelist: allow `v0.0.0-dev` outright, otherwise require the annotation to start with `v1.` and not match `^v1\.[0-4](\.|$)`. Evaluating both expressions over a spread of annotation values: | bundle-version | v1.5.1 policy | v1.6.1 policy | | --- | --- | --- | | `v1.6.1`, `v1.5.1`, `v1.5.0-dev`, `v1.10.0` | allow | allow | | `v1.4.0`, `v1.4.10`, `v1.0.0` | reject | reject | | `v0.0.0-dev` | reject | **allow** | | `v2.0.0` | allow | **reject** | | `1.6.1` (no leading `v`) | allow | **reject** | | `v1.4` (no patch) | allow | **reject** | | `main`, empty string | allow | **reject** | Nothing in this repository trips the new rejections: every CRD the bundle ships is stamped `v1.6.1`. What is exposed is an operator applying a Gateway API CRD set from somewhere else on a cluster that carries the policy. A fork, a vendor build or a nightly whose annotation reads `1.6.1` or `v2.0.0` used to go in and is now denied with `reason: Invalid`, naming a policy the operator may not know is installed, and whatever HelmRelease owns those CRDs then sits un-Ready. Uninstalling the policy is the escape hatch, and the denial message says so. The same hunk fixes an upstream inconsistency: at v1.5.1 the policy pair was stamped `v1.5.0-dev` while its twelve CRDs said `v1.5.1`. At v1.6.1 all fourteen documents agree. The Gateway and HTTPRoute objects the tenant gateway controller renders were serialized across all three certificate modes and checked against both the v1.5.1 and the v1.6.1 schemas; the two runs are identical, so the bump does not change what the controller can create. `go.mod` still pins `sigs.k8s.io/gateway-api` v1.4.1 and this PR deliberately leaves it there. The CRD bundle and the Go client types have been on separate versions since before this change, and nothing the controller emits depends on the gap. One thing to know before the next bump: from v1.6 on, the two things upstream calls the experimental bundle disagree. This package follows `config/crd/experimental`, the kustomization, which emits 12 CRDs at v1.6.1; the published `experimental-install.yaml` for the same tag emits 13, the extra one being `xbackends.gateway.networking.x-k8s.io`, new in v1.6 and not listed in the kustomization. At v1.5.1 the two agreed, so there was nothing to choose between. Nothing in this repository references XBackend and no vendored CRD depends on it, so following the kustomization costs nothing today, but whoever bumps next inherits the choice, and it should be made deliberately rather than by whichever command gets typed. The second commit corrects a comment this bump invalidated. It claimed TLSRoute stays at `v1alpha2` through Gateway API v1.5 and that the promotion to `v1` was still upstream-only; TLSRoute has been served at `v1` since v1.5.0, so the claim was already wrong before this bump and the version range it named is now wrong too. The third commit closes a gap that correction exposed. `cozystack-route-hostname-policy-tls` named `v1alpha2` alone while the bundle serves `tlsroutes` at `v1`, `v1alpha2` and `v1alpha3`: three served versions against one named. The gap is not new and this bump neither creates nor widens it: the v1.5.1 bundle and the v1.6.1 one serve the same three versions, with the same two marked deprecated. Coverage of the two unnamed versions rested entirely on request conversion, because `matchPolicy` is unset on both rules in that file and so defaults to `Equivalent`, which rewrites a request submitted under any served version into the one the rule names before the policy sees it. That is why there is no bypass to demonstrate, and equally why the arrangement is brittle: the conversion has nothing left to convert into once no named version is served, and a rule that selects nothing is not caught by `failurePolicy: Fail`, because nothing failed. The rule now names all three, which is the shape the HTTPRoute rule above it already used, and `hack/route-hostname-policy-version-coverage.bats` reads the served versions out of the vendored bundle and fails when a policy does not cover every one of them, so a future bundle move cannot outrun the list silently, in either direction. That guard carries negative fixtures rather than only watching a correct tree pass. Three of them drive `assert_covers` itself, with the two readers redefined in a subshell, against an omitted served version, an empty served set and an empty named set. Each requires the specific complaint text rather than merely a non-zero exit, so a helper renamed out from under it fails for the right reason instead of reading as a successful rejection. A fourth drives the comparison helper directly. The fourth commit pins the vendored bundle version in the CRD package's unit suite. Nothing outside the package Makefile named a Gateway API version, so re-running `make update` against a different ref landed green: twelve CRDs, every one stamped, document count unchanged, suite passing. The new case reads the `gateway.networking.k8s.io/bundle-version` annotation off every rendered CRD and fails when it is not the version the Makefile pins. The fifth commit extends the version-coverage guard to `cozystack-gateway-hostname-policy`, which names the `gateways` versions literally the same way the two route policies do, and whose version set lives in the vendored bundle and moves on `make update` there. It matched the served set already; the guard now says so on every run instead of leaving one of the three hostname policies unpinned. Two notes for whoever reads the release: the TCPRoute and UDPRoute storage version moves from `v1alpha2` to `v1`, and while nothing in this repository ships either kind and the CRDs declare no conversion block, an upgraded cluster accumulates both entries in `status.storedVersions` — inert today, load-bearing only if a later bundle stops serving `v1alpha2`. The sixth commit gives GRPCRoute the same hostname rule the other two route kinds have. GRPCRoute carries `spec.hostnames` and it can reach a tenant Gateway: the port-443 listeners pin `allowedRoutes.kinds` to HTTPRoute and TLSRoute, but the port-80 listener leaves `kinds` unset, and an unset list resolves to the kinds the listener protocol supports, which for HTTP includes GRPCRoute. A GRPCRoute in a tenant namespace could claim any hostname on the plaintext listener with nothing checking it. Tenants hold no `gateway.networking.k8s.io` RBAC in Cozystack, so this is the same defence-in-depth against a chart bug that the other two rules are, not a tenant-user control. The rule is the HTTPRoute rule with `grpcroutes` in place of `httproutes` and the one version the bundle serves for it, and the coverage guard pins that list to the bundle like the others. Several places in the tree enumerate which route kinds the hostname policies cover, and all of them named two: the port-443 kinds invariant in the tenant gateway controller and the test that pins it, `packages/extra/gateway/README.md`, and the threat model together with the self-assessment. They name three now. The port-443 kinds set also carries an accurate reason now. cilium#45559 governs only that every port-443 listener carries the same set, not what is in it, so the contents are a least-privilege choice: nothing the platform ships needs gRPC, TCP or UDP routing on port 443. The README had also described HTTPS and TLS-passthrough listeners as carrying different kinds, where the controller gives both the same one. One more commit makes the suite's existing assertions bite. A ValidatingAdmissionPolicyBinding whose `policyName` points at another policy renders without complaint and leaves its own policy inert, and the suite pinned `validationActions` on the bindings while never pinning `policyName` at all, so pointing the GRPCRoute binding at the HTTPRoute policy kept every case green with the GRPCRoute control doing nothing. All three bindings now pin the policy they name. In the same file, `uncovered()` assigned to the variable names its caller reads back when printing a mismatch, so a call site with the arguments swapped would have printed a diagnostic naming the wrong side. Two things are named rather than made. The version-coverage guard collects `apiVersions` from every `resourceRules` entry mentioning the resource without also matching on `apiGroups` or `operations`, so a rule in another group, or one scoped to a single verb, would donate its versions to the set coverage is measured against; the chart's own suite pins the group and the resource on the rule it checks, which closes that a layer below. And `packages/system/cozystack-api/templates/api-tlsroute.yaml` renders `gateway.networking.k8s.io/v1alpha2` while the bundle serves `v1` as storage and marks `v1alpha2` deprecated, but the identical pin sits in the kubevirt and kubevirt-cdi TLSRoute templates, and the pinned `sigs.k8s.io/gateway-api` carries no `apis/v1` TLSRoute type for the controller side to follow, so moving one of the three here would leave the tree less consistent than leaving all three. Closes #3789. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff, file by file: neither `packages/system/gateway-api-crds/Makefile` nor its rendered CRD template matches any entry. Leaving every box empty rather than ticking the "no downstream repository" line, because the docs site carries a Gateway API version statement that is already inaccurate and needs a decision that is not mine: `content/en/docs/next/networking/gateway-api.md` and its v1.5 and v1.6 copies say Gateway API v1.5 ships TLSRoute at `v1alpha2` and that the graduation to `v1` is still upstream-only, when v1.5.0 already shipped TLSRoute `v1` as the storage version. ### Release note ```release-note chore(gateway-api-crds): update the vendored Gateway API CRD bundle to v1.6.1. TCPRoute and UDPRoute gain a served v1 that becomes their storage version, with v1alpha2 still served and now deprecated; every other kind keeps the versions it served at v1.5.1. Upstream removed `sessionPersistence.idleTimeout` from HTTPRoute, GRPCRoute and XBackendTrafficPolicy; the field is not set by anything Cozystack renders, but a hand-written route that carries it loses it silently on its next write, with no admission error. The bundle's `safe-upgrades.gateway.networking.k8s.io` ValidatingAdmissionPolicy changes which CRDs it accepts. Cozystack installs it cluster-wide on the host and in every guest cluster with `addons.gatewayAPI.enabled`. Its `gateway.networking.k8s.io/bundle-version` rule was a blacklist on two patterns and is now a whitelist on the `v1.` prefix with a `v0.0.0-dev` escape hatch. A bundle annotated `v0.0.0-dev` is now accepted where it was denied. A bundle annotated `v2.0.0`, `1.6.1` without the leading `v`, `main`, `v1.4` without a patch, or carrying an empty annotation is now denied where it was accepted. Nothing Cozystack ships is affected, since every CRD in the bundle is stamped `v1.6.1`, but applying a Gateway API CRD set from a fork, a vendor build or a nightly on such a cluster can now fail admission with `reason: Invalid`. Uninstalling the policy remains the documented escape hatch. fix(cozystack-basics): the tenant TLSRoute hostname admission policy now names every served TLSRoute version (`v1`, `v1alpha2`, `v1alpha3`) instead of `v1alpha2` alone. Coverage of the other two rested on request conversion, which holds only while a named version stays served, and a rule left selecting nothing is not caught by `failurePolicy: Fail`. feat(cozystack-basics): GRPCRoute hostnames in `tenant-*` namespaces are now gated by the same admission policy that gates HTTPRoute and TLSRoute. A tenant Gateway's port-80 listener leaves `allowedRoutes.kinds` unset, which admits GRPCRoute by protocol default, so a GRPCRoute could claim any hostname on the plaintext listener unchecked. Tenants hold no `gateway.networking.k8s.io` RBAC, so this is defence-in-depth against a chart bug rather than a tenant-user control. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added hostname admission validation for `GRPCRoute` resources. - Expanded `TLSRoute` hostname validation to support all served Gateway API versions. - Ensured route hostname policies consistently cover HTTP, TLS, and gRPC routes. - **Security** - Updated listener policies to restrict port-443 routes to supported HTTP and TLS route types. - Clarified fail-closed hostname validation behavior across Gateway, Route, and Ingress resources. - **Documentation** - Updated security and threat-model documentation to include GRPCRoute handling and validation coverage. - **Maintenance** - Updated the bundled Gateway API CRDs to version 1.6.1. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents b7e5100 + fb740db commit a2cb47c

11 files changed

Lines changed: 2362 additions & 648 deletions

File tree

‎docs/security/self-assessment.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ Actors are decomposed by **trust boundary**: two components are distinct actors
6969
| Authenticate | End user (T4) → Keycloak → kube-apiserver | OIDC token (`groups`, `preferred_username`) | When OIDC is enabled (off by default): kube-apiserver OIDC verification; UIs fronted by oauth2-proxy (the `gatekeeper` Deployment). In the default OIDC-disabled mode the dashboard uses a token-proxy and the LINSTOR GUI is not externally exposed. | In OIDC mode, an identity whose `groups` map to the `<tenant>-view/use/admin/super-admin` roles. In the default mode the token-proxy forwards a Kubernetes-issued ServiceAccount JWT as the bearer token, so authorization follows that ServiceAccount's RBAC. The tenant ServiceAccount is bound into the tenant's view/use/admin **and super-admin** RoleBindings, so a holder of the tenant SA token has super-admin-level access over the tenant subtree (equivalent to the `<tenant>-super-admin` group), not merely the base `cozy:tenant` role. |
7070
| Create an application | Tenant admin (T2) → `cozystack-api` (T1) → Flux (T1) → workload (T3) | Application CR `spec` (Helm values only) | RBAC (delegated to kube-apiserver via `system:auth-delegator`); name and reserved-key validation; admission chain; **server-fixed chart reference**. | Managed service provisioned from a vetted chart; tenant never touched a HelmRelease. |
7171
| Create a tenant / nested tenant | Tenant super-admin (T2) → `cozystack-api` → Flux → new namespace (T2) | Tenant CR (`spec.host`, quotas) | Name checks; **best-effort hierarchical quota cap** (declaration-time snapshot; concurrent writes can briefly overshoot, with a runtime controller backstop); `cozystack-tenant-host-policy` restricts `spec.host` to trusted callers. | New tenant namespace with RBAC, NetworkPolicies, Keycloak groups, quota. |
72-
| Publish a hostname (Gateway/HTTPRoute/TLSRoute/Ingress) | Tenant Application spec (T2) → Flux/controllers (T1) → core kube-apiserver | Rendered Gateway/Route/Ingress hostnames; namespace host label | Tenants have **no** RBAC to write Gateways, Routes, Ingresses, or namespace labels — Flux/controllers render them from the tenant's Application spec. The hostname VAPs (Gateway/Route policies, and `cozystack-ingress-hostname-policy` for the default legacy-Ingress path) validate rendered hostnames against the namespace host label (defense-in-depth). The apex is platform-set (`cozystack-tenant-host-policy` restricts `Tenant.spec.host` to trusted callers; the namespace label is immutable). Route/namespace/Ingress policies fail-closed; the Gateway-listener policy is fail-open when the label is absent. | A tenant cannot claim or hijack another tenant's platform-apex hostname, provided its namespace carries the host label. |
72+
| Publish a hostname (Gateway/HTTPRoute/TLSRoute/GRPCRoute/Ingress) | Tenant Application spec (T2) → Flux/controllers (T1) → core kube-apiserver | Rendered Gateway/Route/Ingress hostnames; namespace host label | Tenants have **no** RBAC to write Gateways, Routes, Ingresses, or namespace labels — Flux/controllers render them from the tenant's Application spec. The hostname VAPs (Gateway/Route policies, and `cozystack-ingress-hostname-policy` for the default legacy-Ingress path) validate rendered hostnames against the namespace host label (defense-in-depth). The apex is platform-set (`cozystack-tenant-host-policy` restricts `Tenant.spec.host` to trusted callers; the namespace label is immutable). Route/namespace/Ingress policies fail-closed; the Gateway-listener policy is fail-open when the label is absent. | A tenant cannot claim or hijack another tenant's platform-apex hostname, provided its namespace carries the host label. |
7373
| Create a network policy | Tenant admin (T2) → `cozystack-api` | `sdn.cozystack.io/SecurityGroup` spec | RBAC; projected to a `CiliumNetworkPolicy` under the API server's ServiceAccount; tenants cannot write raw `cilium.io` objects. | CNI policy programmed on the tenant's behalf. |
7474
| Delete a platform-critical object | Any actor → core kube-apiserver | DELETE request | `cozystack-no-delete-guardrail` denies DELETE on `platform.cozystack.io/no-delete=true` objects. Operational guidance, not adversarial defense: an actor able to update the object can remove the label first (tenants cannot reach these roots; only T0/T1). | Accidental teardown of platform roots blocked. |
7575
| East-west traffic | Workload (T3) → workload/system | Packets | Cilium enforces policy at both endpoints. Each tenant pod's egress is confined to its own subtree, platform system services, and `world` (out-of-cluster); ingress is open to `world`+`cluster`. A flow needs the source's egress to allow it, so an unrelated tenant has no egress path to another tenant's pod. | Cross-tenant pod-to-pod traffic is **denied** (by source egress). Within a subtree, parent→descendant is allowed; child→parent is denied except for specific ancestor services (vminsert, etcd, ingress). Caveats (see Non-goals): open ingress and LoadBalancer hairpin. |
@@ -79,7 +79,7 @@ Actors are decomposed by **trust boundary**: two components are distinct actors
7979
- **Tenant isolation (API / identity layer).** A tenant cannot read, modify, or delete an *unrelated* tenant's Application CRs, workloads, or secrets, and a descendant cannot reach its ancestors. Parent→child access is intentional (ancestor ServiceAccounts and OIDC groups are bound one-way into descendant namespaces).
8080
- **Network isolation between unrelated tenants.** Unrelated tenants cannot reach each other over the pod network: each tenant pod's Cilium egress is confined to its own subtree, platform system services, and `world`, and Cilium requires the source's egress to allow a flow — so there is no egress path between unrelated subtrees. Enforced on egress; ingress is permissively open (see Non-goals for caveats).
8181
- **Constrained self-service provisioning.** Tenants may instantiate only vetted `ApplicationDefinition` kinds with server-fixed chart references; they cannot run arbitrary Helm charts or write raw privileged manifests.
82-
- **Hostname and tenancy integrity.** Tenants cannot claim or hijack a platform-apex hostname belonging to another tenant, on both dataplanes. Gateway API path: HTTPRoute/TLSRoute/Gateway-listener hostnames must be within the tenant's own apex. Default legacy-Ingress path (`cozystack-ingress-hostname-policy`): a hostname within the tenant's own apex or entirely outside the platform root apex (an external custom domain, e.g. a tenant Kubernetes cluster's Proxied ingress) is allowed, while a hostname under the platform root apex but outside the tenant's own sub-apex — plus hostless rules and `defaultBackend` catch-alls — is denied. Fail-closed (the Gateway-listener policy relies on the platform-set, immutable `namespace.cozystack.io/host` label).
82+
- **Hostname and tenancy integrity.** Tenants cannot claim or hijack a platform-apex hostname belonging to another tenant, on both dataplanes. Gateway API path: HTTPRoute/TLSRoute/GRPCRoute/Gateway-listener hostnames must be within the tenant's own apex. Default legacy-Ingress path (`cozystack-ingress-hostname-policy`): a hostname within the tenant's own apex or entirely outside the platform root apex (an external custom domain, e.g. a tenant Kubernetes cluster's Proxied ingress) is allowed, while a hostname under the platform root apex but outside the tenant's own sub-apex — plus hostless rules and `defaultBackend` catch-alls — is denied. Route, namespace and Ingress policies are fail-closed; the Gateway-listener policy is fail-open when the label is absent and relies on the platform-set, immutable `namespace.cozystack.io/host` label.
8383
- **Hierarchical resource-quota enforcement (best-effort).** At declaration time a child tenant's declared quota is checked against its parent's remaining budget; the check is a non-transactional admission snapshot (concurrent writes can briefly overshoot) with the tenant-quota controller as the runtime backstop.
8484
- **Strong isolation for tenant Kubernetes/VM workloads** via Talos worker nodes as KubeVirt VMs (hardware virtualization); the tenant control plane is a Kamaji hosted control plane running as pods on the management cluster, isolated by namespace/RBAC/network rather than by virtualization.
8585
- **Central, token-based authentication** when OIDC is enabled (opt-in; off by default): Keycloak/OIDC with group-to-role mapping into RBAC. The default OIDC-disabled mode fronts the dashboard with a token-proxy instead.

0 commit comments

Comments
 (0)