Repository navigation
[Epic] Move every Ingress to Gateway API: Gateway, ListenerSet, routes #4788
Description
Summary
Cozystack still creates 21 Ingress objects. 14 come from our own templates and 7 from vendored charts and operators that our values switch on. The Gateway API path from #2470 covers nine of them with a parallel route, but it is off by default (gateway.enabled: false), no CI install runs it, and several of those routes drop behaviour the matching Ingress has. The goal of this epic is Gateway API as the only exposure path, with ingress-nginx deprecated. It replaces #2490.
Blocked by #3343, the Cilium 1.20 bump. Cilium 1.19 implements neither ListenerSet nor TLSRoute v1, and the target model below needs both. Items marked "can start now" don't wait for the bump.
Target model
The Gateway stays owned by the TenantGateway controller, one per publishing tenant, as today. It keeps the port-80 listener, the apex and wildcard listeners and the platform passthrough listeners. It gets spec.allowedListeners for the namespaces labelled namespace.cozystack.io/gateway=<owner>.
Every per-application hostname moves to a ListenerSet. The application chart renders it in its own namespace, with an HTTPS (terminate) or TLS (passthrough) listener and the Certificate for it. Certificate references resolve in the ListenerSet's namespace, so there is no ReferenceGrant and no Secret copy. This removes the 64-listener cap on the owner Gateway and lets child tenants share the owner's address. The per-hostname listener synthesis in http01 mode goes away.
Listeners on the Gateway itself are left only for what the platform owns: apex, wildcard (dns01 or existingSecret), port-80 redirect and ACME, native-port passthrough.
Routes stay standard. HTTPRoute for HTTP, TLSRoute v1 for passthrough, with parentRefs pointing at the app's ListenerSet (kind: ListenerSet) or at a Gateway sectionName. Re-encryption to an HTTPS backend goes through BackendTLSPolicy.
Charts pick Ingress or Gateway API through one cozy-lib helper fed by _namespace.gateway. Today some charts check _cluster.gateway-enabled and others _namespace.gateway.
Inventory
Inventory taken on main at 9620be010. In the "Switch" column, cluster is _cluster.gateway-enabled, tenant is _namespace.gateway, and none means the Ingress renders in both modes.
Platform services (publishing tenant)
| Surface | Ingress today | Switch | Gateway API today | Target | Gap |
|---|---|---|---|---|---|
| dashboard | system/dashboard/templates/ingress.yaml, oauth2-proxy backend |
cluster | HTTPRoute, plain backendRef |
Gateway listener + HTTPRoute | /openapi-ui redirect (server-snippet) → RequestRedirect filter; 86400s timeouts → rules[].timeouts; 100m body and buffer sizes, check Envoy defaults |
| keycloak (public, admin) | system/keycloak/templates/ingress.yaml |
cluster | two HTTPRoutes |
Gateway listener + HTTPRoute | cookie session affinity has no standard field; cert-manager annotations disappear when ingress.annotations is emptied |
| cozystack-api | system/cozystack-api/templates/api-ingress.yaml, passthrough |
cluster | TLSRoute v1alpha2 |
TLSRoute v1 |
v1alpha2 → v1 (#3873); layout from #4638 |
| vm-exportproxy | system/kubevirt/templates/vm-exportproxy-ingress.yaml, passthrough |
cluster | TLSRoute v1alpha2 |
layout from #4638 | same as above |
| cdi-uploadproxy | system/kubevirt-cdi/templates/cdi-uploadproxy-ingress.yaml, passthrough |
cluster | TLSRoute v1alpha2 |
layout from #4638 | same as above; CDI derives the upload URL from Ingress, so uploadProxyURLOverride has to be set |
| linstor-gui | system/linstor-gui/templates/ingress.yaml, oauth2-proxy backend |
none | none | Gateway listener + HTTPRoute | not ported; attachedNamespaces names the wrong namespace (#4105) |
Tenant applications
| Surface | Ingress today | Switch | Gateway API today | Target | Gap |
|---|---|---|---|---|---|
| harbor (UI, registry) | apps/harbor/templates/ingress.yaml |
tenant | HTTPRoute, fails when managementWhitelist is set |
ListenerSet + HTTPRoute | 900s timeouts; source-IP allowlist |
| bucket UI | system/bucket/templates/ingress.yaml |
tenant | HTTPRoute |
ListenerSet + HTTPRoute | 99999s timeouts |
| grafana | Grafana CR spec.ingress, created by grafana-operator |
cluster | HTTPRoute on the publishing tenant's Gateway |
ListenerSet + HTTPRoute | wrong switch: a tenant with its own Gateway gets a route its Gateway never admits |
| alerta | system/monitoring/templates/alerta/alerta.yaml |
cluster | HTTPRoute on the publishing tenant's Gateway |
ListenerSet + HTTPRoute | same as grafana |
| bootbox (matchbox) | extra/bootbox/templates/matchbox/ingress.yaml |
none | none | ListenerSet + HTTPRoute | source-IP allowlist (whitelistHTTP, deny-all by default) |
| seaweedfs S3 | vendored s3-ingress.yaml, enabled by s3.ingress.enabled |
none | none | ListenerSet + HTTPRoute + BackendTLSPolicy | HTTPS backend; 5g body, buffering off, 3600s timeouts; host tied to tenant-root (#2399) |
| seaweedfs filer gRPC | extra/seaweedfs/templates/ingress.yaml, GRPCS passthrough |
none | none | ListenerSet (TLS passthrough) + TLSRoute | ingressClassName hardcoded to tenant-root; source-IP allowlist |
| kubernetes, Proxied ingress-nginx hosts | apps/kubernetes/templates/ingress.yaml, passthrough and HTTP on one host |
none | none | ListenerSet with a passthrough and an HTTP listener + TLSRoute + HTTPRoute | AUTO_HTTP and the configuration-snippet workaround become two routes |
| kubernetes apiserver (Kamaji) | KamajiControlPlane.spec.network.ingress, created by Kamaji |
none | none | Kamaji spec.network.gateway (TLSRoute) |
gateway block not wired or tested; ComputePlane inherits it |
| computeplane | wraps the kubernetes app and forces Proxied ingress-nginx | none | none | follows the two rows above |
Created by tooling
| Surface | Today | Target |
|---|---|---|
| cert-manager HTTP-01 solver | http01.ingress solver on the ClusterIssuer, per-Ingress class annotation |
gatewayHTTPRoute solver (exists); host cert-manager has to run with Gateway API enabled (#3725) |
| external-dns | source: ingress |
gateway-httproute, gateway-tlsroute (already supported when gatewayAPI is set) |
| dashboard resource maps | list ingresses only |
add httproutes, tlsroutes, listenersets |
tenant ingress-nginx (extra/ingress, ingress-rd) |
per-tenant controller | deprecated once the default flips, removed in a later release |
nginx behaviour to replace
| nginx feature | Used by | Gateway API replacement |
|---|---|---|
ssl-passthrough |
api, vm-exportproxy, cdi-uploadproxy, seaweedfs filer, kubernetes Proxied, Kamaji | TLS listener mode: Passthrough + TLSRoute |
backend-protocol: HTTPS |
seaweedfs S3 | BackendTLSPolicy |
backend-protocol: GRPCS with passthrough |
seaweedfs filer | TLSRoute (nothing terminates, so no GRPCRoute) |
proxy-read/send-timeout |
dashboard, harbor, bucket, seaweedfs S3, linstor-gui | HTTPRoute.rules[].timeouts |
server-snippet redirect |
dashboard | RequestRedirect filter |
rewrite-target |
dashboard | check if it is needed at all; URLRewrite filter |
configuration-snippet |
kubernetes Proxied | dropped, HTTP and passthrough become separate routes |
whitelist-source-range |
harbor, bootbox, seaweedfs filer | no standard field, see open questions |
| cookie session affinity | keycloak | sessionPersistence is experimental, see open questions |
| body size, buffer sizes, buffering off | dashboard, linstor-gui, harbor, bucket, seaweedfs S3 | check Envoy defaults in the shipped Cilium; implementation config if they are not enough |
per-Ingress http01-ingress-ingressclassname |
every Ingress with cert-manager annotations | solver parentRef on the ClusterIssuer or the per-tenant Issuer |
Phases
Phase 0: prerequisites
- Cilium with ListenerSet and TLSRoute v1 (fix(cilium): bump vendored chart to v1.20.2 #3343)
-
sigs.k8s.io/gateway-apiGo module up to the vendored CRD version (v1.6.1), and the three platformTLSRoutes on v1 (gateway: three charts render TLSRoute at the deprecated v1alpha2 #3873) - Host cert-manager with Gateway API enabled (cert-manager is started without
--enable-gateway-api=truewhen gateway is enabled on platform and tenant level #3725), can start now - Hostname admission policy covers
listenersets(gateway: ListenerSet hostnames escape the hostname admission policy #4066), can start now - cozy-lib helper that picks Ingress or Gateway API from
_namespace.gateway, plus a parentRef helper, instead of the per-chart copies; can start now - Decide who issues the Certificate for a ListenerSet listener: the app chart, or cert-manager's gateway-shim (check ListenerSet support in the shipped cert-manager)
Phase 1: TenantGateway with ListenerSet
-
allowedListenerson the owner Gateway, selected by the same namespace label as routes - http01 mode: per-hostname listeners move from the owner Gateway into app ListenerSets; route status and conflict handling read ListenerSet status
- Child tenants without their own Gateway share the owner's address through ListenerSet; recheck gateway: a second tenant Gateway steals the platform namespaces from the first #3869 after that
- Knobs the Gateway path ignores today:
externalIPs,loadBalancerClass,proxyProtocol(Gateway API (Cilium): no supported knob to enable proxyProtocol on the publishing path #3030); hairpin for in-cluster clients (No hairpin path for hostnames published through the Gateway API: in-cluster clients cannot reach their own published URL #3821)
Phase 2: fix the routes that already exist (can start now)
- dashboard: redirect filter, timeouts, body size check
- keycloak: session affinity decision; cert-manager annotations moved out of
ingress.annotations - harbor, bucket: timeouts; harbor allowlist decision
- grafana, alerta: switch to
_namespace.gatewayand attach to the tenant's own Gateway
Phase 3: port the Ingress-only surfaces
- linstor-gui
- bootbox
- seaweedfs S3 (BackendTLSPolicy) and filer gRPC (passthrough); drop the hardcoded
tenant-rootclass ([seaweedfs] S3 ingress hardcoded to tenant-root instead of using tenant's ingress #2399) - kubernetes Proxied ingress-nginx hosts
- Kamaji apiserver through
spec.network.gateway; ComputePlane follows - Every port comes with helm-unittest coverage for both modes, like the existing
*_wildcard_test.yamlandhost_guard_httproute_test.yaml
Phase 4: everything around the routes
- external-dns sources follow the mode
- Dashboard resource maps list routes and ListenerSets
- KubeVirt and CDI external URLs set explicitly in gateway mode
- Docs:
packages/extra/gateway/README.md,docs/gateway-classes.md, website
Phase 5: flip the default
- An e2e install with
gateway.enabled: truethat issues a real certificate and serves one platform service and one tenant app through the Gateway - Upgrade path for running clusters: what happens to the ingress-nginx LB IP, DNS, certificate re-issue, a migration that sets
gateway: trueon the root tenant -
gateway.enableddefaults totrue; ingress-nginx stays installable - Deprecate the tenant
ingressmodule and the Ingress templates, remove them in a later minor release
Open questions
- Source-IP allowlists (harbor, bootbox, seaweedfs filer). Options: Cilium network policy on the Gateway's Envoy, an implementation-specific filter, or no allowlist on the Gateway path, which is what harbor does today.
- Keycloak session affinity. Experimental
sessionPersistenceif the shipped Cilium implements it, otherwise drop affinity and rely on Keycloak's distributed sessions. - Certificate ownership for ListenerSet listeners: a
Certificaterendered by the chart, or cert-manager annotations on the ListenerSet. - Address on upgrade. Either the Gateway takes over the ingress-nginx IP, or users get a new address and change DNS.
Related
#2470, #2490 (superseded), #2811, #2815, #3913, #3912, #4638, #3087
Reacted by dosubot
Activity