Skip to content

Commit 2bf9ae4

Browse files
lexfreilexfrei
committed
feat(gateway)!: serve passthrough hostnames and stop terminating them
Each tlsPassthroughListeners entry renders a mode=Passthrough listener on its own port, answering the entry's hostname by SNI and admitting TLSRoutes from the publishing tenant's namespace alone. It declares no allowedRoutes.kinds: at the pinned Cilium v1.19.5 a listener that spells its kinds out has them applied to every route on the Gateway rather than to itself, so naming TLSRoute here would reject every HTTPRoute on the Gateway. A hostname a passthrough listener answers no longer earns an HTTPS-terminate listener or a Gateway-issued certificate. That is the break, and it is keyed on service rather than on declaration: the same Cilium builds a passthrough filter chain per route out of the backends that survived resolution, so a listener no TLSRoute attaches to, and a route whose backendRefs resolve to nothing, both match no ClientHello and leave the terminate listener holding the name. Eligibility asks what v1.19.5 asks and no more, the reference allowed and the Service present, and fails closed on a read it cannot complete. Services and ReferenceGrants are watched, so a withdrawal deferred on a missing backend lands on the pass that sees it arrive, and the Service watch takes only the creates and deletes, existence being all that is read. Which hostname a route claims is answered the same way. spec.hostnames is optional on a TLSRoute, and ComputeHosts substitutes the hostname of each TLS listener the route's parentRefs select for a route that declares none, so such a route carries the SNI and takes the name over exactly as one that spells it out. A declared hostname broader than the listener's is that substitution from the other side: the route is served on the listener's name and not on the wildcard it wrote, so a claim covering a reserved name counts as a claim on it. The substitution reaches only the listeners that would admit the route. A native-port listener takes the publishing tenant's namespace alone, so Cilium never puts a route from elsewhere on it and never serves that route on its hostname; borrowing the name anyway refuses the route for a namespace, over a hostname its own object does not carry, in the same pass that sheds a terminate listener because that same route is serving a name it borrowed legitimately. A route that names the listener in its sectionName keeps the borrow, since there the refusal is the answer to what it asked for. A route the Gateway renders no listener for hears it from this controller instead of from Cilium alone: the hostname a passthrough listener took, the hostname nothing answers, the sectionName or port naming a listener that answers something else, the sectionName naming no rendered passthrough listener, and the native-port listener that admits the tenant alone, each with its own Gateway API reason and a message naming the hostnames that lost. An HTTPRoute pinning a rendered passthrough listener by sectionName is refused there too, since such a listener forwards a stream it never terminates and takes TLSRoute alone by protocol. Nothing else tells that route: the port-443 listeners name HTTPRoute in their kinds to keep every port-443 set uniform, which the Gateway reports back as InvalidRouteKinds rather than as a grant, and the pin reads a listener's kinds against every route on the Gateway, so it reports the route accepted. The field is confined to certMode http01, stated as the modes that can carry it so one added later is refused until someone rules on it. The wildcard modes cover the whole apex with one terminate listener the pinned Cilium cannot keep apart from a passthrough listener beneath it, and edge terminates TLS at the class provider and renders no TLS listener at all. An HTTPRoute claiming a wildcard is refused under http01 before the race and the overlap are read. The HTTP-01 challenge issues no wildcard certificate, so the terminate listener rendered for the claim could never carry the name, and the listener name built from the hostname's first label carried the asterisk, which the Gateway CRD admits in a hostname and refuses in a name, so the apiserver refused the whole Gateway write and every other listener on it froze where it stood. The renderer now writes that label as "wildcard", so every hostname the CRD admits yields a name it admits, and no name that was already admissible moves. That refusal is a statement about the terminate listener, so it is judged where that listener is decided rather than ahead of everything else. The port-80 listener carries no hostname and serves a wildcard claim as readily as any other, so a route that selects it, by pinning the port or by naming the listener, never needed the listener HTTP-01 cannot supply and is served rather than refused. Judged first, the refusal reached that route too and reported it unserved while Cilium was serving it in plaintext. A parentRef port on an HTTPRoute is read the way servableOn reads it for a TLSRoute. The terminate listener sits on 443, so a route pinning another port attaches to no listener the render would give it, and it used to earn a listener and an ACME certificate for a route that attached to nothing. Port 80 is the plain-HTTP listener, which admits the publishing tenant alone: a route from there pinning it is served plain and earns neither a listener nor a refusal. A route status write that fails is told apart by kind. A conflict or a route that vanished between the list and the write is a race between the two status writers this Gateway has: the reconcile requeues and retries the write without pinning Ready=False on the tenant, and the status pass runs after the http-to-https redirect, so such a race cannot withhold the one object that guard exists to provide. A write the apiserver will refuse on every pass still fails the reconcile, after every other route has been written, because the condition is the only object saying why a hostname is not served, and a pass that reported success over a lost write left the route reading Accepted=True until the resync. The retryable half is logged on the way out. Declining to pin Ready=False leaves the requeue as the only thing acting on it, and the requeue says nothing: the tenant goes on reading Ready, no event is recorded, and returning no error opts out of the backoff that would widen the interval, so a write failing on every pass retries at a fixed period indefinitely with nothing anywhere to show for it. A TLSRoute claiming a hostname broader than the listener's is pulled into the eligibility question for each name it covers, because the pinned Cilium serves it on the listener's own name. That is right only where the listener answering a name is named exactly that name. Under a wildcard entry the substitution hands such a route the wildcard, so its chain carries "*.db.<apex>" while the terminate chain carries the exact name beneath, and Envoy matches an exact server name ahead of a wildcard, so nothing collides; pulling the route in there withdrew a served endpoint and handed its name to the database behind the passthrough listener. The pull-in is now keyed on an exact listener match. Every condition this controller writes on an HTTPRoute, and the one it writes on a TLSRoute declaring no hostnames, is written under http01, and the whole-apex modes collect no claims, so a switch of certMode left those verdicts standing over listeners the new mode does not render. The whole-apex pass now drops this controller's entry from HTTPRoutes and from hostname-less TLSRoutes; the edge refusal stays confined to routes that declare hostnames, since a refusal written on a lent hostname would have no path back. Within http01 the same verdicts outlive their listeners for a second reason, and no mode change is needed to reach it. The status pass walks the routes that attached rather than the ones that claimed a hostname, and retracts its entry from the difference, because a claim set is not a route's own property once a hostname can be borrowed: a TLSRoute declaring none takes the hostname of every passthrough listener its parentRef selects, so removing that listener empties the claim set while the route stands unchanged, and an HTTPRoute reaches the same state by dropping its hostnames. Judged on claims alone such a route keeps whatever the last pass that did judge it left behind, describing a listener the Gateway no longer has. An HTTPRoute pinned by sectionName to a passthrough listener this Gateway does not render is refused with NoMatchingParent, the reading a TLSRoute in the same shape already got, instead of earning a terminate listener it could never attach to. A route served on the port-80 listener, by parentRef port or by naming it, earns no terminate listener and is refused nothing, but it publishes the name from its namespace all the same, so it stays in the hostname race. Every recount now ranks it with the field it recounts, because a recount that left it out crowned the namespace that had lost to it and dropped a loss the first count recorded: a third route refused in a third namespace, or a passthrough entry declared for the name, flipped the loser to Accepted=True. Selecting the port-80 listener by name is judged as selecting it by port: from a namespace the listener admits the route is served plain, from anywhere else the listener turns it away and the route reads NotAllowedByListeners, and a sectionName and a parentRef port that select different listeners select none and read NoMatchingParent, which is what Gateway API requires when both are given. Before this a route naming the listener from a refused namespace earned a terminate listener and a certificate it could never attach to. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 50e1e2b commit 2bf9ae4

13 files changed

Lines changed: 11857 additions & 2117 deletions

File tree

‎cmd/cozystack-controller/main.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ import (
5555
cosiv1alpha1 "sigs.k8s.io/container-object-storage-interface-api/apis/objectstorage/v1alpha1"
5656
gatewayv1 "sigs.k8s.io/gateway-api/apis/v1"
5757
gatewayv1alpha2 "sigs.k8s.io/gateway-api/apis/v1alpha2"
58+
gatewayv1beta1 "sigs.k8s.io/gateway-api/apis/v1beta1"
5859
// +kubebuilder:scaffold:imports
5960
)
6061

@@ -71,6 +72,7 @@ func init() {
7172
utilruntime.Must(internalv1alpha1.AddToScheme(scheme))
7273
utilruntime.Must(gatewayv1.Install(scheme))
7374
utilruntime.Must(gatewayv1alpha2.Install(scheme))
75+
utilruntime.Must(gatewayv1beta1.Install(scheme))
7476
utilruntime.Must(cmv1.AddToScheme(scheme))
7577
utilruntime.Must(helmv2.AddToScheme(scheme))
7678
utilruntime.Must(cosiv1alpha1.AddToScheme(scheme))

‎internal/controller/tenantgateway/backends.go‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
/*
2+
Copyright 2026 The Cozystack Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package tenantgateway
18+
19+
import (
20+
"context"
21+
"fmt"
22+
23+
corev1 "k8s.io/api/core/v1"
24+
apierrors "k8s.io/apimachinery/pkg/api/errors"
25+
"k8s.io/apimachinery/pkg/types"
26+
gatewayv1 "sigs.k8s.io/gateway-api/apis/v1"
27+
gatewayv1alpha2 "sigs.k8s.io/gateway-api/apis/v1alpha2"
28+
gatewayv1beta1 "sigs.k8s.io/gateway-api/apis/v1beta1"
29+
)
30+
31+
// tlsRouteForwards reports whether route names at least one backend the
32+
// pinned Cilium would forward to, which is what decides whether the
33+
// route puts a filter chain on its hostname.
34+
//
35+
// The question is answered exactly as v1.19.5 answers it and no more
36+
// precisely. toTLSRoutes (operator/pkg/model/ingestion/gateway.go:587-616)
37+
// keeps a backendRef when IsBackendReferenceAllowed passes and
38+
// getServiceSpec finds the Service, then tlsPassthroughFilterChains
39+
// (operator/pkg/model/translation/envoy_listener.go:408-411) skips a
40+
// route whose surviving list is empty. Neither step reads the Service's
41+
// ports or its endpoints, so neither does this: a backendRef naming a
42+
// port the Service does not publish still yields a chain that carries
43+
// the SNI and forwards to a closed port, which is a broken backend
44+
// rather than an unclaimed hostname.
45+
//
46+
// Any rule with a surviving backend is enough. Cilium appends one model
47+
// route per rule, so a route whose second rule resolves carries the SNI
48+
// however little the first one does.
49+
//
50+
// A ServiceImport backend counts as unresolvable. Cilium resolves those
51+
// through mcs-api, whose CRDs this platform does not install, so the
52+
// conservative answer is also the accurate one here — and it errs
53+
// towards keeping a terminate listener rather than shedding one.
54+
func (r *Reconciler) tlsRouteForwards(ctx context.Context, route *gatewayv1alpha2.TLSRoute, grants []gatewayv1beta1.ReferenceGrant) (bool, error) {
55+
for _, rule := range route.Spec.Rules {
56+
for _, be := range rule.BackendRefs {
57+
if !backendRefIsService(be.BackendObjectReference) {
58+
continue
59+
}
60+
ns := backendRefNamespace(be.BackendObjectReference, route.Namespace)
61+
if ns != route.Namespace && !serviceReferenceGranted(grants, route.Namespace, ns, string(be.Name)) {
62+
continue
63+
}
64+
err := r.Get(ctx, types.NamespacedName{Namespace: ns, Name: string(be.Name)}, &corev1.Service{})
65+
if apierrors.IsNotFound(err) {
66+
continue
67+
}
68+
// A read that failed for any other reason says nothing
69+
// about whether the Service is there, and a withdrawal
70+
// decided on it would shed a listener on the strength of an
71+
// unavailable apiserver. Fail the reconcile instead and let
72+
// the requeue decide once the answer is knowable.
73+
if err != nil {
74+
return false, fmt.Errorf("get backend Service %s/%s: %w", ns, be.Name, err)
75+
}
76+
return true, nil
77+
}
78+
}
79+
return false, nil
80+
}
81+
82+
// backendRefIsService mirrors Cilium's helpers.IsService: an absent
83+
// kind or group means Service in the core group, so the common
84+
// backendRef that names neither is one.
85+
func backendRefIsService(ref gatewayv1.BackendObjectReference) bool {
86+
return (ref.Kind == nil || *ref.Kind == "Service") &&
87+
(ref.Group == nil || *ref.Group == corev1.GroupName)
88+
}
89+
90+
// backendRefNamespace mirrors Cilium's helpers.NamespaceDerefOr, which
91+
// treats an empty namespace the same as an absent one. Getting that
92+
// wrong would read a ref written with `namespace: ""` as
93+
// cross-namespace and demand a grant for the route's own namespace.
94+
func backendRefNamespace(ref gatewayv1.BackendObjectReference, routeNamespace string) string {
95+
if ref.Namespace != nil && *ref.Namespace != "" {
96+
return string(*ref.Namespace)
97+
}
98+
return routeNamespace
99+
}
100+
101+
// serviceReferenceGranted answers the cross-namespace half of
102+
// Cilium's helpers.isReferenceAllowed for the one direction that
103+
// matters here: a TLSRoute reaching a Service in another namespace.
104+
//
105+
// The grant lives in the namespace being referenced, names the
106+
// referring kind and namespace in from, and the referenced kind in to.
107+
// An unnamed to entry admits every Service in the grant's namespace,
108+
// a named one only that Service. Same-namespace references never reach
109+
// this function: they need no grant and the caller decides them first.
110+
func serviceReferenceGranted(grants []gatewayv1beta1.ReferenceGrant, routeNamespace, backendNamespace, backendName string) bool {
111+
for _, grant := range grants {
112+
if grant.Namespace != backendNamespace {
113+
continue
114+
}
115+
for _, from := range grant.Spec.From {
116+
if string(from.Group) != gatewayv1.GroupName || from.Kind != "TLSRoute" || string(from.Namespace) != routeNamespace {
117+
continue
118+
}
119+
for _, to := range grant.Spec.To {
120+
if string(to.Group) != corev1.GroupName || to.Kind != "Service" {
121+
continue
122+
}
123+
if to.Name == nil || string(*to.Name) == backendName {
124+
return true
125+
}
126+
}
127+
}
128+
}
129+
return false
130+
}

‎internal/controller/tenantgateway/backends_test.go‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
/*
2+
Copyright 2026 The Cozystack Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package tenantgateway
18+
19+
import (
20+
"testing"
21+
22+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
23+
gatewayv1 "sigs.k8s.io/gateway-api/apis/v1"
24+
gatewayv1beta1 "sigs.k8s.io/gateway-api/apis/v1beta1"
25+
)
26+
27+
// TestServiceReferenceGranted walks each field a ReferenceGrant is
28+
// matched on. A grant admitting more than it should widens the
29+
// withdrawal to routes Cilium will not serve, which is the defect this
30+
// resolution exists to close; one admitting less strands a route that
31+
// does forward. Both directions need a fixture, so every leg gets a
32+
// case that differs from the admitting one in that field alone.
33+
func TestServiceReferenceGranted(t *testing.T) {
34+
named := gatewayv1beta1.ObjectName("api-backend")
35+
other := gatewayv1beta1.ObjectName("something-else")
36+
grant := func(ns string, from gatewayv1beta1.ReferenceGrantFrom, to gatewayv1beta1.ReferenceGrantTo) gatewayv1beta1.ReferenceGrant {
37+
return gatewayv1beta1.ReferenceGrant{
38+
ObjectMeta: metav1.ObjectMeta{Name: "grant", Namespace: ns},
39+
Spec: gatewayv1beta1.ReferenceGrantSpec{
40+
From: []gatewayv1beta1.ReferenceGrantFrom{from},
41+
To: []gatewayv1beta1.ReferenceGrantTo{to},
42+
},
43+
}
44+
}
45+
fromTLSRoute := gatewayv1beta1.ReferenceGrantFrom{
46+
Group: gatewayv1.GroupName,
47+
Kind: "TLSRoute",
48+
Namespace: "tenant-foo",
49+
}
50+
toAnyService := gatewayv1beta1.ReferenceGrantTo{Group: "", Kind: "Service"}
51+
52+
cases := []struct {
53+
name string
54+
grants []gatewayv1beta1.ReferenceGrant
55+
want bool
56+
}{
57+
{
58+
name: "no grants at all",
59+
grants: nil,
60+
want: false,
61+
},
62+
{
63+
name: "unnamed to entry admits every Service",
64+
grants: []gatewayv1beta1.ReferenceGrant{grant("cozy-public", fromTLSRoute, toAnyService)},
65+
want: true,
66+
},
67+
{
68+
name: "named to entry admits that Service",
69+
grants: []gatewayv1beta1.ReferenceGrant{grant("cozy-public", fromTLSRoute, gatewayv1beta1.ReferenceGrantTo{Group: "", Kind: "Service", Name: &named})},
70+
want: true,
71+
},
72+
{
73+
name: "named to entry admits no other Service",
74+
grants: []gatewayv1beta1.ReferenceGrant{grant("cozy-public", fromTLSRoute, gatewayv1beta1.ReferenceGrantTo{Group: "", Kind: "Service", Name: &other})},
75+
want: false,
76+
},
77+
{
78+
name: "grant in the wrong namespace",
79+
grants: []gatewayv1beta1.ReferenceGrant{grant("cozy-elsewhere", fromTLSRoute, toAnyService)},
80+
want: false,
81+
},
82+
{
83+
name: "from names another kind",
84+
grants: []gatewayv1beta1.ReferenceGrant{grant("cozy-public",
85+
gatewayv1beta1.ReferenceGrantFrom{Group: gatewayv1.GroupName, Kind: "HTTPRoute", Namespace: "tenant-foo"},
86+
toAnyService)},
87+
want: false,
88+
},
89+
{
90+
name: "from names another namespace",
91+
grants: []gatewayv1beta1.ReferenceGrant{grant("cozy-public",
92+
gatewayv1beta1.ReferenceGrantFrom{Group: gatewayv1.GroupName, Kind: "TLSRoute", Namespace: "tenant-bar"},
93+
toAnyService)},
94+
want: false,
95+
},
96+
{
97+
name: "to names another kind",
98+
grants: []gatewayv1beta1.ReferenceGrant{grant("cozy-public", fromTLSRoute, gatewayv1beta1.ReferenceGrantTo{Group: "", Kind: "Secret"})},
99+
want: false,
100+
},
101+
{
102+
name: "to names another group",
103+
grants: []gatewayv1beta1.ReferenceGrant{grant("cozy-public", fromTLSRoute, gatewayv1beta1.ReferenceGrantTo{Group: "multicluster.x-k8s.io", Kind: "Service"})},
104+
want: false,
105+
},
106+
{
107+
name: "one grant among several admits",
108+
grants: []gatewayv1beta1.ReferenceGrant{
109+
grant("cozy-elsewhere", fromTLSRoute, toAnyService),
110+
grant("cozy-public", fromTLSRoute, toAnyService),
111+
},
112+
want: true,
113+
},
114+
}
115+
for _, tc := range cases {
116+
t.Run(tc.name, func(t *testing.T) {
117+
got := serviceReferenceGranted(tc.grants, "tenant-foo", "cozy-public", "api-backend")
118+
if got != tc.want {
119+
t.Errorf("serviceReferenceGranted = %v, want %v", got, tc.want)
120+
}
121+
})
122+
}
123+
}

0 commit comments

Comments
 (0)