Refactor Gateway API reconciler by youngnick · Pull Request #41232 · cilium/cilium

youngnick · 2025-08-18T05:10:12Z

This PR removes the separate reconcilers for each Route type in favor of updating status inside the Gateway
reconciler.

This also adds new indicies on the Gateway reconciler that should make reconciliation more efficient.

Migrates all tests that were handled by the various Route reconcilers to the Gateway reconciler.

Moves all indexers used in the Gateway controller-runtime Manager into a new indexers package.

It also fixes an existing issue with GRPCRoute reconciliation (#39021 and #40922), by adding correct supportedKinds handling.

While fixing this, also fixed some inconsistencies in Condition handling across all Gateway API objects, so this is a much bigger commit than expected.

Programmed is now added as Pending before addresses are added to the Gateway, and is updated to Programmed when addresses are added.

Tests have been updated to reflect this.

Also added a test that, when there is an address set on the Gateway, the Programmed Condition is correctly updated.

The address is set by taking it from the generated Loadbalancer Service, so this test supplies a preconfigured Loadbalancer Service to ensure an address gets added to the Gateway correctly.

Ordering of Conditions in status has also been made more consistent.

The PEM checking performed in isValidPEMFormat has been loosened, and this function will now return true for all byte slices that contain at least one valid PEM-encoded certificate or key. Previous behavior was that it would only return if all bytes decoded successfully, which can cause issues if there are extra line feeds or extraneous material in the certificate - that don't matter to Envoy or to the eventual functionality.

Removes the now-redundant ReferenceGrant reconciler, which was a placeholder in case we needed to update ReferenceGrant status at a later date.

If we do need to do that in the future, which is unlikely, we can use similar mechanisms to what we now use for Route status in the Gateway reconciler.

Fixes: #40809

Cilium's Gateway API reconciler has been completely refactored and should be more reliable and performant as a result.

gandro

I lack domain knowledge in the service mesh department, so I'll rely on those CODEOWNERs to judge if the new implementation is sound (ultimately I only got pulled in because of the logfield changes).

I did not spot anything majorly off from a rough glance. A couple of commented out pieces of code that stood out to me, but nothing blocking.

operator/pkg/gateway-api/cell.go

operator/pkg/gateway-api/helpers.go

operator/pkg/gateway-api/gateway_status.go

youngnick · 2025-08-19T07:53:20Z

/test

mhofstetter

I have a question regarding reuse of a (HTTP)Route between Gateway API (Gateway) & GAMMA (K8s Service) - see my inline comment.

I think it would be good to clarify this first as this might when thinking about moving the status update into the Gateway/GAMMA reconcilers.

operator/pkg/gateway-api/gateway.go

operator/pkg/gateway-api/gateway_reconcile.go

operator/pkg/gateway-api/gateway.go

operator/pkg/gateway-api/gamma_reconcile.go

youngnick · 2025-08-20T03:45:21Z

/test

mhofstetter

LGTM - even though it's quiet hard to properly review a PR this size that also includes formatting changes 😅 Let's put some trust into the tests (incl. upstream Conformance tests 🙏 )

The rest are mostly nits.

Approving because i will be on PTO starting tomorrow. Maybe it's worth to let @sayboras quickly glance over it as well once he's back?

Thanks for the effort you put into this! 🙇

operator/pkg/gateway-api/gateway.go

operator/pkg/gateway-api/gamma_reconcile.go

operator/pkg/gateway-api/gateway.go

operator/pkg/gateway-api/gateway_reconcile.go

jfmatth · 2025-08-28T02:08:42Z

@youngnick How do I go about testing / installing this? I don't see an updated image. Sorry for the ignorance.

youngnick · 2025-08-29T04:36:15Z

@jfmatth, you can test using the CI images, with the images and tags available at https://github.com/cilium/cilium/actions/runs/17229742144/job/48881742368?pr=41232

Then, you'll need to specify the images like this (if you're using cilium install):

          --helm-set=image.repository=quay.io/cilium/cilium-ci \
          --helm-set=image.useDigest=false \
          --helm-set=image.tag=4c81d839d95582368967185aa92f4604cf8fcbd6 \
          --helm-set=operator.image.repository=quay.io/cilium/operator \
          --helm-set=operator.image.suffix=-ci \
          --helm-set=operator.image.tag=4c81d839d95582368967185aa92f4604cf8fcbd6 \
          --helm-set=operator.image.useDigest=false \

If you're using Helm natively:

          --set=image.repository=quay.io/cilium/cilium-ci \
          --set=image.useDigest=false \
          --set=image.tag=4c81d839d95582368967185aa92f4604cf8fcbd6 \
          --set=operator.image.repository=quay.io/cilium/operator \
          --set=operator.image.suffix=-ci \
          --set=operator.image.tag=4c81d839d95582368967185aa92f4604cf8fcbd6 \
          --set=operator.image.useDigest=false \

I'm pretty confident this will fix that problem but if you could verify and put a response on #34982, then I'd really appreciate it!

jfmatth · 2025-08-30T20:17:12Z

@youngnick I'm not having any luck getting the gateway to get an assigned IP once I run my cilium-gateway.yaml file?

I have a branch I'm trying all this in the Talos README.md you'll find everything

https://github.com/jfmatth/home-charts/tree/enh/cilium-fix-gateway

I tried the same build w/o specifying your images and it works, i.e. gateway is 'programmed' otherwise it never finds an endpoint.

Any help would be appreciated.

youngnick · 2025-09-01T04:22:23Z

One of the issues I fixed in this PR is that the Gateway could show as reconciled before an address was assigned - when you are seeing the working Gateway with the previous version, does it have an address correcctly assigned? It's possible you may need at least two IP addresses available, since there's a shared Cilium LB Service deployed by default as well (I think).

jfmatth · 2025-09-01T12:02:54Z

Nick, thank you. Very interesting. The behavior would match (mostly) that of what the Traefik GAPI does. I thought I assigned one HTTP address, but will check again when I can. J

…

On Sun, Aug 31, 2025 at 11:22 PM Nick Young ***@***.***> wrote: *youngnick* left a comment (cilium/cilium#41232) <#41232 (comment)> One of the issues I fixed in this PR is that the Gateway could show as reconciled before an address was assigned - when you are seeing the working Gateway with the previous version, does it have an address correcctly assigned? It's possible you may need at least two IP addresses available, since there's a shared Cilium LB Service deployed by default as well (I think). — Reply to this email directly, view it on GitHub <#41232 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACGTMUE4W5Z6WHZDKC3ART3QPCZNAVCNFSM6AAAAACED7CYGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTENBQHAYTKMRRGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

jfmatth · 2025-09-02T01:22:58Z

So doing more testing locally and I now see the LoadBalancer which I don't recall before, cool.

I applied my gateway file

apiVersion: gateway.networking.k8s.io/v1

kind: Gateway

metadata:
  name: gateway
  namespace: kube-system
  annotations:
    cert-manager.io/cluster-issuer: clusterissuer

spec:
  gatewayClassName: cilium

  listeners:
    - name: http
      protocol: HTTP
      port: 80

      allowedRoutes:
        namespaces:
          from: All
        kinds:
          - kind: HTTPRoute

Still no IP address, but applied an IP pool and I now see the LB has one, but the gateway is still PROGRAMMED = False

PS C:\Users\john\development\home-charts\talos> k get svc -A
NAMESPACE                       NAME                            TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)                  AGE
default                         kubernetes                      ClusterIP      10.96.0.1        <none>            443/TCP                  7m45s
kube-system                     cilium-envoy                    ClusterIP      None             <none>            9964/TCP                 7m
kube-system                     cilium-gateway-gateway          LoadBalancer   10.98.177.169    192.168.100.140   80:30324/TCP             4m11s
kube-system                     hubble-peer                     ClusterIP      10.111.86.44     <none>            443/TCP                  7m
kube-system                     kube-dns                        ClusterIP      10.96.0.10       <none>            53/UDP,53/TCP,9153/TCP   7m22s
kube-system                     metrics-server                  ClusterIP      10.110.188.188   <none>            443/TCP                  7m18s
kubelet-serving-cert-approver   kubelet-serving-cert-approver   ClusterIP      10.106.218.115   <none>            9090/TCP                 7m15s
PS C:\Users\john\development\home-charts\talos> k get gateway -A
NAMESPACE     NAME      CLASS    ADDRESS   PROGRAMMED   AGE
kube-system   gateway   cilium             False        4m15s

I have not httproutes.

jfmatth · 2025-09-02T01:29:01Z

I added an HTTProute but still no IP on the gateway. I suspect it's all me, but wanted to ask.

here is gateway describe

PS C:\Users\john\development\home-charts\whoami> k describe  gateway -A
Name:         gateway
Namespace:    kube-system
Labels:       <none>
Annotations:  cert-manager.io/cluster-issuer: clusterissuer
API Version:  gateway.networking.k8s.io/v1
Kind:         Gateway
Metadata:
  Creation Timestamp:  2025-09-02T01:20:15Z
  Generation:          1
  Resource Version:    1742
  UID:                 9e8a5412-8fbd-44a1-910e-cad39b17baec
Spec:
  Gateway Class Name:  cilium
  Listeners:
    Allowed Routes:
      Kinds:
        Group:  gateway.networking.k8s.io
        Kind:   HTTPRoute
      Namespaces:
        From:  All
    Name:      http
    Port:      80
    Protocol:  HTTP
Status:
  Conditions:
    Last Transition Time:  2025-09-02T01:26:14Z
    Message:               Unable to ensure Endpoints resource
    Observed Generation:   1
    Reason:                NoResources
    Status:                False
    Type:                  Accepted
    Last Transition Time:  2025-09-02T01:20:15Z
    Message:               Unable to create Endpoints resource
    Observed Generation:   1
    Reason:                NoResources
    Status:                False
    Type:                  Programmed
  Listeners:
    Attached Routes:  1
    Conditions:
      Last Transition Time:  2025-09-02T01:26:14Z
      Message:               Resolved Refs
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
      Last Transition Time:  2025-09-02T01:26:14Z
      Message:               Listener Accepted
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2025-09-02T01:26:14Z
      Message:               Address not ready yet
      Observed Generation:   1
      Reason:                Pending
      Status:                False
      Type:                  Programmed
    Name:                    http
    Supported Kinds:
      Group:  gateway.networking.k8s.io
      Kind:   HTTPRoute
Events:       <none>

youngnick · 2025-09-02T07:07:39Z

Yes, you can see from there, the Gateway is not being processed correctly because the Cilium operator cannot create or update ("ensure") the Endpoints resources.

jfmatth · 2025-09-02T11:57:26Z

Thanks for the quick reply @youngnick

I didn't show it, but I defined 2 IP's in the pool, I could see one get assigned to the LB and I added an HTTPRoute. Still I didn't see the G/W show programmed, what am I not doing?

The IPs were valid.

youngnick · 2025-09-03T00:50:57Z

tbh I'm not sure sorry @jfmatth. Could you open a fresh issue with this, preferably with a sysdump if you can?

In the meantime, It seems weird that the operator cannot create Endpoints for the generated service, is it missing some RBAC? If you check the operattor logs for messages about that Gateway, are there any errors shown? (You should be able to grep the operator logs for the Gateway name)

jfmatth · 2025-09-04T13:06:18Z

@youngnick I'll do more testing when i'm back in the office, on the road currently :)

One question I did have is: Do the IPPool IP's need to be on the hosts? I think they do? My test rig on the road didn't have that, so that might be why it's not assigning to the GW87

jfmatth · 2025-09-07T16:19:37Z

tbh I'm not sure sorry @jfmatth. Could you open a fresh issue with this, preferably with a sysdump if you can?

In the meantime, It seems weird that the operator cannot create Endpoints for the generated service, is it missing some RBAC? If you check the operattor logs for messages about that Gateway, are there any errors shown? (You should be able to grep the operator logs for the Gateway name)

Not to belabor this @youngnick but it does seem to be an RBAC issue. After spending hours finding the permission issue in the logs then trying to fix those ones, after about 4 clusterrole changes, I resorted to just giving cilium-operator clusteradmin and it worked! Obviously, this isn't a solution, but had to show you

I haven't opened a new issue since this might be related to your update. I'm still happy to and reference everything you and I have been discussing.

One last thing to mention, this is a Talos 1.10.6 cluster, in case that wasn't clear.

After apply the adminrole...

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cilium-operator-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: cilium-operator
    namespace: kube-system

And now the gateway is programmed w/o any routes, but a valid gateway

PS C:\Users\john\development\home-charts\talos> kubectl apply -f cilium-gateway.yaml
gateway.gateway.networking.k8s.io/gateway created
PS C:\Users\john\development\home-charts\talos> k get gateway -A
NAMESPACE     NAME      CLASS    ADDRESS           PROGRAMMED   AGE
kube-system   gateway   cilium   192.168.100.140   True         9s

Here is my cilium-rbac.yaml file to show you the 'along the way' changes.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cilium-operator
rules:
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cilium-operator-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cilium-operator
subjects:
  - kind: ServiceAccount
    name: cilium-operator
    namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cilium-operator
rules:
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "watch", "list", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cilium-operator-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: cilium-operator
    namespace: kube-system

This commit adds support for using GRPCRoutes for east/west traffic routing with GAMMA. It includes a small amount of refactoring of the Gateway API reconciler as well, to enable the HTTPRoute and GRPCRoute translation logic to share code across both north/south and east/west traffic paths more easily. Unfortunately, I also needed to restructure the GAMMA tests to use the same format as the north/south Gateway API tests, which has made this a _large_ change. The main changes: * Added GRPCRoute support and small refactor to ingestion in `operator/pkg/mode/ingestion`, `gamma.go` and `gateway.go`. * GAMMA reconciler in `operator/pkg/gateway-api` migrated to use `operator/pkg/gateway-api/indexers`, added in cilium#41232. * GRPCRoute support added to `operator/pkg/gateway-api` as well. * Tests in `opreator/pkg/gateway-api/testdata/gamma` updated to use `input/` and `output/` directories, as in `operator/pkg/gateway-api/testdata/gateway-api`. Signed-off-by: Nick Young <[email protected]>

This commit adds support for using GRPCRoutes for east/west traffic routing with GAMMA. It includes a small amount of refactoring of the Gateway API reconciler as well, to enable the HTTPRoute and GRPCRoute translation logic to share code across both north/south and east/west traffic paths more easily. Unfortunately, I also needed to restructure the GAMMA tests to use the same format as the north/south Gateway API tests, which has made this a _large_ change. The main changes: * Gateway API conformance test workflow now requires `kubectl --server-side`, as the HTTPRoute CustomResourceDefinition will fail to apply without it (the `last-applied-config` annotation is too large to fit in 256kb). Updated `Makefile.kind` and `.github/workflows/conformance-gateway-api.yaml` accordingly. * Added GRPCRoute support and small refactor to ingestion in `operator/pkg/mode/ingestion`, `gamma.go` and `gateway.go`. * GAMMA reconciler in `operator/pkg/gateway-api` migrated to use `operator/pkg/gateway-api/indexers`, added in #41232. * GRPCRoute support added to `operator/pkg/gateway-api` as well. * Tests in `opreator/pkg/gateway-api/testdata/gamma` updated to use `input/` and `output/` directories, as in `operator/pkg/gateway-api/testdata/gateway-api`. Signed-off-by: Nick Young <[email protected]>

youngnick requested review from a team as code owners August 18, 2025 05:10

youngnick requested review from gandro and mhofstetter August 18, 2025 05:10

maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Aug 18, 2025

youngnick force-pushed the refactor-gwapi-reconciler branch from 070aa3e to f18956a Compare August 18, 2025 06:00

youngnick added release-note/misc This PR makes changes that have no direct user impact. and removed dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. labels Aug 18, 2025

youngnick force-pushed the refactor-gwapi-reconciler branch 2 times, most recently from 953db3a to ea55fd1 Compare August 18, 2025 07:10

gandro approved these changes Aug 18, 2025

View reviewed changes

operator/pkg/gateway-api/cell.go Outdated Show resolved Hide resolved

operator/pkg/gateway-api/helpers.go Outdated Show resolved Hide resolved

operator/pkg/gateway-api/helpers.go Outdated Show resolved Hide resolved

operator/pkg/gateway-api/gateway_status.go Outdated Show resolved Hide resolved

youngnick force-pushed the refactor-gwapi-reconciler branch from ea55fd1 to 7588899 Compare August 19, 2025 07:52

youngnick mentioned this pull request Aug 19, 2025

Cilium-Envoy stops receiving config updates once there is one invalid listener config with Gateway API #34982

Closed

1 task

mhofstetter reviewed Aug 19, 2025

View reviewed changes

maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Aug 19, 2025

mhofstetter removed the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Aug 19, 2025

maintainer-s-little-helper bot added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Aug 20, 2025

youngnick force-pushed the refactor-gwapi-reconciler branch from 7588899 to 0e09191 Compare August 20, 2025 03:28

youngnick force-pushed the refactor-gwapi-reconciler branch 3 times, most recently from 74f6540 to 511cc80 Compare August 20, 2025 04:34

youngnick mentioned this pull request Aug 20, 2025

gateway-api: Fix GRPCRoute status not being displayed correctly #39038

Closed

mhofstetter approved these changes Aug 20, 2025

View reviewed changes

mhofstetter added the dont-merge/needs-rebase This PR needs to be rebased because it has merge conflicts. label Aug 20, 2025

youngnick added backport/author The backport will be carried out by the author of the PR. needs-backport/1.18 This PR / issue needs backporting to the v1.18 branch labels Sep 17, 2025

This was referenced Sep 17, 2025

[v1.18] gateway-api: Refactor Gateway API reconciler #41720

Merged

Gateway API: Overzealous "Invalid CertificateRef" in Gateway object #39425

Closed

julianwiedmann added backport-done/1.18 The backport for Cilium 1.18.x for this PR is done. and removed needs-backport/1.18 This PR / issue needs backporting to the v1.18 branch labels Sep 22, 2025

youngnick mentioned this pull request Oct 17, 2025

Gateway API: GRPCRoute not attaching to Gateway #39021

Closed

3 tasks

cilium-release-bot bot added this to cilium v1.19.0 Feb 3, 2026

cilium-release-bot bot moved this to Released in cilium v1.19.0 Feb 3, 2026

Conversation

youngnick commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gandro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

youngnick commented Aug 19, 2025

Uh oh!

mhofstetter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

youngnick commented Aug 20, 2025

Uh oh!

mhofstetter left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jfmatth commented Aug 28, 2025

Uh oh!

youngnick commented Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jfmatth commented Aug 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

youngnick commented Sep 1, 2025

Uh oh!

jfmatth commented Sep 1, 2025 via email

Uh oh!

jfmatth commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jfmatth commented Sep 2, 2025

Uh oh!

youngnick commented Sep 2, 2025

Uh oh!

jfmatth commented Sep 2, 2025

Uh oh!

youngnick commented Sep 3, 2025

Uh oh!

jfmatth commented Sep 4, 2025

Uh oh!

jfmatth commented Sep 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

youngnick commented Aug 18, 2025 •

edited

Loading

mhofstetter left a comment •

edited

Loading

youngnick commented Aug 29, 2025 •

edited

Loading

jfmatth commented Aug 30, 2025 •

edited

Loading

jfmatth commented Sep 2, 2025 •

edited

Loading

jfmatth commented Sep 7, 2025 •

edited

Loading