Gateway: Combining listeners with HTTPS and TLS protocol

Is there an existing issue for this?

I have searched the existing issues

What happened?

I'm trying to create a Gateway that combines both HTTPS and TLS listeners to proxy external services, but I'm met with

Skipped a listener block: [spec.listeners[1].tls.certificateRef: Required value: listener has no certificateRefs, spec.listeners[1].tls.mode: Unsupported value: "Passthrough": supported values: "Terminate"]

from the Gateway resource when trying to add a TLS listener. From what I gather from the Gateway API documentation I should be able to combine these two listener types.

The Gateway I'm trying to create is

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway
  namespace: gateway
  annotations:
    cert-manager.io/issuer: cloudflare-issuer
spec:
  gatewayClassName: cilium
  infrastructure:
    annotations:
      io.cilium/lb-ipam-ips: 192.168.1.221
  listeners:
    - name: https-gateway
      protocol: HTTPS
      port: 443
      hostname: "*.<DOMAIN>"
      tls:
        certificateRefs:
          - kind: Secret
            name: cert-stonegarden
      allowedRoutes:
        namespaces:
          from: All
    - name: proxmox-tls-passthrough
      protocol: TLS
      port: 443
      hostname: "proxmox.<DOMAIN>"
      tls:
        mode: Passthrough
      allowedRoutes:
        namespaces:
          from: All

with the following Routes

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: home-assistant
  namespace: home-assistant
spec:
  parentRefs:
    - name: gateway
      namespace: gateway
  hostnames:
    - "home-assistant.<DOMAIN>"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: home-assistant
          port: 80

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
  name: proxmox
  namespace: proxmox
spec:
  parentRefs:
    - name: gateway
      namespace: gateway
  hostnames:
    - "proxmox.<DOMAIN>"
  rules:
    - backendRefs:
        - name: proxmox
          port: 443

each having their own Service and EndpointSlice resources.

As a workaround I'm able to create two separate Gateways, one with each listener. This works, but I then have two different Gateways and consequently two IPs to maintain.

Cilium Version

1.51.1

Kernel Version

6.1.0-20-amd64

Kubernetes Version

v1.29.3

Regression

No response

Sysdump

No response

Relevant log output

No response

Anything else?

I encountered this issue trying to proxy both Home Assistant OS and Proxmox as external services through a Gateway.
If it's any help I've described it further in a blog post here.

Cilium Users Document

Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

I agree to follow this project's Code of Conduct

@vehagn

HI @vehagn, can you give some more info about where you got that initial error from? I had a look at this and couldn't reproduce it with Cilium 1.15.1 and either Kubernetes 1.24 or 1.29. I created a similar Gateway and it was accepted by Cilium okay:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway
  namespace: default
spec:
  gatewayClassName: cilium
  listeners:
    - name: https-gateway
      protocol: HTTPS
      port: 443
      hostname: "*.test"
      tls:
        certificateRefs:
          - kind: Secret
            name: selfsigned
      allowedRoutes:
        namespaces:
          from: All
    - name: proxmox-tls-passthrough
      protocol: TLS
      port: 443
      hostname: "proxmox.test"
      tls:
        mode: Passthrough
      allowedRoutes:
        namespaces:
          from: All

@youngnick

Thank you for looking into this @youngnick!

Seeing that you couldn't reproduce the error made me investigate further.
I now think the issue lies with Cert-manager and not Cilium.

I tried to create the same Gateway as you did and did not experience any issues before I added

  annotations:
    cert-manager.io/issuer: cloudflare-issuer

can you give some more info about where you got that initial error from?

I got the error from the events for the Gateway. I initially only looked at them in the Argo CD Web GUI, but looking again now using kubectl describe I see that the source is cert-manager-gateway-shim.

I'll report the issue over at the Cert-manager repo and think this issue can be closed.

I did some further digging and it appears that I'm able to connect a TLSRoute to a Gateway which only supports HTTPRoute kinds. I would expect this to throw some kind of error.

The involved resources are

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: test
  namespace: gateway
spec:
  gatewayClassName: cilium
  listeners:
    - protocol: HTTPS
      port: 443
      name: https-gateway
      hostname: "*.example.com"
      tls:
        certificateRefs:
          - kind: Secret
            name: test-cert
      allowedRoutes:
        namespaces:
          from: All

(Notice I've removed the TLS-listener)

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
  name: test
  namespace: test
spec:
  parentRefs:
    - name: test
      namespace: gateway
  hostnames:
    - "test.example.com"
  rules:
    - backendRefs:
        - name: test-svc
          port: 443

Looking at the status of both they appear to have successfully linked

❯ kubectl -n test get tlsroute test -oyaml | yq '.status'

parents:
  - conditions:
      - lastTransitionTime: "2024-05-04T08:35:56Z"
        message: Accepted TLSRoute
        observedGeneration: 1
        reason: Accepted
        status: "True"
        type: Accepted
      - lastTransitionTime: "2024-05-04T08:35:56Z"
        message: Service reference is valid
        observedGeneration: 1
        reason: ResolvedRefs
        status: "True"
        type: ResolvedRefs
    controllerName: io.cilium/gateway-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: test
      namespace: gateway

❯ kubectl -n gateway get gateway test -oyaml | yq '.status'

addresses:
  - type: IPAddress
    value: 192.168.1.224
conditions:
  - lastTransitionTime: "2024-05-04T08:35:43Z"
    message: Gateway successfully scheduled
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2024-05-04T08:35:43Z"
    message: Gateway successfully reconciled
    observedGeneration: 1
    reason: Programmed
    status: "True"
    type: Programmed
listeners:
  - attachedRoutes: 1
    conditions:
      - lastTransitionTime: "2024-05-04T08:35:56Z"
        message: Invalid CertificateRef
        reason: Invalid
        status: "False"
        type: Programmed
      - lastTransitionTime: "2024-05-04T08:35:56Z"
        message: Listener Accepted
        observedGeneration: 1
        reason: Accepted
        status: "True"
        type: Accepted
      - lastTransitionTime: "2024-05-04T08:35:56Z"
        message: Invalid CertificateRef
        reason: InvalidCertificateRef
        status: "False"
        type: ResolvedRefs
    name: https-gateway
    supportedKinds:
      - group: gateway.networking.k8s.io
        kind: HTTPRoute

Ignoring the Invalid CertificateRef messages the TLSRoute appears to have been successfully accepted by the Gateway.

Is this the intended behaviour, or should I open a new issue?

@vehagn

Thanks for the follow-up @vehagn. Yeah, we shouldn't really support this by the Gateway API spec, although it will work in Cilium's implementation now.

We actually configure all the TLS or HTTPS rules on a single Listener with SNI matching first, then HTTP termination if it's a terminated HTTP connection (which TLSRoute is not). Because of this, the config you've given will actually work with Cilium, even though by the Gateway API spec it should not be allowed.

If you could open another issue and mention me, I would really appreciate it. I suspect we may need to add some upstream conformance tests of the Protocol -> Route type mappings visible on https://gateway-api.sigs.k8s.io/guides/tls/ (although I don't know if we've documented that requirement anywhere else, sigh).

But all of that is for me to worry about upstream - having a clear issue here will help me discuss this with the upstream community.

Thanks again for your follow-up!

(Once you've created the new issue, I can link it here and close this one out.)

Covered by #32371.

Navigation Menu

Repository navigation

Gateway: Combining listeners with HTTPS and TLS protocol #32292

Is there an existing issue for this?

What happened?

Cilium Version

Kernel Version

Kubernetes Version

Regression

Sysdump

Relevant log output

Anything else?

Cilium Users Document

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Gateway: Combining listeners with HTTPS and TLS protocol #32292

Description

Is there an existing issue for this?

What happened?

Cilium Version

Kernel Version

Kubernetes Version

Regression

Sysdump

Relevant log output

Anything else?

Cilium Users Document

Code of Conduct

Activity

youngnick commented on May 3, 2024

vehagn commented on May 3, 2024

vehagn commented on May 4, 2024

youngnick commented on May 6, 2024

youngnick commented on Jun 18, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions