Skip to content

gateway-api: rejected ProtocolConflict listeners still serve traffic #46917

Description

@asauber

Is there an existing issue for this?

  • I have searched the existing issues

Version

equal or higher than v1.19.5 and lower than v1.20.0

What happened?

When two listeners share a port and conflict with ProtocolConflict, the Gateway status correctly marks both as Accepted=False / Conflicted=ProtocolConflict. However, the translation pipeline still programs both listeners into Envoy. As a result, both rejected listeners incorrectly listen and serve traffic.

We currenly have a test which (incorrectly) validates that the resulting CEC contains listening filter chains in the case of both an HTTPS and TLS listener. The test asserts that the status of each listener Accepted=False and Programmed=False, but Envoy is programmed to listen for traffic on 443.

Note that this only happens if the Gateway has at least one other accepted listener. That accepted listener allows the Gateway to be Programmed, and the rejected listeners appear in the resulting Envoy config.

This can be confirmed by installing the same type of manifest into a cluster and observing that Envoy listens for traffic on 443.

Reproduction

Stand up a fresh cluster running Gateway API

make kind-down || true
make kind
WAIT_DURATION=5m make kind-servicemesh-install-cilium-fast

Generate and install certificates

Two independent certificates are needed: one served by Envoy for the HTTPS Terminate listener, and one served by the TLS passthrough backend.

CERTDIR="$(mktemp -d)"
echo "certs in ${CERTDIR}"

# Cert presented by Envoy for the HTTPS Terminate listener.
openssl req -x509 -newkey rsa:2048 -nodes -days 3 \
  -keyout "${CERTDIR}/gw.key" -out "${CERTDIR}/gw.crt" \
  -subj "/CN=gateway-terminated/O=gateway" \
  -addext "subjectAltName=DNS:https.example.com,DNS:tls.example.com"

# Cert presented by the TLS passthrough backend.
openssl req -x509 -newkey rsa:2048 -nodes -days 3 \
  -keyout "${CERTDIR}/be.key" -out "${CERTDIR}/be.crt" \
  -subj "/CN=tls.example.com/O=backend-passthrough" \
  -addext "subjectAltName=DNS:tls.example.com"

kubectl create namespace gw-conflict \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n gw-conflict create secret tls tls-checks-certificate \
  --cert="${CERTDIR}/gw.crt" --key="${CERTDIR}/gw.key" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n gw-conflict create secret tls passthrough-backend-cert \
  --cert="${CERTDIR}/be.crt" --key="${CERTDIR}/be.key" \
  --dry-run=client -o yaml | kubectl apply -f -

Apply the backend and the conflicting Gateway

Save the manifest below as repro.yaml. It declares a single TLS passthrough backend (nginx) plus a Gateway with two conflicting listeners on port 443 (HTTPS Terminate and TLS Passthrough) and a TLSRoute attached to the passthrough listener.

The http listener on port 80 is the accepted listener which allows the Gateway to be Programmed.

apiVersion: v1
kind: ConfigMap
metadata:
  name: tls-backend-conf
  namespace: gw-conflict
data:
  default.conf: |
    server {
      listen 443 ssl;
      server_name tls.example.com;
      ssl_certificate     /etc/nginx/tls/tls.crt;
      ssl_certificate_key /etc/nginx/tls/tls.key;
      location / {
        return 200 "hello from tls-passthrough backend\n";
        add_header Content-Type text/plain;
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tls-backend
  namespace: gw-conflict
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tls-backend
  template:
    metadata:
      labels:
        app: tls-backend
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 443
          volumeMounts:
            - name: tls
              mountPath: /etc/nginx/tls
              readOnly: true
            - name: conf
              mountPath: /etc/nginx/conf.d
              readOnly: true
      volumes:
        - name: tls
          secret:
            secretName: passthrough-backend-cert
        - name: conf
          configMap:
            name: tls-backend-conf
---
apiVersion: v1
kind: Service
metadata:
  name: tls-backend
  namespace: gw-conflict
spec:
  selector:
    app: tls-backend
  ports:
    - name: https
      port: 443
      targetPort: 443
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gateway-tlsroute-mixed
  namespace: gw-conflict
spec:
  gatewayClassName: cilium
  listeners:
    # one valid listener is required so the Gateway is Programmed
    - name: http
      port: 80
      protocol: HTTP
    # these two listeners conflict
    - name: https
      port: 443
      protocol: HTTPS
      allowedRoutes:
        namespaces:
          from: Same
      tls:
        mode: Terminate
        certificateRefs:
          - name: tls-checks-certificate
    - name: tls
      port: 443
      protocol: TLS
      allowedRoutes:
        namespaces:
          from: Same
      tls:
        mode: Passthrough
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
  name: tlsroute-mixed
  namespace: gw-conflict
spec:
  parentRefs:
    - name: gateway-tlsroute-mixed
  hostnames:
    - tls.example.com
  rules:
    - backendRefs:
        - name: tls-backend
          port: 443

Apply it and wait for the backend to be ready:

kubectl apply -f repro.yaml
kubectl -n gw-conflict rollout status deploy/tls-backend --timeout=120s

Observe the issue

echo -e "\n[Listener status]"
kubectl -n gw-conflict get gateway gateway-tlsroute-mixed -o jsonpath='{range .status.listeners[*]}{.name}{": Accepted="}{.conditions[?(@.type=="Accepted")].status}{" Conflicted="}{.conditions[?(@.type=="Conflicted")].reason}{"\n"}{end}'

echo -e "\n[Backing Service exposes 443]"
kubectl -n gw-conflict get svc cilium-gateway-gateway-tlsroute-mixed \
  -o jsonpath='{.spec.ports[*].port}{"\n"}'

echo -e "\n[CEC programs both 443 filter chains]"
kubectl -n gw-conflict get ciliumenvoyconfig cilium-gateway-gateway-tlsroute-mixed -o yaml \
  | grep -E 'routeConfigName: listener-secure|statPrefix: tls-passthrough' || true

GWIP="$(kubectl -n gw-conflict get gateway gateway-tlsroute-mixed \
  -o jsonpath='{.status.addresses[0].value}')"
echo "gateway address: ${GWIP}"

echo -e "\n[HTTPS Terminate listener answers on 443]"
echo | openssl s_client -connect "${GWIP}:443" -servername https.example.com 2>/dev/null \
  | openssl x509 -noout -subject

echo -e "\n[TLS Passthrough listener answers on 443]"
echo | openssl s_client -connect "${GWIP}:443" -servername tls.example.com 2>/dev/null \
  | openssl x509 -noout -subject
curl -sk --resolve "tls.example.com:443:${GWIP}" https://tls.example.com:443/

Both listeners report Accepted=False Conflicted=ProtocolConflict, yet the Service exposes 443, the CEC contains both the listener-secure and tls-passthrough filter chains, and both TLS paths on 443 serve traffic.

[Listener status]
http: Accepted=True Conflicted=
https: Accepted=False Conflicted=ProtocolConflict
tls: Accepted=False Conflicted=ProtocolConflict

[Backing Service exposes 443]
80 443

[CEC programs both 443 filter chains]
            routeConfigName: listener-secure
          statPrefix: tls-passthrough:tls.example.com
gateway address: 172.18.255.197

[HTTPS Terminate listener answers on 443]
subject=CN = gateway-terminated, O = gateway

[TLS Passthrough listener answers on 443]
subject=CN = tls.example.com, O = backend-passthrough
hello from tls-passthrough backend

Cleanup

kubectl delete namespace gw-conflict --ignore-not-found

Related

How can we reproduce the issue?

See the reproduction steps in the description section

Cilium Version

v1.20.0-pre.4

Kernel Version

6.8.0-124-generic #124-Ubuntu

Kubernetes Version

v1.35.0

Regression

No response

Sysdump

No response

Relevant log output

Anything else?

No response

Cilium Users Document

  • Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions