Is there an existing issue for this?
Version
equal or higher than v1.19.5 and lower than v1.20.0
What happened?
When two listeners share a port and conflict with ProtocolConflict, the Gateway status correctly marks both as Accepted=False / Conflicted=ProtocolConflict. However, the translation pipeline still programs both listeners into Envoy. As a result, both rejected listeners incorrectly listen and serve traffic.
We currenly have a test which (incorrectly) validates that the resulting CEC contains listening filter chains in the case of both an HTTPS and TLS listener. The test asserts that the status of each listener Accepted=False and Programmed=False, but Envoy is programmed to listen for traffic on 443.
Note that this only happens if the Gateway has at least one other accepted listener. That accepted listener allows the Gateway to be Programmed, and the rejected listeners appear in the resulting Envoy config.
This can be confirmed by installing the same type of manifest into a cluster and observing that Envoy listens for traffic on 443.
Reproduction
Stand up a fresh cluster running Gateway API
make kind-down || true
make kind
WAIT_DURATION=5m make kind-servicemesh-install-cilium-fast
Generate and install certificates
Two independent certificates are needed: one served by Envoy for the HTTPS Terminate listener, and one served by the TLS passthrough backend.
CERTDIR="$(mktemp -d)"
echo "certs in ${CERTDIR}"
# Cert presented by Envoy for the HTTPS Terminate listener.
openssl req -x509 -newkey rsa:2048 -nodes -days 3 \
-keyout "${CERTDIR}/gw.key" -out "${CERTDIR}/gw.crt" \
-subj "/CN=gateway-terminated/O=gateway" \
-addext "subjectAltName=DNS:https.example.com,DNS:tls.example.com"
# Cert presented by the TLS passthrough backend.
openssl req -x509 -newkey rsa:2048 -nodes -days 3 \
-keyout "${CERTDIR}/be.key" -out "${CERTDIR}/be.crt" \
-subj "/CN=tls.example.com/O=backend-passthrough" \
-addext "subjectAltName=DNS:tls.example.com"
kubectl create namespace gw-conflict \
--dry-run=client -o yaml | kubectl apply -f -
kubectl -n gw-conflict create secret tls tls-checks-certificate \
--cert="${CERTDIR}/gw.crt" --key="${CERTDIR}/gw.key" \
--dry-run=client -o yaml | kubectl apply -f -
kubectl -n gw-conflict create secret tls passthrough-backend-cert \
--cert="${CERTDIR}/be.crt" --key="${CERTDIR}/be.key" \
--dry-run=client -o yaml | kubectl apply -f -
Apply the backend and the conflicting Gateway
Save the manifest below as repro.yaml. It declares a single TLS passthrough backend (nginx) plus a Gateway with two conflicting listeners on port 443 (HTTPS Terminate and TLS Passthrough) and a TLSRoute attached to the passthrough listener.
The http listener on port 80 is the accepted listener which allows the Gateway to be Programmed.
apiVersion: v1
kind: ConfigMap
metadata:
name: tls-backend-conf
namespace: gw-conflict
data:
default.conf: |
server {
listen 443 ssl;
server_name tls.example.com;
ssl_certificate /etc/nginx/tls/tls.crt;
ssl_certificate_key /etc/nginx/tls/tls.key;
location / {
return 200 "hello from tls-passthrough backend\n";
add_header Content-Type text/plain;
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tls-backend
namespace: gw-conflict
spec:
replicas: 1
selector:
matchLabels:
app: tls-backend
template:
metadata:
labels:
app: tls-backend
spec:
containers:
- name: nginx
image: nginx:1.27
ports:
- containerPort: 443
volumeMounts:
- name: tls
mountPath: /etc/nginx/tls
readOnly: true
- name: conf
mountPath: /etc/nginx/conf.d
readOnly: true
volumes:
- name: tls
secret:
secretName: passthrough-backend-cert
- name: conf
configMap:
name: tls-backend-conf
---
apiVersion: v1
kind: Service
metadata:
name: tls-backend
namespace: gw-conflict
spec:
selector:
app: tls-backend
ports:
- name: https
port: 443
targetPort: 443
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: gateway-tlsroute-mixed
namespace: gw-conflict
spec:
gatewayClassName: cilium
listeners:
# one valid listener is required so the Gateway is Programmed
- name: http
port: 80
protocol: HTTP
# these two listeners conflict
- name: https
port: 443
protocol: HTTPS
allowedRoutes:
namespaces:
from: Same
tls:
mode: Terminate
certificateRefs:
- name: tls-checks-certificate
- name: tls
port: 443
protocol: TLS
allowedRoutes:
namespaces:
from: Same
tls:
mode: Passthrough
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
name: tlsroute-mixed
namespace: gw-conflict
spec:
parentRefs:
- name: gateway-tlsroute-mixed
hostnames:
- tls.example.com
rules:
- backendRefs:
- name: tls-backend
port: 443
Apply it and wait for the backend to be ready:
kubectl apply -f repro.yaml
kubectl -n gw-conflict rollout status deploy/tls-backend --timeout=120s
Observe the issue
echo -e "\n[Listener status]"
kubectl -n gw-conflict get gateway gateway-tlsroute-mixed -o jsonpath='{range .status.listeners[*]}{.name}{": Accepted="}{.conditions[?(@.type=="Accepted")].status}{" Conflicted="}{.conditions[?(@.type=="Conflicted")].reason}{"\n"}{end}'
echo -e "\n[Backing Service exposes 443]"
kubectl -n gw-conflict get svc cilium-gateway-gateway-tlsroute-mixed \
-o jsonpath='{.spec.ports[*].port}{"\n"}'
echo -e "\n[CEC programs both 443 filter chains]"
kubectl -n gw-conflict get ciliumenvoyconfig cilium-gateway-gateway-tlsroute-mixed -o yaml \
| grep -E 'routeConfigName: listener-secure|statPrefix: tls-passthrough' || true
GWIP="$(kubectl -n gw-conflict get gateway gateway-tlsroute-mixed \
-o jsonpath='{.status.addresses[0].value}')"
echo "gateway address: ${GWIP}"
echo -e "\n[HTTPS Terminate listener answers on 443]"
echo | openssl s_client -connect "${GWIP}:443" -servername https.example.com 2>/dev/null \
| openssl x509 -noout -subject
echo -e "\n[TLS Passthrough listener answers on 443]"
echo | openssl s_client -connect "${GWIP}:443" -servername tls.example.com 2>/dev/null \
| openssl x509 -noout -subject
curl -sk --resolve "tls.example.com:443:${GWIP}" https://tls.example.com:443/
Both listeners report Accepted=False Conflicted=ProtocolConflict, yet the Service exposes 443, the CEC contains both the listener-secure and tls-passthrough filter chains, and both TLS paths on 443 serve traffic.
[Listener status]
http: Accepted=True Conflicted=
https: Accepted=False Conflicted=ProtocolConflict
tls: Accepted=False Conflicted=ProtocolConflict
[Backing Service exposes 443]
80 443
[CEC programs both 443 filter chains]
routeConfigName: listener-secure
statPrefix: tls-passthrough:tls.example.com
gateway address: 172.18.255.197
[HTTPS Terminate listener answers on 443]
subject=CN = gateway-terminated, O = gateway
[TLS Passthrough listener answers on 443]
subject=CN = tls.example.com, O = backend-passthrough
hello from tls-passthrough backend
Cleanup
kubectl delete namespace gw-conflict --ignore-not-found
Related
How can we reproduce the issue?
See the reproduction steps in the description section
Cilium Version
v1.20.0-pre.4
Kernel Version
6.8.0-124-generic #124-Ubuntu
Kubernetes Version
v1.35.0
Regression
No response
Sysdump
No response
Relevant log output
Anything else?
No response
Cilium Users Document
Code of Conduct
Is there an existing issue for this?
Version
equal or higher than v1.19.5 and lower than v1.20.0
What happened?
When two listeners share a port and conflict with ProtocolConflict, the Gateway status correctly marks both as
Accepted=False/Conflicted=ProtocolConflict. However, the translation pipeline still programs both listeners into Envoy. As a result, both rejected listeners incorrectly listen and serve traffic.We currenly have a test which (incorrectly) validates that the resulting CEC contains listening filter chains in the case of both an HTTPS and TLS listener. The test asserts that the status of each listener Accepted=False and Programmed=False, but Envoy is programmed to listen for traffic on 443.
Note that this only happens if the Gateway has at least one other accepted listener. That accepted listener allows the Gateway to be
Programmed, and the rejected listeners appear in the resulting Envoy config.This can be confirmed by installing the same type of manifest into a cluster and observing that Envoy listens for traffic on 443.
Reproduction
Stand up a fresh cluster running Gateway API
Generate and install certificates
Two independent certificates are needed: one served by Envoy for the HTTPS Terminate listener, and one served by the TLS passthrough backend.
Apply the backend and the conflicting Gateway
Save the manifest below as
repro.yaml. It declares a single TLS passthrough backend (nginx) plus a Gateway with two conflicting listeners on port443(HTTPS Terminate and TLS Passthrough) and aTLSRouteattached to the passthrough listener.The
httplistener on port80is the accepted listener which allows the Gateway to beProgrammed.Apply it and wait for the backend to be ready:
Observe the issue
Both listeners report
Accepted=FalseConflicted=ProtocolConflict, yet the Service exposes443, the CEC contains both thelistener-secureandtls-passthroughfilter chains, and both TLS paths on443serve traffic.Cleanup
Related
How can we reproduce the issue?
See the reproduction steps in the description section
Cilium Version
v1.20.0-pre.4
Kernel Version
6.8.0-124-generic #124-Ubuntu
Kubernetes Version
v1.35.0
Regression
No response
Sysdump
No response
Relevant log output
Anything else?
No response
Cilium Users Document
Code of Conduct