You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Single invalid TLS Secret on one ListenerSet causes SDS xDS update to be rejected for the entire Gateway & breaking unrelated listeners #9463
Description:
We run Envoy Gateway with a single Gateway and ~32 per-app ListenerSets (one HTTPS listener each, distinct hostnames and distinct TLS Secrets, managed by cert-manager).
One of those TLS Secrets was in KEY_VALUES_MISMATCH state (the Secret had the new tls.key but still the old tls.crt, because the HTTP-01 challenge could not succeed — the underlying domain no longer pointed at us).
Effect on Envoy Gateway:
The EG controller pushed all TLS Secrets inline via SDS.
Envoy rejected the entire xDS update with code 13, error: Failed to load private key from <inline>, Cause: KEY_VALUES_MISMATCH.
Because the update was rejected wholesale, only ~13 of ~32 certs ended up loaded in Envoy.
kubectl rollout restart of the envoy deployment did not recover — the same xDS rejection happened immediately on the new pods, so they came up with the same partial cert set.
The Gateway/ListenerSet status correctly flagged the bad listener with ResolvedRefs=False, reason=PartiallyInvalidCertificateRef, message: "certificate ... has expired ...". So EG already knew the cert was invalid — but still pushed it to Envoy, which then nuked the whole update.
Create a Gateway with several ListenerSets, each referencing a different TLS Secret (one HTTPS listener each).
Replace one Secret's tls.key with a key that doesn't match its tls.crt (or use an expired cert with mismatched key).
Observe in envoy-gateway controller logs:
xds: Envoy rejected the last update with code 13 ... Failed to load private key from <inline>, Cause: KEY_VALUES_MISMATCH
Check Envoy admin /certs on the data plane pod: cert count is much lower than the number of listeners; other unrelated listeners fail TLS (ERR_CONNECTION_RESET).
Restart the envoy pod(s) — issue persists until the bad Secret is fixed/removed.
Environment:
Envoy Gateway: v1.8.1
Envoy: v1.38.1
K8s: OpenTelekomCloud CCE, v1.33.5
Topology: 1 Gateway (HTTP :80 only on the Gateway itself), N ListenerSets each with one HTTPS listener and a distinct TLS Secret. Secrets managed by cert-manager, previously via Ingress, now via ListenerSets.
Activity
arkodg commentedon Jul 12, 2026
cc @cnvergence @kkk777-7
arkodg commentedon Jul 27, 2026
can this be closed @zhaohuabing @cnvergence ?
zhaohuabing commentedon Jul 27, 2026
Fixed in #9487.