Description:

What issue is being seen? Describe what should be happening instead of
the bug, for example: The expected value isn't returned, etc.

When running Envoy Gateway with mergeGateways: true enabled, deploying an invalid TLS certificate/private key for one Gateway results in inconsistent listener activation across other Gateways' and the proxies. This results in requests flapping as some proxies serve correct certificate and route, whereas some subsequent requests may end up in a proxy missing the correct listeners.

How is the parsing order determined? Seems like the Gateway with the invalid TLS secret correctly fails and is reported in errorState but listeners belonging to other Gateways are not consistently available across all proxies. While examining this, some proxies correctly route traffic based on SNI, while others fall back to different listeners and serve an incorrect certificate (e.g. routing foo.domain.tld traffic to domain.tld or some other wildcard certificate).

Expected behavior:

Invalid TLS certificate should not break other working gateways and their listeners. Other Gateways and their listeners should be otherwise available.

Repro steps:

Include sample requests, environment, etc. All data and inputs
required to reproduce the bug.

Deploy Envoy Gateway with:

mergeGateways: true

• Create multiple Gateway resources managed by the same Envoy Gateway instance.
• Configure at least three Gateways (just to get enough probability to hit the issue) with HTTPS listeners and distinct hostnames (SNI-based routing).
• Set up Gateways with correct TLS secrets and verify correct routing.
• Update one Gateway's TLS secret with some invalid data like bad end of line to break it
• Observe Gateway logs to verify that the parsing fails
• Requests should be flapping / occasionally hit wrong filter chains

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Environment:

Include the environment like gateway version, envoy version and so on.

• Envoy Gateway: 1.6.4
• mergeGateways: true

Logs:

Include the access logs and the Envoy logs.

Failed to load private key from <inline>
Cause: error:09000066:PEM routines:OPENSSL_internal:BAD_END_LINE