Skip to content

Single invalid TLS Secret on one ListenerSet causes SDS xDS update to be rejected for the entire Gateway & breaking unrelated listeners #9463

Closed

Description

@kevinkupski

Description:
We run Envoy Gateway with a single Gateway and ~32 per-app ListenerSets (one HTTPS listener each, distinct hostnames and distinct TLS Secrets, managed by cert-manager).

One of those TLS Secrets was in KEY_VALUES_MISMATCH state (the Secret had the new tls.key but still the old tls.crt, because the HTTP-01 challenge could not succeed — the underlying domain no longer pointed at us).

Effect on Envoy Gateway:

  • The EG controller pushed all TLS Secrets inline via SDS.
  • Envoy rejected the entire xDS update with code 13, error: Failed to load private key from <inline>, Cause: KEY_VALUES_MISMATCH.
  • Because the update was rejected wholesale, only ~13 of ~32 certs ended up loaded in Envoy.
  • ~19 unrelated *.example.com-style listeners stopped serving TLS, giving clients ERR_CONNECTION_RESET.
  • kubectl rollout restart of the envoy deployment did not recover — the same xDS rejection happened immediately on the new pods, so they came up with the same partial cert set.

The Gateway/ListenerSet status correctly flagged the bad listener with ResolvedRefs=False, reason=PartiallyInvalidCertificateRef, message: "certificate ... has expired ...". So EG already knew the cert was invalid — but still pushed it to Envoy, which then nuked the whole update.

Looks very similar to #7608.

Repro steps:

  1. Create a Gateway with several ListenerSets, each referencing a different TLS Secret (one HTTPS listener each).
  2. Replace one Secret's tls.key with a key that doesn't match its tls.crt (or use an expired cert with mismatched key).
  3. Observe in envoy-gateway controller logs:
    xds: Envoy rejected the last update with code 13 ... Failed to load private key from <inline>, Cause: KEY_VALUES_MISMATCH
    
  4. Check Envoy admin /certs on the data plane pod: cert count is much lower than the number of listeners; other unrelated listeners fail TLS (ERR_CONNECTION_RESET).
  5. Restart the envoy pod(s) — issue persists until the bad Secret is fixed/removed.

Environment:

  • Envoy Gateway: v1.8.1
  • Envoy: v1.38.1
  • K8s: OpenTelekomCloud CCE, v1.33.5
  • Topology: 1 Gateway (HTTP :80 only on the Gateway itself), N ListenerSets each with one HTTPS listener and a distinct TLS Secret. Secrets managed by cert-manager, previously via Ingress, now via ListenerSets.

Logs:
Limited, see description.

Activity

  1. arkodg commented on Jul 12, 2026

    @arkodg
    Contributor
  2. added this to the v1.9.0-rc.1 Release milestone on Jul 12, 2026
  3. arkodg commented on Jul 27, 2026

    @arkodg
    Contributor

    can this be closed @zhaohuabing @cnvergence ?

  4. zhaohuabing commented on Jul 27, 2026

    @zhaohuabing
    Member

    Fixed in #9487.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions