Description:
Currently, we have MTLS connections for:

• Envoy Gateway <> Envoy
• Envoy <> Rate Limit server

Envoy Gateway generates client and server certificates for all of the above components and typically provides them as mounted secrets to relevant pods. A job that runs in the helm pre-install and pre-upgrade hooks is responsible for rotation.

It is a common security practice to use short-lived certificates that are rotated frequently. In Envoy Gateway, CA certificates and leaf certificates are handled with the same level of security (storage, access, ... ), and should both be rotated.

To support frequent and non-disruptive rotation, the following is required:

• All components are capable of dynamically reloading certificates when they change.
• Certificates are rotated in a backwards-compatible manner. For example, a new trusted CA is added and the old CA is removed only after all components were able to load the latest CA/leaf certificate. This is especially relevant for data-plane components like Rate Limit where client-facing failures may occur if Envoy and Rate Limit are not both synchronized on the latests CA/leaf certificates.

Currently, these requirements are not met:

• Dynamic reloading:
  • Envoy Gateway supports re-reading the TLS Key, Cert and CA Cert for each connection:

    gateway/internal/crypto/cert_load.go

    Line 54 in 2385672

    return loadConfig()

  • Envoy supports dynamic reload of TLS Key, Cert and CA Cert through the SDS path-based reference to mounted secrets:

    gateway/internal/infrastructure/common/proxy_sds.go

    Line 10 in 2385672

    // xDS certificate rotation is supported by using SDS path-based resource files.

  • Rate Limit supports dynamic reload of TLS Key/Cert but not CA: https://github.com/envoyproxy/ratelimit/blob/6a2e8262874f012d08830cc34ba8058e66a33819/src/provider/cert_provider.go#L15
• Backwards compatible changes: Envoy Gateway currently overwrites certificates in-place, meaning the change is not backwards-compatible.

  gateway/internal/cmd/certgen.go

  Line 81 in 2385672

  func outputCertsForKubernetes(ctx context.Context, cli client.Client, cfg *config.Server, certs *crypto.Certificates) error {

[optional Relevant Links:]

Any extra documentation required to understand the issue.