Overview of the Issue

I can't reproduce this but creating a ticket to see if other people have run into the same issue.

One day I applied a proxy defaults config and my proxy went into this super tight loop where it was spamming errors and Consul was spamming these errors. Thousands of these messages every couple of seconds:

[ERROR] agent.envoy.xds: got error response from envoy proxy: service_id=frontend-sidecar-proxy \
typeUrl=type.googleapis.com/envoy.config.listener.v3.Listener xdsVersion=v3 nonce=000510fe \
error="rpc error: code = Internal desc = Error adding/updating listener(s) backend:127.0.0.1:6001: \
route: unknown cluster 'backend.default.dc1.internal.48b8d386-6855-4145-3bf2-90dd6fb29e96.consul

(newlines added for readability)

It looks like a race condition occurred where we were trying to XDS add a listener and expected a cluster to be there when it wasn't.

To fix it I had to restart the proxy.

Consul info for both Client and Server

Consul 1.10.4