Description

Adopts ordered ADS response delivery behind the default-off KGW_ENABLE_ORDERED_ADS setting.

When enabled, kgateway passes sotwv3.WithOrderedADS() to go-control-plane. kgateway’s ADS snapshot cache already produces responses in dependency-oriented type order:

CDS -> EDS -> LDS -> RDS

The default SotW server does not always preserve that order on the wire. It drains separate per-type response channels with reflect.Select, so when several responses are ready at once on a busy stream, an RDS update can be sent before the CDS update containing the cluster it references. Envoy can then briefly hold a valid route whose cluster has not arrived yet, producing a transient 503 NC.

WithOrderedADS() switches ADS streams to a single multiplexed response path. This preserves the snapshot cache response order on the wire and closes the same-snapshot busy-stream addition reordering window.

Busy streams are common in kgateway: backend changes can update CDS and EDS together, route or listener changes can update LDS and RDS together, and proxy startup can answer all ADS watches from one initial snapshot. Making this ordering deterministic removes a rare but user-visible and difficult-to-debug class of transient configuration failure.

This is deliberately a small, reversible change:

• No CRD, API, proxy-bootstrap, or snapshot-construction changes.
• The callback and authentication wiring are unchanged.
• KGW_ENABLE_ORDERED_ADS remains an escape hatch while the new server path soaks.

What this does not solve

Ordered ADS preserves the cache’s existing type order; it does not make every multi-snapshot xDS transition safe.

If Envoy has not ACKed a prior CDS response, the CDS watch is closed. A later snapshot can still send an RDS update through its open watch before the later CDS update can be sent. Ordered ADS cannot order a CDS response that is not ready.

Similarly, the fixed type order is wrong for combined removals: if a snapshot removes a cluster and de-references it from a route, CDS removal is sent before the RDS de-reference. Ordered ADS makes this deterministic; the default path merely made it nondeterministic.

Follow-up work in EP #13586 referenced-only cluster discovery should add reference-ahead and de-reference grace behavior so kgateway avoids publishing those risky combinations in the same vulnerable window.

This is also not a Delta xDS migration. kgateway continues to use SotW ADS.

Trade-offs and rollout

The ordered path is a less-traveled go-control-plane server path. It funnels response types through one buffered multiplexed channel instead of preserving independent per-type response channels and backpressure. A response can be superseded while a resource type is re-subscribed; SotW version handling is expected to re-answer from the new watch, but this changed recovery behavior warrants a soak.

The setting is default-off initially. This PR enables KGW_ENABLE_ORDERED_ADS=true in one e2e matrix leg, including xds_warming coverage, and in one Gateway API conformance leg. We should watch for NACKs, clusters stuck warming, xds_warming flakes, and snapshot-to-ACK latency before considering a default flip.

Tests

This adds stream-level regression coverage for the important boundaries:

• Quiet-stream additions deliver CDS before RDS.
• ACK skew can still deliver RDS before a later CDS update, even with ordered ADS.
• Combined removals still deliver cluster removal before route de-reference in ordered mode.

The tests cover both default and ordered modes so the intended behavioral difference remains explicit.

Change Type

/kind feature

Changelog

Added default-off support for ordered ADS delivery via KGW_ENABLE_ORDERED_ADS=true, preserving CDS, EDS, LDS, and RDS response order for same-snapshot busy-stream additions.