Description

Greatly helps #14184, with an explicit trade-off: it reintroduces the race that #13868 closed (a chance of brief Envoy 5xx with NC response flags around controller restarts), but shrinks that window with a configurable first-connect grace period — trading #13868's unbounded failure mode for a bounded, self-healing one.

The problem

#13868 gated per-client xDS publication behind whole-snapshot readiness checks: if any route-referenced cluster was missing from the per-client CDS map, or any referenced EDS cluster lacked a ClusterLoadAssignment, the client's entire snapshot was withheld. When the gate stays unsatisfiable — ExternalName references, per-client derivation backlog under large unique-client fan-out — this produces the two failure modes in #14184: warm proxies stranded indefinitely on stale endpoints (upstream connect failures to deleted pods), and first-time proxies starved of any config until they fail their startup probe and crashloop. Because the gate was level-triggered and unconditional, no restart or rollout could clear it.

The fix

This PR removes the gates and replaces them with a time-based guard at the other end of the pipe: when a new xDS stream sends its first DiscoveryRequest, the control plane registers the client (kicking off per-client translation) and then sleeps briefly — default 1s, tunable via KGW_XDS_FIRST_CONNECT_DELAY, 0 or negative disables — before returning from the OnStreamRequest callback.

Why that placement works (verified against the pinned go-control-plane, pkg/server/sotw/v3/ads.go): the server invokes OnStreamRequest before creating the stream's first watch/subscription, so delaying the callback's return delays the first response — and each stream's requests are processed on that stream's own goroutine, so the sleep delays only this client. The sleep runs after the registration lock is released and fires at most once per stream (guarded by a new-stream flag from client registration). The delay value lives in an atomic so the test override can't race live stream goroutines. By the time the first watch opens, per-client translation has had a head start, so the first snapshot the client observes is almost always fully converged.

Notes on scope of the delay: it applies to every new stream, including a warm Envoy reconnecting after a controller restart — that proxy keeps serving its existing config during the delay and merely receives its first post-reconnect update up to one grace period later. Other clients and translation are unaffected.

The trade-off, quantified

This restores the original (pre-#13868) reconnect race in the tail case where per-client translation takes longer than the grace period. It manifests as a brief, self-healing burst of NC/500s rather than the gate's indefinite starvation. Field data from an affected user (5,000 req/s, ~1,500 failed requests during a controller restart) puts the pre-gate race window around 300ms; the 1s default covers that with margin. Environments with much larger unique-client fan-out × backend counts can raise the delay; the fan-out itself is being addressed separately (#14343 base+overlay restructuring; DISABLE_POD_LOCALITY_XDS as an immediate lever).

Commit structure (please review commit-by-commit)

Four commits, ordered so the mechanical reverts don't obscure the reviewable delta. Every commit compiles and passes its tests on its own (the reverts leave no dangling references — the BlackholeClusterName constant-ification in the third commit replaces string literals that exist at the second).

1. Revert "proxy-syncer bug fix, but for a bug that never actually happens (proxy-syncer bug fix, but for a bug that never actually happens #13958)" — pure git revert. proxy-syncer bug fix, but for a bug that never actually happens #13958's empty-cluster substitution existed only to serve the gates; without them it would publish a near-empty state-of-the-world CDS to a new client. Reverted first since it was built on top of fix: remedies a reconnect-time xDS race #13868.
2. Revert "fix: remedies a reconnect-time xDS race (fix: remedies a reconnect-time xDS race #13868)" — pure git revert of the readiness gates, the ReferencedClusters proto-walk machinery, and their tests.
3. fix: delay first xDS response per new stream instead of gating snapshots (+402/−26 across 9 files) — the core change:
   • the first-connect delay in pkg/krtcollections/uniqueclients.go: atomic-backed and env-tunable, slept once per new stream (guarded by a new-stream flag from client registration) after registration kicks off per-client translation, outside all locks, with a test shim (export_test.go) and delay-zeroing in TestUniqueClients;
   • re-adds wellknown.BlackholeClusterName and constant-ifies its uses (kept from fix: remedies a reconnect-time xDS race #13868 as standalone cleanup);
   • re-adds three regression tests pinning the anti-starvation properties (errored clusters fail closed without blocking publication; unresolvable BackendRefs at startup and at runtime never withhold the snapshot);
   • updates the defer-guard comment in perclient.go and the no-op Delete branch comment in proxy_syncer.go to record the fix: remedies a reconnect-time xDS race #13868 -> v2.3.0 upstream connect failures bug #14184 history (including the pre-existing SnapshotCache leak for departed clients) instead of restoring the 2024-era comment text.
4. test(e2e): pin anti-starvation properties of per-client xDS publication (+428/−20 across 10 files) — the test strategy for the delay:
   • makes the delay's environment read lazy (first use instead of package init) so test binaries can configure it from TestMain;
   • a focused contract test (TestFirstConnectDelayGatesFirstRequestPerStream) pinning the delay's five properties: the first request of a new stream absorbs the grace period; registration (which kicks off per-client translation) precedes the sleep; follow-up requests on the same stream do not sleep; a new stream sleeps again; zero disables. This is where the delay's behavior is verified — not the slow live-ADS setup suite;
   • pkg/kgateway/setup tests now run with the delay zeroed via TestMain (unless the environment overrides it): the delay is a mitigation, not a correctness requirement, so those eventually-consistent assertions must hold against the raw reconnect race the delay merely narrows — running at zero both surfaces bugs the grace period would mask and cuts the suite from ~226s to ~86s;
   • a new e2e suite XdsStarvation (test/e2e/features/xds_starvation, registered and assigned to cluster-seven): the end-to-end regression pin for the starvation family, built on the credible absence shape (an ExternalName Service, which never produces EndpointSlices). With that never-ready reference in the config, the valid route still serves, a gateway rollout's fresh pods become Ready instead of crash-looping, and a route created after a controller restart still reaches the reconnected proxies. Each of these deadlocked under the reverted gates;
   • rides along with an unrelated one-line Makefile fix found while running the suite locally: kind-create's cluster-existence guard used grep $(CLUSTER_NAME), which false-matches any cluster whose name merely contains the target (e.g. kind-foo matches kind), silently skipping creation and letting subsequent kubectl steps run against whatever cluster owns the current context. Now grep -x.

Change Type

/kind fix

Changelog

Fix a bug where per-client xDS snapshots could be withheld indefinitely (reverts #13868/#13958), stranding running gateway pods on stale endpoints and starving newly created pods of any configuration until they crashloop. The control plane now publishes snapshots unconditionally and instead delays each newly connected client's first xDS response by a short grace period (default 1s, configurable via KGW_XDS_FIRST_CONNECT_DELAY; 0 disables) so per-client translation can converge before the first snapshot is served. Trade-off: in environments where per-client translation exceeds the grace period, a controller restart can once again cause a brief, self-healing burst of upstream-connect (NC) errors on reconnecting proxies; raise KGW_XDS_FIRST_CONNECT_DELAY if you observe this.

Additional Notes

• Errored-cluster fail-closed behavior (required by Gateway API conformance for e.g. invalid BackendTLSPolicy) predates fix: remedies a reconnect-time xDS race #13868 and is unaffected: errored clusters are still excluded from CDS; commit 3 adds a regression test pinning this.
• The "retain last good" semantics also predate the gates and are preserved: a deferred snapshot surfaces as a KRT Delete whose handler is intentionally a no-op, so Envoy keeps its last coherent config.
• Known residual risk: environments where per-client translation exceeds the grace period on reconnect (very large unique-client fan-out × many backends) can still see a transient partial first snapshot. KGW_XDS_FIRST_CONNECT_DELAY is the escape hatch; the structural fixes to the window itself are tracked separately ([WIP] perf: split shared vs per-client cluster state #14343, DISABLE_POD_LOCALITY_XDS).
• Relationship to the wider v2.3.0 upstream connect failures bug #14184 work: this is the minimal, low-risk mitigation of the starvation family. A larger track (per-cluster publish-time resolution with bounded withholds, descended from [WIP] fix: bound per-client xDS publication so deferred snapshots cannot withhold config forever #14237) is being developed separately; it and this PR touch the same gate code from opposite directions, so whichever lands second reconciles against the other. [WIP] perf: split shared vs per-client cluster state #14343 is complementary to both and independently shrinks the convergence window this PR's delay papers over.