Description

Refs #14184.

A fix for the per-client xDS pipeline: the cost amplifier that turns input churn into queue backlog.

Content-hash verdict cache for strict validation

Under KGW_VALIDATION_MODE=STRICT, every translated cluster is validated by forking
envoy --mode validate (~200–500ms per call). Per-client translation runs once per
(backend, client) pair and re-runs on every recompute — all serialized on a single KRT queue
goroutine. At field scale (hundreds of backends, several connected clients) one client
connecting costs thousands of serialized forks; the queue falls minutes behind, and the
snapshot consistency gates — which check per-client rows for presence — evaluate against
mid-drain state and defer publication. The visible symptom is the issue's:
defer building snapshot until all referenced clusters are ready listing clusters that exist
and are healthy. They were queued, not missing. Worse, a gateway pod that crashloops because
its first snapshot never arrives reconnects as a brand-new client and retriggers the full
fan-out — the backlog feeds itself.

A validation verdict is a pure function of the config bytes, so this PR wraps the binary
validator with an LRU cache keyed on the SHA-256 of the marshalled bootstrap:

• Only deterministic verdicts are memoized (success and ErrInvalidXDS); transient failures
  (exec errors, context cancellation) are never cached. Cached invalid verdicts keep the inner
  error text verbatim and still satisfy errors.Is(err, ErrInvalidXDS).
• Concurrent misses on identical content collapse to one fork (singleflight).
• Memory is bounded by entry count (default 4096, KGW_VALIDATOR_CACHE_SIZE); entries are a
  32-byte hash plus a verdict — the config itself is never stored — so steady state is well
  under 1 MB. Eviction is plain LRU.

Cluster config content is identical across clients and recomputes in the overwhelmingly common
case, so the expensive forks collapse even though each (backend, client) pair still translates
independently. KGW_VALIDATOR_MODE selects CACHE (default) or BINARY (the previous
behavior — a one-env-var rollback). Memoization cannot change validation outcomes, only skip
redundant invocations, which is why CACHE is a safe default.

Measured (in-repo benchmarks, 2ms stubbed validation; the real fork costs 100–250x more):

Change Type

/kind fix

Changelog

Strict validation (KGW_VALIDATION_MODE=STRICT) now caches validation verdicts by config content
hash, eliminating redundant envoy invocations across per-client translation and recomputes. New
settings: KGW_VALIDATOR_MODE (CACHE [default] | BINARY) and KGW_VALIDATOR_CACHE_SIZE.

Additional Notes

• The cache lives in pkg/validator behind the existing Validator interface; the translator
  is unchanged apart from the injected implementation.
• Pairs with the publish-budget PR: this one removes the backlog that makes consistency-gate
  deferrals long; that one bounds the wait when deferrals happen anyway. Either stands alone.
• Tests: cache hit/miss/eviction, verdict-type discipline, singleflight collapse (one inner
  call under 16 concurrent misses). Benchmarks committed in pkg/validator/validator_bench_test.go
  and pkg/kgateway/proxy_syncer/perclient_bench_test.go.