Summary

Switch the inference-perf validation workload's Dynamo frontend from KV-cache-aware routing (DYN_ROUTER_MODE=kv) to load-aware least-loaded routing (DYN_ROUTER_MODE=least-loaded).

Motivation / Context

The inference-perf frontend routed by KV overlap, which can keep sending a transiently-slow worker its full share of requests. On EKS H100 at the 2048-concurrency saturation knee this produced stochastic worker-stall / throughput degradation — TTFT p99 spiking from ~700 ms to 4–77 s and aggregate throughput dropping 30–85% — while GKE H100 stayed consistently clean. least-loaded balances by each worker's active in-flight load, so a backed-up worker stops receiving new requests until it drains, removing the positive-feedback loop behind the stall.

Why least-loaded is the best fit for the AICR performance benchmark. The performance phase drives the workload with AIPerf under deliberately uniform, reproducible conditions — a single served model, a pinned prompt pool, fixed input/output token counts (stddev 0), and greedy decoding (temperature: 0) — against a homogeneous pool of identical single-GPU vLLM workers (one H100 per worker via DRA). Under those conditions KV-overlap (kv) routing has no advantage to exploit: there is no worker heterogeneity to weigh, and the synthetic prompts carry no meaningful cross-request prefix-cache reuse for the router to capitalize on. What KV routing does do is bias requests toward an "overlap-matching" worker, which lets a transiently-slow worker keep accumulating load — exactly the saturation-knee failure mode in #1197. Least-loaded optimizes for the one signal that actually governs tail latency in this benchmark — instantaneous per-worker queue depth — so it is the most appropriate mode for AICR's AIPerf-driven throughput/TTFT measurement. (KV-aware routing remains the right default for production traffic with real prompt-prefix reuse; this change scopes only the benchmark workload.)

Fixes: N/A
Related: #1197, #1043 (epic)

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• Documentation update

Component(s) Affected

• Validator (pkg/validator)
• Docs/examples (docs/, examples/)

Implementation Notes

• One-line behavior change: DYN_ROUTER_MODE on the Frontend container in validators/performance/testdata/inference/dynamo-deployment.yaml flips kv → least-loaded.
• least-loaded is a first-class Dynamo 1.2 router mode (set via the same frontend env var). No image/runtime bump required — the manifest already targets Dynamo 1.2.0.
• Workers keep the vLLM ZMQ KV-cache event publisher; least-loaded simply does not consume those events. Left in place to avoid widening scope and to keep kv mode trivially restorable.
• The gateway-epp routing path (--router-mode direct sidecars, external endpoint-picker) is unchanged and out of scope.
• Docs updated to match: docs/user/validation.md, docs/contributor/validator.md, and the investigation report docs/contributor/inference-perf-fluctuation.md.

Testing

yamllint validators/performance/testdata/inference/dynamo-deployment.yaml   # clean

YAML + testdata/docs only — no Go source changed, so test/e2e/Go-lint cannot regress. End-to-end behavior (degradation no longer reproduces under least-loaded on EKS H100) needs a cluster run before this leaves draft.

Risk Assessment

• Low — Single env-value change in a validator testdata manifest, easy to revert; kv mode remains available.

Rollout notes: Affects only the inference-perf validation workload, not user-deployed recipes/bundles.

Checklist

• Tests pass locally (make test with -race) — N/A, no Go changes
• Linter passes (yamllint on changed manifest)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality — N/A (testdata value change)
• I updated docs if user-facing behavior changed
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)