Summary
The inference-perf performance validator hardcodes the Dynamo frontend router mode to DYN_ROUTER_MODE: kv (in validators/performance/testdata/inference/dynamo-deployment.yaml). Make it configurable like the other AICR_INFERENCE_PERF_* methodology knobs, so operators can A/B the routing strategy (e.g. kv vs least-loaded) without editing the image.
Motivation
Dynamo v1.2 adds frontend router strategies (kv, round-robin, random, least-loaded, device-aware-weighted). Today there's no way to characterize their throughput/TTFT impact through the validator — the only routing lever is recipe inference-routing-mode (dynamo-router | gateway-epp), which selects the routing layer, not the dynamo-router sub-strategy. The strategies only apply under inference-routing-mode=dynamo-router.
Proposed change
- Add env knob
AICR_INFERENCE_PERF_ROUTER_MODE (default kv), validated against the Dynamo frontend enum (kv, round-robin, random, least-loaded, device-aware-weighted); invalid value fails closed (ErrCodeInvalidRequest), consistent with the other perf knobs.
- Template the value into the DGD as
DYN_ROUTER_MODE: ${ROUTER_MODE} (replacing the hardcoded kv).
- Only affects the
dynamo-router path (gateway-epp routes via the EPP).
Notes
A working implementation already exists locally and was used to A/B least-loaded vs kv on the aicr3 H100 cluster (built as aicr-validators/performance:lb-exp). This issue tracks landing it as a first-class, documented knob (catalog env doc + cli/validation docs) and wiring it through release image builds. Can submit the PR.
Type of Change
- New feature (validation methodology knob)
Summary
The inference-perf performance validator hardcodes the Dynamo frontend router mode to
DYN_ROUTER_MODE: kv(invalidators/performance/testdata/inference/dynamo-deployment.yaml). Make it configurable like the otherAICR_INFERENCE_PERF_*methodology knobs, so operators can A/B the routing strategy (e.g.kvvsleast-loaded) without editing the image.Motivation
Dynamo v1.2 adds frontend router strategies (
kv,round-robin,random,least-loaded,device-aware-weighted). Today there's no way to characterize their throughput/TTFT impact through the validator — the only routing lever is recipeinference-routing-mode(dynamo-router|gateway-epp), which selects the routing layer, not the dynamo-router sub-strategy. The strategies only apply underinference-routing-mode=dynamo-router.Proposed change
AICR_INFERENCE_PERF_ROUTER_MODE(defaultkv), validated against the Dynamo frontend enum (kv,round-robin,random,least-loaded,device-aware-weighted); invalid value fails closed (ErrCodeInvalidRequest), consistent with the other perf knobs.DYN_ROUTER_MODE: ${ROUTER_MODE}(replacing the hardcodedkv).dynamo-routerpath (gateway-epp routes via the EPP).Notes
A working implementation already exists locally and was used to A/B
least-loadedvskvon the aicr3 H100 cluster (built asaicr-validators/performance:lb-exp). This issue tracks landing it as a first-class, documented knob (catalogenvdoc + cli/validation docs) and wiring it through release image builds. Can submit the PR.Type of Change