Skip to content

feat(validator): make inference-perf routing strategy configurable (AICR_INFERENCE_PERF_ROUTER_MODE) #1374

Description

@yuanchen8911

Summary

The inference-perf performance validator hardcodes the Dynamo frontend router mode to DYN_ROUTER_MODE: kv (in validators/performance/testdata/inference/dynamo-deployment.yaml). Make it configurable like the other AICR_INFERENCE_PERF_* methodology knobs, so operators can A/B the routing strategy (e.g. kv vs least-loaded) without editing the image.

Motivation

Dynamo v1.2 adds frontend router strategies (kv, round-robin, random, least-loaded, device-aware-weighted). Today there's no way to characterize their throughput/TTFT impact through the validator — the only routing lever is recipe inference-routing-mode (dynamo-router | gateway-epp), which selects the routing layer, not the dynamo-router sub-strategy. The strategies only apply under inference-routing-mode=dynamo-router.

Proposed change

  • Add env knob AICR_INFERENCE_PERF_ROUTER_MODE (default kv), validated against the Dynamo frontend enum (kv, round-robin, random, least-loaded, device-aware-weighted); invalid value fails closed (ErrCodeInvalidRequest), consistent with the other perf knobs.
  • Template the value into the DGD as DYN_ROUTER_MODE: ${ROUTER_MODE} (replacing the hardcoded kv).
  • Only affects the dynamo-router path (gateway-epp routes via the EPP).

Notes

A working implementation already exists locally and was used to A/B least-loaded vs kv on the aicr3 H100 cluster (built as aicr-validators/performance:lb-exp). This issue tracks landing it as a first-class, documented knob (catalog env doc + cli/validation docs) and wiring it through release image builds. Can submit the PR.

Type of Change

  • New feature (validation methodology knob)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions