Summary

The inference-throughput gate in current inference-perf overlays is a fixed absolute full-node value, while the recipes are SKU-agnostic (matched only on service + accelerator, no node shape). On a node with materially fewer GPUs than the one the floor was calibrated against, the gate false-fails a perfectly healthy run.

Evidence

The floor is a fixed absolute full-node number that does not track node GPU count. All performance-enabled inference overlays pin >= 50000 tok/s, despite being calibrated on different node shapes:

• recipes/overlays/h100-eks-ubuntu-inference-dynamo.yaml — measured on p5.48xlarge, 8 GPUs, throughput ~= 108,790 tok/s; floor >= 50000 (~46% of measurement).
• recipes/overlays/gb200-eks-ubuntu-inference-dynamo.yaml — measured on p6e-gb200.36xlarge, 4 GPUs, throughput ~= 65,952 tok/s; floor >= 50000 (~76% of measurement).
• Also >= 50000 on h100-gke, h100-aks (PR feat(recipe): add AKS H100 Dynamo perf check #1232), and rtx-pro-6000-eks.
• b200-gke-cos-inference-dynamo.yaml has inference-perf disabled pending a reference benchmark, so it is not affected today.

Because the value is a single absolute number rather than per-GPU, the same 50000 is asked of an 8-GPU H100 node, a 4-GPU GB200 node, and (via PR #1232) a 2-GPU AKS node alike.

The evaluator does not rescue smaller full nodes. scaledThroughputThreshold (validators/performance/inference_perf.go:135) only scales the gate down when fewer GPUs are benchmarked than the node exposes (gpuCount < gpuCountPerNode) — i.e. partial occupancy. gpuCountPerNode is discovered at runtime from the node's actual nvidia.com/gpu allocatable. On a full 2-GPU node gpuCount == gpuCountPerNode == 2 → no scaling → the full 50000 floor applies to 2 GPUs.

Sub-8-GPU H100 nodes exist on every provider (so the recipe will legitimately land on them):

Linear estimate (extrapolated from the 8-GPU H100 baseline, ~13,600 tok/s/GPU):

The floor implicitly requires >= ~4 GPUs to be physically achievable.

Proposed fix: single-GPU (per-GPU) normalized target

Introduce a per-GPU constraint rather than redefining the existing one in place, so fixed-shape custom recipes that rely on the absolute-total semantics are not silently broken:

• Add inference-throughput-per-gpu for SKU-agnostic recipes. The evaluator scales it by the number of GPUs actually benchmarked:

  effectiveThreshold = perGPUFloor x gpuCount
  

• Keep inference-throughput as the legacy absolute-total gate for fixed-shape recipes (unchanged semantics).
• Reject a config that sets both for the same check (ErrCodeInvalidRequest) — they express conflicting intents.

The per-GPU form is sound because inference throughput is ~count-linear at fixed concurrency-per-GPU (the recipes already pin inference-concurrency-per-gpu: 256). A single per-GPU value then yields a correct gate on every node shape from one SKU-agnostic recipe:

Each shape gets a proportional, achievable floor.

Implementation sketch

1. Add the inference-throughput-per-gpu constraint name to the inference-perf evaluator (validators/performance/inference_perf.go); document the per-GPU contract alongside the existing absolute inference-throughput.
2. When inference-throughput-per-gpu is present, require gpuCount > 0 (fail closed with ErrCodeInternal if result.gpuCount <= 0 — do not silently treat it as zero or pass through) and gate on perGPUFloor x gpuCount (correct under any occupancy — subsumes the partial-occupancy case scaledThroughputThreshold handles today). When inference-throughput is present, keep the current absolute/full-node-relative behavior. Reject both-set.
3. Migrate the SKU-agnostic inference overlays to inference-throughput-per-gpu with a conservative per-accelerator per-GPU value (H100, GB200, RTX Pro 6000 differ — keep the method consistent, values per accelerator).
4. Update inference-perf unit tests for the new constraint + scaling, and the both-set rejection.

Alternatives considered

Pin the benchmark to 1 GPU by default (fixed small workload, fixed per-GPU floor). Simpler and fully deterministic, but validates a synthetic 1-GPU shape rather than the deployed topology. Per-GPU-scaled-by-benchmarked-count is preferred because it gates the real deployment.

Per-SKU full-node overlays. Instead of normalizing one recipe to any node, make the recipe specific to the node (add SKU / GPU-count as a criteria dimension) so a measured full-node absolute floor is always valid for that shape.

• Pros: most accurate gate — a real per-SKU baseline catches modest regressions, not just gross failures; no evaluator change (the current absolute compare already works); honest that a 2-GPU PCIe NC80 is a different perf class than 8-GPU SXM rather than "8 ÷ 4".
• Cons: combinatorial explosion of overlays (reintroduces node shape on top of service × accelerator × os × intent × platform, each needing its own measured baseline); requires hardware access to calibrate every SKU — exactly the gap PR feat(recipe): add AKS H100 Dynamo perf check #1232 already calls out ("until Azure-specific baselines are available"), so it blocks on testbed time that the normalized approach does not; still needs the partial-occupancy down-scaling at the edges.
• These approaches are not mutually exclusive: the per-GPU normalized floor is the default portable gross-failure gate that only needs a per-accelerator baseline, not per-SKU baselines; keeping the legacy absolute inference-throughput constraint (above) is the opt-in hook for a per-SKU overlay to override with a measured full-node number when someone has the testbed data and wants a tight regression gate for that specific shape.

Note on TTFT

inference-ttft-p99 is a per-request latency metric evaluated at fixed inference-concurrency-per-gpu. Because the benchmark keeps per-GPU load constant, TTFT is not expected to scale with node GPU count the way total throughput does. It is not the source of the SKU-count bug and does not need GPU-count normalization here. (TTFT can still vary with cloud SKU, CPU shape, frontend placement, model-load behavior, and request routing — so this is a statement about GPU-count normalization, not a portability guarantee.)

Scope / related

• The training NCCL busbw gate has related but distinct node-shape sensitivity: it is topology/transport-class dependent, not count-linear. Track that in a separate issue.
• PR feat(recipe): add AKS H100 Dynamo perf check #1232 adds the AKS inference perf phase by mirroring the existing sibling throughput floor. It should either defer the throughput floor for AKS (TTFT-only for now) or adopt the normalized throughput semantics once they land.