Summary
The inference-perf check defaulted to a tiny smoke-test model (Qwen/Qwen3-0.6B) with only global tuning knobs, so it couldn't characterize modern accelerators or adapt per GPU type. It is now configurable per recipe, backed by a model-weights cache, with per-accelerator thresholds re-measured on real hardware — so aicr validate --phase performance reflects GPU compute, not serving overhead, and works out of the box.
Implemented on branch feat/inference-perf-configurable-model (stacked on merged PR #1096); validated end to end on RTX PRO 6000, H100 (EKS + GKE), and GB200.
Main changes
1. Per-recipe configuration
inference-model and inference-concurrency-per-gpu are now read from the recipe's validation.performance.constraints — symmetric with the inference-throughput / inference-ttft-p99 thresholds that already live there. Each <accel>-<service>-…-inference-dynamo overlay pins its own values. Resolution precedence is recipe constraint > catalog env (AICR_INFERENCE_PERF_*) > compiled default. New defaults: Qwen/Qwen3-8B at 256 concurrency/GPU (the empirical throughput sweet spot across all tested accelerators).
2. Model-weights cache
A PVC-backed cache is on by default: a one-time populate Job downloads the model once, and all workers mount it read-only (HF_HUB_OFFLINE=1). This removes the per-worker Hugging Face download that tripped per-IP HTTP 429 throttling on large models, and needs no token for ungated models. Disable with AICR_INFERENCE_PERF_MODEL_CACHE_SIZE=off. The cache fails fast with actionable guidance when no StorageClass is resolvable (instead of leaving the PVC Pending until timeout); on EKS the aws-ebs-csi-driver component now provisions a default gp3 StorageClass so it works zero-config (GKE already has standard-rwo).
3. Updated / improved thresholds
The per-accelerator gates (previously placeholders tuned to 0.6B at conc 16: >= 5000 / <= 200) were re-measured at 8B / 256 and pinned alongside the model + concurrency in each overlay:
| Accelerator (overlay) |
throughput gate |
TTFT p99 gate |
measured (8B/256) |
| RTX PRO 6000 (eks) |
>= 50000 |
<= 1000 |
59,260 tok/s · 445 ms · 100% util |
| H100 (eks + gke) |
>= 50000 |
<= 1000 |
~108k tok/s · ~700 ms · 100% util |
| GB200 (eks) |
>= 50000 |
<= 2000 |
65,952 tok/s · 1,240 ms · 80% util |
| B200 (gke) |
>= 50000 |
<= 2000 |
untested — mirrors GB200 (flagged in-file) |
The evaluator applies a 10% tolerance on top of these.
Remaining work
- Precision (FP8 / NVFP4, typically via the checkpoint ID) — not yet a knob.
- Tensor-parallel size (TP > 1) — the Dynamo deploy template still hardcodes TP=1 / one worker per GPU. Needed for models that exceed one GPU (e.g. 32B BF16 on an 80 GB H100) and to fully load a Blackwell GPU on GB200.
Motivation (from 8B characterization)
At the original 0.6B size the benchmark was serving-overhead-bound and never loaded the GPU. Defaulting to 8B at 256 concurrency/GPU fixes that on Hopper/Ada — H100 and RTX PRO 6000 reach ~100% GPU util at their throughput peak. GB200 reaches only ~80% on 8B (it is over-provisioned for a model this small), which is what the remaining precision + TP / larger-model work is for. TTFT SLO context: interactive ~200 ms, voice ~500 ms, MLPerf server 2 s (70B) – 6 s (405B).
Related
Summary
The
inference-perfcheck defaulted to a tiny smoke-test model (Qwen/Qwen3-0.6B) with only global tuning knobs, so it couldn't characterize modern accelerators or adapt per GPU type. It is now configurable per recipe, backed by a model-weights cache, with per-accelerator thresholds re-measured on real hardware — soaicr validate --phase performancereflects GPU compute, not serving overhead, and works out of the box.Implemented on branch
feat/inference-perf-configurable-model(stacked on merged PR #1096); validated end to end on RTX PRO 6000, H100 (EKS + GKE), and GB200.Main changes
1. Per-recipe configuration
inference-modelandinference-concurrency-per-gpuare now read from the recipe'svalidation.performance.constraints— symmetric with theinference-throughput/inference-ttft-p99thresholds that already live there. Each<accel>-<service>-…-inference-dynamooverlay pins its own values. Resolution precedence is recipe constraint > catalogenv(AICR_INFERENCE_PERF_*) > compiled default. New defaults: Qwen/Qwen3-8B at 256 concurrency/GPU (the empirical throughput sweet spot across all tested accelerators).2. Model-weights cache
A PVC-backed cache is on by default: a one-time populate Job downloads the model once, and all workers mount it read-only (
HF_HUB_OFFLINE=1). This removes the per-worker Hugging Face download that tripped per-IP HTTP 429 throttling on large models, and needs no token for ungated models. Disable withAICR_INFERENCE_PERF_MODEL_CACHE_SIZE=off. The cache fails fast with actionable guidance when no StorageClass is resolvable (instead of leaving the PVCPendinguntil timeout); on EKS theaws-ebs-csi-drivercomponent now provisions a defaultgp3StorageClass so it works zero-config (GKE already hasstandard-rwo).3. Updated / improved thresholds
The per-accelerator gates (previously placeholders tuned to 0.6B at conc 16:
>= 5000/<= 200) were re-measured at 8B / 256 and pinned alongside the model + concurrency in each overlay:>= 50000<= 1000>= 50000<= 1000>= 50000<= 2000>= 50000<= 2000The evaluator applies a 10% tolerance on top of these.
Remaining work
Motivation (from 8B characterization)
At the original 0.6B size the benchmark was serving-overhead-bound and never loaded the GPU. Defaulting to 8B at 256 concurrency/GPU fixes that on Hopper/Ada — H100 and RTX PRO 6000 reach ~100% GPU util at their throughput peak. GB200 reaches only ~80% on 8B (it is over-provisioned for a model this small), which is what the remaining precision + TP / larger-model work is for. TTFT SLO context: interactive ~200 ms, voice ~500 ms, MLPerf server 2 s (70B) – 6 s (405B).
Related
feat/inference-perf-configurable-model— the changes above; cache-on-by-default reproduces the hand-tuned sweep numbers within <1%.