Part of epic #1043.

Summary

The inference-perf validation phase shows large run-to-run performance fluctuation on EKS H100 clusters at the 2048-concurrency saturation point (256/GPU × 8 vLLM decode workers). A subset of runs exhibit a severe worker-stall / throughput-degradation in which TTFT p99 jumps from ~700 ms (healthy) to 4–77 s and aggregate throughput drops ~30–85% (123k → 18–84k tok/s), while the same deployment on GKE H100 is consistently clean. The behavior is stochastic and time-varying — entire windows pass cleanly, then a later window fails repeatedly with byte-identical configuration.

This is the underlying performance issue surfaced by (now-closed) #1192. The gate-hardening work in PR #1196 (relax TTFT p99 → <= 2000 ms + pin AIPerf inputs) removes the false-negative knee-jitter but does not fix the severe stall — a 4–77 s degradation fails regardless of the ceiling. Filing this to track the root cause and the long-term fix separately from the gate change.

Environment

• Cluster: EKS, H100 GPU nodes (8× H100 per node), DRA-based 1-GPU-per-worker.
• Workload: DynamoGraphDeployment — Dynamo frontend + 8 vLLM decode workers; AIPerf benchmark Job (aicr-validation namespace) drives 2048 concurrent requests.
• Dynamo runtime: 1.0.2 (post fix(validators): bump dynamo runtime image 0.9.0 -> 1.0.2 (fixes #1192) #1193; the 0.9.0 futures-util panic is a separate, fixed issue).
• Routing: default round-robin (capacity-blind).
• Comparison baseline: GKE H100, identical recipe/overlay path.

Observations / Tests

1. Combined gate-hardening image (PR #1193 + #1196), 3× EKS H100 + 3× GKE H100 @ 2048

* peak 178 is undersampled (6 s collector cadence missed the backup); the 7.4 s TTFT confirms a real degradation.

GKE: 3/3 clean. EKS H100: 1/3 clean, 2 severe degradations.

2. Byte-identical replay of an earlier 6/6-clean setup (this afternoon)

To rule out the #1196 tweaks as the cause, we replayed the exact morning configuration that had produced 6/6 clean passes — same validator image (dynamo102, no AIPerf determinism flags, no worker CPU requests), same recipe (TTFT 1000), same snapshot, same phase, same collector (diag-cpu.sh):

The same failure mode reproduces with zero config changes — confirming the fluctuation is environmental / time-varying, not caused by the gate-hardening tweaks. The morning's 6/6 was simply a quiet window. Note the recurring ~84k tok/s degraded-throughput signature (s2 here, EKS-combo run 3) at peak Running 256 — i.e., degradation without an obvious single-worker queue backup.

Key findings

1. Fluctuation is real and cluster-specific — EKS H100 degrades stochastically; GKE H100 does not (consistently 300–420 ms, 138–141k tok/s).
2. Two distinct failure shapes: (a) one worker backs up to high in-flight (peak 986, 77 s TTFT); (b) broad throughput degradation (~84k, 4–7 s TTFT) at balanced peak Running 256.
3. CPU contention is ruled out — direct node measurement during degraded runs shows CPU PSI some avg10 ≈ 0 and load ~7–25 of 192 vCPU (≈170 idle vCPUs every run). AIPerf co-locating on the GPU node (confirmed: aicr-validation pod on the worker node) is not a CPU-contention mechanism.
4. Not the gate tweaks — identical-config replay reproduces the degradation.
5. Not a TTFT-ceiling problem — the severe stalls (4–77 s) exceed even the relaxed 2000 ms gate; fix(validators): update and tune inference performance validation #1196 fixes only knee-jitter false-negatives.
6. Bursty in time — quiet windows (6/6 clean) alternate with active windows (2/3 fail).

Mitigations

• Applied (PR fix(validators): update and tune inference performance validation #1196): relax TTFT p99 → <= 2000 ms + pin AIPerf inputs (--random-seed, stddev 0, --num-dataset-entries, temperature:0). Removes false-negatives; does not address the stall.
• Rejected: worker CPU/memory requests (no contention measured); moving AIPerf off the GPU node (non-GPU nodes too small to drive 2048).

Proposed long-term solution

• Load-aware routing: adopt least-loaded / power-of-two routing (Dynamo 1.2.x) instead of capacity-blind round-robin, so a transiently-slow worker stops receiving its full share. Requires bumping the Dynamo runtime to 1.2.x (tracked alongside dynamo: centralize runtime image reference — SSOT + registry-override parity + drift guard #1194).
• Sub-knee operating point: consider a throughput-primary gate operating below the 256/GPU saturation knee (e.g. 1024 concurrency) where TTFT tail volatility collapses.
• Root-cause the EKS-specific stall: investigate why EKS H100 degrades where GKE H100 does not (NCCL/network, DRA device binding, vLLM scheduler under round-robin imbalance).

References

• Detailed investigation report: docs/contributor/inference-perf-fluctuation.md (added in PR fix(validators): update and tune inference performance validation #1196).
• Related: inference-perf: validator times out on healthy cluster when dynamo frontend discovery bootstrap races (false negative) #1192 (false-negative discovery, closed), fix(validators): bump dynamo runtime image 0.9.0 -> 1.0.2 (fixes #1192) #1193 (Dynamo 0.9.0→1.0.2 runtime bump), dynamo: centralize runtime image reference — SSOT + registry-override parity + drift guard #1194 (image-reference centralization + dynamo bump tracking).