Skip to content

feat(validation): close addressable performance-phase gaps on EKS and GKE #1005

Description

@yuanchen8911

Summary

Close the subset of performance-phase floor gaps that don't require testbed access:

  1. recipes/overlays/gb200-eks-ubuntu-inference-dynamo.yaml — add an inference-perf block using the same placeholder smoke thresholds as h100-eks-ubuntu-inference-dynamo.yaml:65-79. GB200 outperforms H100, so the H100 floors (inference-throughput >= 5000, inference-ttft-p99 <= 200) are trivially satisfied — they assert "the system runs," not "GB200 is fast enough." Comment that the values are placeholders pending GB200-tuned numbers.
  2. recipes/overlays/h100-eks-ubuntu-inference-nim.yaml — add the same placeholder inference-perf block (NIM uses the same check name and threshold set).
  3. pkg/recipe/validation_phase_floor_test.go::requiresPerformance — return false when accelerator is unset. This exempts accelerator-unbound intermediates (aks-training, eks-training, gke-cos-training, oke-training) from the strict-mode perf floor because the threshold is accelerator-specific and only meaningful on concrete leaves.

Motivation / Context

AICR_VALIDATION_FLOOR_STRICT=1 go test ./pkg/recipe/... -run TestOverlayValidationPhaseFloor flags 13 overlays today. PR #1001 closes the deployment-phase gap (separate axis). This issue closes 6 of the 13 remaining perf warnings without needing testbeds:

Closed by Overlay
Perf block added gb200-eks-ubuntu-inference-dynamo, h100-eks-ubuntu-inference-nim
Floor-test refinement aks-training, eks-training, gke-cos-training, oke-training

The remaining 7 are testbed-blocked AKS / OCI / LKE leaves, tracked in dedicated issues.

Design notes

  • The two Dynamo / NIM placeholder blocks follow the precedent set by h100-eks-ubuntu-inference-dynamo.yaml — loose smoke-test floors with a comment explaining they're not tight perf gates. Tightening to empirical thresholds is a separate per-accelerator workload-tuning effort.
  • The requiresPerformance refinement is a 3-line rule change: if c.Accelerator == "" || c.Accelerator == "any" { return false }. The rationale: an intent-level intermediate without accelerator binding has no meaningful perf threshold; concrete leaves (h100-eks-training ≥ 300 GB/s, gb200-eks-training ≥ 40 NET / 500 NVLS GB/s) carry their own threshold via per-phase replace. The floor test was over-broad for these intermediates.

Done when

  • go test ./pkg/recipe/... -run TestOverlayValidationPhaseFloor -v continues to pass in default mode.
  • AICR_VALIDATION_FLOOR_STRICT=1 go test ./pkg/recipe/... -run TestOverlayValidationPhaseFloor -v failures drop from 13 to 7 (testbed-blocked only).
  • The 6 newly-closed cases never reappear in either mode.

Related

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions