Skip to content

feat(performance): add GB200 EKS support for NCCL all-reduce bandwidth check #531

Description

@sara4dev

Summary

The NCCL all-reduce bandwidth performance check (nccl-all-reduce-bw) skips on GB200 EKS clusters because service=eks, accelerator=gb200 is not in supportedNCCLCombinations.

Current Behavior

Skipping NCCL All Reduce bandwidth validation: unsupported service/accelerator combination service=eks accelerator=gb200

GB200 is only under CriteriaServiceAny, not EKS:

supportedNCCLCombinations = map[recipe.CriteriaServiceType][]recipe.CriteriaAcceleratorType{
    recipe.CriteriaServiceEKS: {recipe.CriteriaAcceleratorH100},           // ← missing GB200
    recipe.CriteriaServiceGKE: {recipe.CriteriaAcceleratorH100},
    recipe.CriteriaServiceAny: {recipe.CriteriaAcceleratorB200, recipe.CriteriaAcceleratorGB200},
}

Proposed Fix

  1. Add recipe.CriteriaAcceleratorGB200 to the EKS entry in supportedNCCLCombinations
  2. Create validators/performance/testdata/gb200/eks/runtime.yaml

Interconnect considerations

GB200 EKS nodes (p6e-gb200.36xlarge) have both MNNVL (Multi-Node NVLink) and EFA:

  • MNNVL: All 18 nodes connected via NVLink — NCCL uses this for high-bandwidth inter-node GPU communication (expected 800+ GB/s bus bandwidth)
  • EFA: Also available for network communication

Ideally the performance check should test both interconnects:

  • NVLink/MNNVL test: Based on gb200/any/runtime.yaml — no EFA resources, NCCL auto-detects NVLink
  • EFA test: Based on h100/eks/runtime.yaml — with FI_PROVIDER=efa, EFA resource limits

This would give a complete picture of inter-node bandwidth capabilities.

Context

Tested on dynamo-aws-dev-01:

  • 18 GB200 GPU nodes, arm64, p6e-gb200.36xlarge
  • Connected via MNNVL + EFA
  • Expected NVLink bus bandwidth: 800+ GB/s for all-reduce

Metadata

Metadata

Fields

No fields configured for Enhancement.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions