Summary
The NCCL all-reduce bandwidth performance check (nccl-all-reduce-bw) skips on GB200 EKS clusters because service=eks, accelerator=gb200 is not in supportedNCCLCombinations.
Current Behavior
Skipping NCCL All Reduce bandwidth validation: unsupported service/accelerator combination service=eks accelerator=gb200
GB200 is only under CriteriaServiceAny, not EKS:
supportedNCCLCombinations = map[recipe.CriteriaServiceType][]recipe.CriteriaAcceleratorType{
recipe.CriteriaServiceEKS: {recipe.CriteriaAcceleratorH100}, // ← missing GB200
recipe.CriteriaServiceGKE: {recipe.CriteriaAcceleratorH100},
recipe.CriteriaServiceAny: {recipe.CriteriaAcceleratorB200, recipe.CriteriaAcceleratorGB200},
}
Proposed Fix
- Add
recipe.CriteriaAcceleratorGB200 to the EKS entry in supportedNCCLCombinations
- Create
validators/performance/testdata/gb200/eks/runtime.yaml
Interconnect considerations
GB200 EKS nodes (p6e-gb200.36xlarge) have both MNNVL (Multi-Node NVLink) and EFA:
- MNNVL: All 18 nodes connected via NVLink — NCCL uses this for high-bandwidth inter-node GPU communication (expected 800+ GB/s bus bandwidth)
- EFA: Also available for network communication
Ideally the performance check should test both interconnects:
- NVLink/MNNVL test: Based on
gb200/any/runtime.yaml — no EFA resources, NCCL auto-detects NVLink
- EFA test: Based on
h100/eks/runtime.yaml — with FI_PROVIDER=efa, EFA resource limits
This would give a complete picture of inter-node bandwidth capabilities.
Context
Tested on dynamo-aws-dev-01:
- 18 GB200 GPU nodes, arm64,
p6e-gb200.36xlarge
- Connected via MNNVL + EFA
- Expected NVLink bus bandwidth: 800+ GB/s for all-reduce
Summary
The NCCL all-reduce bandwidth performance check (
nccl-all-reduce-bw) skips on GB200 EKS clusters becauseservice=eks, accelerator=gb200is not insupportedNCCLCombinations.Current Behavior
GB200 is only under
CriteriaServiceAny, not EKS:Proposed Fix
recipe.CriteriaAcceleratorGB200to the EKS entry insupportedNCCLCombinationsvalidators/performance/testdata/gb200/eks/runtime.yamlInterconnect considerations
GB200 EKS nodes (
p6e-gb200.36xlarge) have both MNNVL (Multi-Node NVLink) and EFA:Ideally the performance check should test both interconnects:
gb200/any/runtime.yaml— no EFA resources, NCCL auto-detects NVLinkh100/eks/runtime.yaml— withFI_PROVIDER=efa, EFA resource limitsThis would give a complete picture of inter-node bandwidth capabilities.
Context
Tested on
dynamo-aws-dev-01:p6e-gb200.36xlarge