Goal

Make the nccl-all-reduce-bw-net performance validator work on a ConnectX RoCE fabric (e.g. the DGXC GB300 p6e-gb300r clusters), so RoCE training recipes get a real NET performance gate instead of failing or skipping. Today the NET test is hardwired for AWS EFA, so --phase performance on a RoCE recipe cannot run the NET transport.

This issue tracks the RoCE NET variant specifically. The broader question of making fabric a first-class, discoverable dimension is captured as an open question below — not committed here.

Background — nccl-all-reduce-bw-net is EFA-hardwired

The NET validator forces the EFA libfabric provider and requests the EFA extended resource, neither of which exists on RoCE nodes.

In validators/performance/testdata/gb200/eks/runtime-net.yaml:

• EFA-packaged image: public.ecr.aws/hpc-cloud/nccl-tests:cuda12.8.1-efa1.43.2-ofiv1.16.3-ncclv2.27.7-...
• EFA-provider env: FI_PROVIDER=efa, FI_EFA_USE_DEVICE_RDMA=1, FI_EFA_FORK_SAFE=1
• Requests the EFA extended resource: vpc.amazonaws.com/efa (buildEFAResourceLine in nccl_all_reduce_bw_constraint.go)

GB300 reuses the GB200 NCCL testdata — nccl_all_reduce_bw_constraint.go aliases accelerator == gb300 → gb200 for templatePath, so GB300 inherits the EFA-only NET test.

On a RoCE node this fails two ways:

1. vpc.amazonaws.com/efa does not exist → the NET pod is Unschedulable.
2. Even if scheduled, FI_PROVIDER=efa + the EFA-only image cannot drive ConnectX RoCE → no NET transport.

The nccl-all-reduce-bw-nvls variant is unaffected — it exercises NVLink/IMEX (ComputeDomain), independent of the external fabric.

Confirmed by the DGXC platform team (GB300 = ConnectX-8 RoCE, deliberate). Required RoCE config:

• Use IB/verbs, not the EFA libfabric provider: drop FI_PROVIDER=efa; set NCCL_IB_HCA to the CX8 interface names.
• RoCE devices must be explicitly claimed via DRA — a pod gets no RDMA without a ResourceClaimTemplate referencing roce.networking.k8s.aws.
• Canonical workload pattern (DGXC docs, "Configuring Workloads — AWS GB300"): https://dgxcloud.gitlab-master-pages.nvidia.com/platform/docs/latest/users/tasks/workloads/aws/gb300/

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: eight-roce-rct
spec:
  spec:
    devices:
      requests:
      - name: roce
        exactly:
          deviceClassName: roce.networking.k8s.aws
          count: 8            # adjust to GPUs/NICs per pod
---
# pod / worker container:
spec:
  containers:
  - name: worker
    resources:
      claims:
      - name: roce-channel
  resourceClaims:
  - name: roce-channel
    resourceClaimTemplateName: eight-roce-rct

Scope (this issue)

A RoCE NET variant of the NCCL all-reduce test:

1. Transport config: drop FI_PROVIDER=efa / FI_EFA_*; use the verbs/IB RDMA path over the ConnectX VFs (NCCL_IB_HCA=CX8 interface names, appropriate NCCL_SOCKET_IFNAME).
2. Resource request: request RoCE NICs via the RoCE DRA class (roce.networking.k8s.aws) instead of vpc.amazonaws.com/efa.
3. Image: an nccl-tests image with the RDMA/verbs transport (the current one is EFA-packaged).
4. Testdata selection by fabric: pick the NET runtime by fabric rather than the current gb300→gb200 alias that always yields the EFA template.
5. Verify-transport assertion: the RoCE variant must assert the verbs/RoCE transport banner instead of the EFA banner.

Implementation: #1428 (RoCE NET variant keyed by AICR_NCCL_FABRIC=roce, default efa).

Acceptance

• A RoCE NCCL all-reduce NET test that passes on a ConnectX-RoCE GB300 cluster (e.g. aicr-gb300), with the transport-confirmation assertion adapted to RoCE.
• nccl-all-reduce-bw-net selects the EFA or RoCE variant for the node's actual fabric.
• A RoCE-GB300 NCCL bandwidth baseline is captured (the current gb300 NET gate >= 40 is a GB200/EFA-inherited placeholder, never measured on RoCE — see [Epic]: Performance-phase constraints: empirically-grounded thresholds for testbed-blocked overlays #1043/nccl-all-reduce-bw training gate is a fixed absolute fabric-specific busbw value applied to SKU-agnostic recipes → false-fails EKS/H100 small SKUs #1256).

Open question — make fabric a first-class dimension (not committed here)

The same accelerator can run different fabrics, even on the same service: GB300 on EKS is ConnectX-8 RoCE on the NVIDIA-integrated p6e-gb300r, but EFAv4 on the public p6e-gb300. So (accelerator, service) does not determine the fabric. Fabric affects both the driver/runtime config (verbs/IB vs libfabric; RoCE DRA vs vpc.amazonaws.com/efa) and the perf thresholds.

We do not run both fabrics in one recipe (one fabric per repo/recipe — see #1318), so there is no in-recipe runtime selection. But the right variant still has to be chosen for the node's actual fabric. Open design question — how should fabric be expressed/known?

• Add fabric as a recipe criterion (efa / roce), and/or
• Discover it from the snapshot via the existing snapshot → fingerprint → criteria path: EFA = AWS EFA PCI function (vendor 1d0f) / vpc.amazonaws.com/efa; ConnectX RoCE = Mellanox VFs (vendor 15b3) and/or /sys/class/infiniband/*/ports/*/link_layer == Ethernet (works pre-GPU-stack, like PCI-based accelerator detection).

This is TBD and should likely be split into its own issue once more fabric-divergent recipes exist; the RoCE NET variant above does not block on it (it can key off AICR_NCCL_FABRIC in the interim).

Links

• Implementation: feat(validators): RoCE NET variant for nccl-all-reduce-bw #1428.
• Supersedes the fabric-awareness concept from Make EKS networking fabric-aware (EFA vs ConnectX RoCE) — GB300 is fabric-hardcoded #1410 (closed — networking component selection is moot with one fabric per repo).
• Performance-phase epic: [Epic]: Performance-phase constraints: empirically-grounded thresholds for testbed-blocked overlays #1043. Validator-capability epic: [Epic]: Validator capability: performance / conformance checks must cover every shipped workload variant #1044.
• Related: EFA on AWS: legacy device plugin vs networking DRA driver — should we migrate? #1326 (EFA device plugin vs networking DRA driver), nccl-all-reduce-bw training gate is a fixed absolute fabric-specific busbw value applied to SKU-agnostic recipes → false-fails EKS/H100 small SKUs #1256 (fabric-specific busbw threshold), Add and validate GB300 recipe overlays #1318 (GB300 overlays), nccl-all-reduce-bw declared but skipped for gke+b200, aks+h100, eks+h200 (gates never enforced) #1337.