Update — confirmed by the DGXC platform team (GB300 = RoCE, deliberate)

DGXC confirmed the GB300 fabric is intentional infrastructure, which resolves the "is this deliberate / is an EFA GB300 coming?" question below:

• p6e-gb300r is an AWS/NVIDIA co-designed instance with 8 ConnectX-8 (CX8) NICs instead of EFA devices — the hardware dictates the fabric. RoCE is the going-forward fabric for GB300 on AWS (no EFA-based GB300 expected).
• Confirmed across all four production GB300 clusters (dynamo, osmo, nemo-automation, nemo-rl): efa.enabled: false, aws-net-dra.enabled: true → aws-net-k8s-dra-plugin surfacing 8 RoCE devices/node as roce.networking.k8s.aws DRA resources.
• DGXC docs: "Using GB300 with AWS EKS" — https://dgxcloud.gitlab-master-pages.nvidia.com/platform/docs/latest/users/tasks/cluster/aws/aws-gb300/

Implication for this issue: the recipe's RoCE default for GB300 matches confirmed DGXC infrastructure. But AICR is open source and must keep EFA as a first-class fabric — the broader AWS community runs the documented P6/P6e family on EFA/EFAv4 (it's what aws-efa on the eks base already serves for H100/GB200/B200/etc.). So fabric-awareness here is about supporting both EFA and RoCE, selected by detected fabric — RoCE is additive, not a replacement for EFA.

Two consequences:

• Don't let GB300's RoCE bake a DGXC-specific assumption into the public recipe. p6e-gb300r + CX8/RoCE is a DGXC co-designed instance; a non-DGXC consumer on standard AWS could have different hardware. The open-source-correct design detects the node fabric rather than hardcoding accelerator=gb300 → RoCE. (Hardcoding RoCE for gb300 is an acceptable interim while the only public GB300 is the DGXC -r variant, but the durable design is detection.)
• EFA stays fully supported for every EFA accelerator/instance; RoCE is selected only where the hardware is RoCE. The lowest-effort shape (a deviceType-agnostic DeviceClass over the one DRA driver; verified below) supports both fabrics without keying on accelerator at all.

Problem

The GB300 EKS recipe (and AICR EKS networking generally) is fabric-hardcoded: a recipe ships exactly one RDMA networking component regardless of the node's actual fabric. There is no way to express, detect, or select between EFA and ConnectX RoCE.

This surfaced on the DGXC GB300 clusters (p6e-gb300r.36xlarge), which expose a ConnectX RoCE fabric (8× Mellanox ConnectX VFs, rocep*, no EFA hardware). On these nodes the legacy aws-efa device plugin crash-loops (not EFA enabled — wrong fabric, no version fixes it). RoCE is instead managed by the AWS networking DRA driver (aws-net-k8s-dra-plugin, DeviceClass roce.networking.k8s.aws). See #1326 for the original EFA-vs-DRA discussion and the field evidence.

The interim fix on PR #1336 hardcodes the opposite assumption for GB300: it disables aws-efa and adds aws-net-dra unconditionally. That fixes the DGXC RoCE clusters we run, but it means an EFA-based GB300 node would be misconfigured — aws-efa disabled, and a RoCE DRA driver with no RoCE NICs to manage. We traded one hardcoded fabric for the other.

Note: the RoCE-vs-EFA split observed is per-instance-variant, not "all GB300 is RoCE". Verified against AWS's official EC2 docs:

• p6e-gb300r is not a publicly documented AWS EC2 instance type — it appears in no AWS instance-type spec page, product page, or announcement. It looks like a DGXC/capacity-block/preview SKU specific to how these clusters are provisioned.
• AWS's accelerated-computing specs list the documented P6/P6e family (p6-b200.48xlarge, p6-b300.48xlarge, p6e-gb200.36xlarge) as EFA-only — RoCE is not mentioned anywhere for P6/P6e. P6e-GB300 itself has no published spec row yet; the P6 product page mentions P6e-GB300 UltraServers in prose, but the SKU/UltraServer tables still list only GB200/B300/B200 (GA-announced Dec 2025).
• The RoCE DRA mechanism is also undocumented by AWS: the EKS DRA docs document only efa.networking.k8s.aws (the EFA DRA driver / DRANET), not the roce.networking.k8s.aws class we rely on. That class comes from aws-net-k8s-dra-plugin, which ships only on a private NGC repo (see EFA on AWS: legacy device plugin vs networking DRA driver — should we migrate? #1326). So both the instance variant and the RoCE networking driver are non-public / provisioning-specific.
• It's a GB300-only divergence within DGXC. Verified across the DGXC cluster manifests: GB200 clusters use the standard documented instance (p6e.36xlarge) with EFA (aws-efa) — none use RoCE; only the GB300 p6e-gb300r clusters use ConnectX RoCE (aws-net-dra). So the same platform deliberately switched fabrics between GB200 (EFA) and GB300 (RoCE). The ConnectX/RoCE fabric matches NVIDIA's own NVL72 reference networking, suggesting a DGXC/NVIDIA-provisioned config rather than a standard AWS GB300 offering. Open question for AWS/NVIDIA: why did GB300 move from GB200's EFA to RoCE, and is a public EFA-based GB300 still expected?
  • Spans multiple AWS accounts (so it's a provisioning standard, not a one-off): observed on uzbmpaylks-dgxc-k8s-aws-use2-non-prod (aicr-gb300, acct 897722687756) and ompicmfkmg-dgxc-k8s-aws-use2-non-prod (test-skill-dev3, acct 615299774277), both us-east-2; the DGXC manifests show ~11 p6e-gb300r clusters total. Raised with the DGXC platform team for confirmation of intent / go-forward fabric.

So an EFA-based GB300 is the expected behavior for the documented P6e family, and our RoCE p6e-gb300r is the outlier. Either way, fabric must be selected per node/fabric, not by accelerator=gb300. (AWS sources: EC2 accelerated-computing specs instancetypes/ac.html; EC2 P6 product page; EFA overview; EKS EFA device-management/DRA docs.)

Current state / gaps

• Express: recipe.Criteria has only service / accelerator / intent / os / platform / nodes — no fabric dimension.

• Detect: neither the collector nor pkg/fingerprint captures NIC/RDMA fabric (EFA vs RoCE) yet — but the fabric is PCI-visible and can be detected the same way the accelerator already is. The accelerator dimension is sourced from the nvidia.com/gpu.product label or the PCI device ID (gpu.hardware.model → populatePCIDiscovery → ParseGPUSKU), then projected via Fingerprint.ToCriteria. The PCI-enumeration infra in pkg/collector/gpu already does this, so fabric detection is an extension of an existing path, not a net-new detector. Detection signals:

  • EFA → AWS EFA PCI function (vendor 1d0f, EFA device class), or the vpc.amazonaws.com/efa node resource.
  • ConnectX RoCE → Mellanox VFs (vendor 15b3), and/or /sys/class/infiniband/*/ports/*/link_layer == Ethernet.

  Like PCI-based accelerator detection, this works pre-GPU-stack (raw PCI enumeration; no driver/GFD label/operator required), as long as the snapshot agent runs on the GPU node with PCI/sysfs access.

• Select: component enable/disable is static (overrides.enabled) and set in overlays. Two confirmed mechanics constrain the design:

  • Mixins cannot disable an inherited component — a mixin colliding with an existing component (e.g. aws-efa from the eks base) may only add Namespace/ManifestFiles/PreManifestFiles, not overrides. Disabling must happen in an overlay.
  • Mixins are constraint-gated — a mixin whose constraints fail is excluded, but only against a measurement that distinguishes fabric (none exists) and only when a snapshot is present (the criteria-only aicr recipe path has no measurements).

Options

Option A — New fabric criterion + fabric detection (full automatic)

• Detection (reuses existing infra): extend the PCI-enumerating GPU collector (or add a small network collector beside it) to emit one RDMA-NIC measurement using the signals above, add a Fabric dimension to pkg/fingerprint, and project it via ToCriteria — exactly the snapshot → fingerprint → criteria flow the accelerator already uses. This is the most novel piece but builds on the existing PCI path rather than being net-new, and works pre-stack.
• pkg/recipe/criteria.go: add CriteriaFabricType (efa/roce/any), Criteria.Fabric, Parse/registry/match/specificity.
• Fabric-specific overlays (e.g. gb300-eks-ubuntu-inference-{efa,roce}) selected by the fabric criterion (overlays can disable aws-efa; mixins can't).
• api/aicr/v1/server.yaml enum, --fabric CLI flag (override / no-snapshot path), full enum-addition doc sweep, tests.
• Scope: large (the "new criterion" class — the biggest recipe change), but lower than it first appears since detection reuses the GPU PCI-enumeration path. Works for both snapshot- and flag-driven paths, and is consistent with every other dimension (which are likewise auto-detected from a snapshot and overridable by flag).

Option B — Constraint-gated mixins (no new criterion)

• Add a fabric measurement only; two mixins (fabric-efa, fabric-roce), each constraint-gated and each adding its plugin.
• Blocker: mixins can't disable aws-efa, so the eks base must stop shipping aws-efa and both fabric paths add their own → restructures the shared eks base, affecting all EKS recipes. Snapshot-only (no flag path).
• Scope: medium-large, high blast radius.

Option C — Explicit, no detection (interim)

• Keep RoCE as the gb300 default (matches the DGXC clusters we run); provide an opt-in EFA path — a gb300-eks-ubuntu-inference-efa overlay variant, or a documented override (--set awsefa:enabled=true --set awsnetdra:enabled=false).
• No collector/criterion work; aligns with AICR's "explicit intent, never silently upgrade" principle.
• Scope: small. Both fabrics reachable (one default, one opt-in) but selection is manual.

Refinement: the DRA plugin may be fabric-agnostic (differentiate at DeviceClass, not component)

The DGXC RoCE driver (aws-net-k8s-dra-plugin, driver dra.networking.k8s.aws) appears fabric-agnostic at detection: it enumerates every RDMA NIC and classifies each with a deviceType attribute (dra.vpc.amazonaws.com/deviceType, observed = roce on our nodes); its --set deviceType=… / deviceClasses chart structure implies efa/ena are also supported. A DeviceClass then selects by deviceType via CEL.

If that holds, the per-fabric split is not at the component level — you'd install the one DRA plugin everywhere (replacing aws-efa), and the only fabric-specific choice is which DeviceClass to create and which class a workload's ResourceClaim references (roce.networking.k8s.aws vs an efa class). That is materially smaller than Option A's "fabric-specific overlays with different components": the fabric criterion would steer the DeviceClass / claim, not the installed component. Whoever implements should evaluate this shape first.

Note this is distinct from the AWS-public DRANET driver (dra.net, efa.networking.k8s.aws), which is a separate EFA-only driver — it would not manage our RoCE NICs.

Even simpler — a deviceType-agnostic DeviceClass (verified on aicr-gb300). The roce/efa split is just a CEL filter on the deviceType attribute over the same driver:

device.driver == "dra.networking.k8s.aws" && device.attributes["dra.vpc.amazonaws.com"].deviceType == "roce"

Dropping the deviceType clause yields a fabric-agnostic class that matches whatever RDMA NICs the driver publishes:

device.driver == "dra.networking.k8s.aws"

Tested on aicr-gb300: a DeviceClass with that driver-only selector + a pod whose ResourceClaim referenced it scheduled and allocated a real RoCE device (rocep202s0, driver dra.networking.k8s.aws). So one driver + one deviceType-agnostic DeviceClass would expose the node's RDMA NICs regardless of fabric (RoCE here; EFA wherever the driver tags devices efa) — potentially no fabric criterion or per-fabric branching at all. Workloads reference that single class.

Tradeoff: a driver-only class hands out "any RDMA NIC from this driver" — you lose the ability to request a specific fabric (fine for single-fabric GPU nodes; matters only if a node has mixed fabrics or a workload must pin EFA-vs-RoCE).

Open verification (blocks committing to this shape):

• AWS publicly documents only DRANET (dra.net / efa.networking.k8s.aws); there is no public AWS doc for eks/networking-device-dra-plugin / dra.networking.k8s.aws / roce.networking.k8s.aws. So the plugin's efa deviceType support is unconfirmed from docs.
• All 19 DGXC clusters configure only deviceType: roce; none use efa. The chart ships only on a private NGC repo (we vendored the rendered roce manifests, which don't reveal efa support).
• Unverified: (1) does the plugin support deviceType: efa? (2) can it create/serve multiple DeviceClasses (roce + efa) at once? Needs the chart or AWS/NVIDIA confirmation.

Recommendation

• Now: Option C — make the RoCE-only scope explicit and add a documented EFA override so the recipe is honest and both fabrics are reachable.
• Later (if auto-selection is wanted): Option A — the only approach that is automatic and consistent with the other criteria dimensions; the fabric collector is the foundational, highest-effort piece.
• Avoid Option B — restructuring the shared eks base for all services is high blast radius for a GB300-specific need.

Scope note: the performance validators are fabric-specific too

Fabric-awareness isn't only the networking component — the NCCL NET perf test is fabric-specific in the same way: nccl-all-reduce-bw-net testdata hardwires EFA (FI_PROVIDER=efa, the EFA libfabric image, vpc.amazonaws.com/efa resource), and GB300 reuses it via the gb300→gb200 testdata alias — so GB300-RoCE training would fail the NET check. That perf-validator half is tracked separately in #1413 and should be coordinated with whatever fabric signal this issue introduces.

References

• Support RoCE fabric for the nccl-all-reduce-bw NET test/validation #1413 — companion: nccl-all-reduce-bw-net is EFA-hardwired; needs a RoCE/verbs variant (the perf-validator half of fabric-awareness)
• EFA on AWS: legacy device plugin vs networking DRA driver — should we migrate? #1326 — EFA device plugin vs networking DRA driver (field evidence, two AWS DRA drivers, RoCE confirmation)
• [Epic]: Performance-phase constraints: empirically-grounded thresholds for testbed-blocked overlays #1043 — performance-phase epic
• PR WIP: feat(recipes): GB300 EKS service-bound overlays (re-file of #1319) #1336 — GB300 EKS overlays; currently ships the RoCE-hardcoded interim (aws-net-dra + aws-efa disabled)