Summary / Question

AICR EKS recipes provision EFA/RDMA networking via the legacy EFA device
plugin (aws-efa-k8s-device-plugin). AWS now ships a networking DRA driver
(eks/networking-device-dra-plugin, Helm chart aws-net-k8s-dra-plugin) that
allocates network interfaces (EFA / RoCE / ENA) through Dynamic Resource
Allocation. Should AICR migrate its recipes from the legacy device plugin to
the DRA approach — and if so, under what conditions?

Background — two ways to expose AWS RDMA NICs

1. Legacy: EFA device plugin (what AICR ships today)

• Component aws-efa → chart aws-efa-k8s-device-plugin (e.g. v0.5.26),
  kube-system.
• Advertises a single flat extended resource vpc.amazonaws.com/efa; pods
  request vpc.amazonaws.com/efa: N and the device plugin mounts the EFA
  devices.
• Only handles EFA; instance types must be EFA-enabled.

2. New: AWS networking DRA driver

• Image eks/networking-device-dra-plugin (chart aws-net-k8s-dra-plugin,
  app v1.0.0).
• Registers a DRA driver dra.networking.k8s.aws and device classes
  (e.g. roce.networking.k8s.aws); pods get interfaces via ResourceClaims.
• Covers EFA, RoCE, and ENA branch interfaces — finer-grained,
  topology-aware. Requires the DRA framework (beta 1.32, GA 1.34).

Why this came up (observed on a GB300 EKS cluster)

On a managed DGXC GB300 EKS cluster (p6e-gb300r.36xlarge):

• The node's RDMA NICs are RoCE (rocep*), not EFA.
• AICR's aws-efa device plugin crash-loops:
  Instance type p6e-gb300r.36xlarge is not EFA enabled /
  No valid EFA devices found.
• The platform instead runs the networking DRA driver (v1.0.0), which
  manages the RoCE NICs via roce.networking.k8s.aws.

So on this shape the legacy EFA plugin is both non-functional and redundant —
the recipe's hard assumption of EFA is wrong for RoCE-based GPU instances.

Considerations for the migration question

• K8s version floor. DRA is GA only in 1.34+. Recipes targeting older K8s
  can't use the DRA driver; the legacy plugin would still be needed there.
• Instance coverage. RoCE shapes (e.g. p6e-gb300r) are only served by the
  DRA driver; EFA-only shapes (p5/p4d-style) work with either.
• Driver maturity. The AWS networking DRA driver is new (v1.0.0); the EFA
  device plugin is long-established.
• Ownership. On platform-managed (DGXC) clusters the DRA driver is already
  installed cluster-wide, so AICR shipping aws-efa there causes duplication /
  crash-loops. AICR may want to detect and defer rather than install.
• One mechanism per cluster. Running both is redundant/confusing; pick one
  per cluster (see related deployment-validator finding).

Options to weigh

1. Keep legacy EFA device plugin (status quo) — simplest, but broken on RoCE
   shapes and on K8s/instances moving to DRA.
2. Migrate to the networking DRA driver — future-facing, covers RoCE+EFA,
   but raises the K8s floor to 1.34 and adopts a v1.0.0 component.
3. Conditional selection — choose device-plugin vs DRA per instance type
   and/or K8s version in the recipe/overlay.
4. Detect and defer — when a cluster already provides networking (device
   plugin or DRA driver), don't install aws-efa at all.

Asks

• Decision on direction (one of the above, or hybrid).
• If migrating/conditional: which recipes/overlays and what K8s floor.

Related: the deployment validator surfaces the same divergence (recipe assumes
EFA; cluster is RoCE/DRA).