Summary

Wire the GB200/EKS overlays (training and inference) to self-fulfill the NVreg_GrdmaPciTopoCheckOverride=1 kernel module flag, pin the driver to the NVIDIA-recommended floor for GB200+EFA (580.126.20), and adopt the transport-specific nccl-all-reduce-bw-net / -nvls performance checks that were introduced in #640.

Motivation / Context

After #640 landed, the NET + NVLS NCCL variants existed in the validator catalog but no shipped overlay referenced them — GB200/EKS users still ran the legacy auto-detect nccl-all-reduce-bw and got no transport-specific signal. Separately, GB200/EKS has two operator-side prerequisites that the existing overlay left as manual configuration:

1. NVreg_GrdmaPciTopoCheckOverride=1 — required on the NVIDIA driver so EFA can attach dma-buf to the Grace PCI topology. Without it, NCCL silently falls back to Socket on NET.
2. Driver version floor — NVIDIA's recommended driver for GB200+EFA is 580.126.20; the repo-wide default stayed on 580.105.08 for H100/B200.

This PR closes both gaps declaratively in the overlay, so GB200/EKS recipes come up correctly out of the box, and both the training and inference intents assert the two fabrics (EFA + MNNVL) actually carry traffic.

Fixes: N/A
Related: #640 (validator-side NET/NVLS implementation)

Type of Change

• New feature (non-breaking change that adds functionality)
• Documentation update

Component(s) Affected

• Recipe engine / data (pkg/recipe)
• Docs/examples (docs/, examples/)

Implementation Notes

Four commits, each independently revertable:

1. feat(recipes/gb200-eks): self-fulfill NVreg_GrdmaPciTopoCheckOverride=1 — Adds manifestFiles: [components/gpu-operator/manifests/kernel-module-params.yaml] (existing template, already in the component directory) and points ClusterPolicy.spec.driver.kernelModuleConfig.name at the resulting ConfigMap. The NVIDIA driver DaemonSet then mounts nvidia.conf at load time and the kernel comes up with the flag already set. The existing NVreg preflight check stays in place as a belt-and-suspenders guard for operators who override the module config at a higher layer.

2. feat(recipes/gb200-eks): bump driver to 580.126.20 — Override gpu-operator.driver.version at the GB200/EKS overlay layer only. H100/B200 and non-EKS GB200 (OKE, GKE, AKS) keep the global 580.105.08 default from components/gpu-operator/values.yaml. The version recommendation is GB200+EFA-specific on EKS; narrower blast radius than a global bump.

3. feat(recipes/gb200-eks): adopt nccl-all-reduce-bw-net and -nvls constraints — Replaces the inherited nccl-all-reduce-bw >= 720 from gb200-any-training with nccl-all-reduce-bw-net >= 40 and nccl-all-reduce-bw-nvls >= 500 on gb200-eks-training. ValidationPhase replaces rather than merges, so this is a clean swap on GB200/EKS recipes only; non-EKS GB200 and non-GB200 accelerators keep the legacy entry unchanged. Thresholds are deliberately conservative, sized for a 2-node GB200 pair — will be raised once production NVL72 data is available.

4. feat(recipes/gb200-eks): extend NCCL variants + NVreg fulfillment to inference — Mirrors the above into gb200-eks-inference.yaml. NCCL all-reduce-bw is fabric-health, not training-specific: multi-node inference (tensor-parallel serving, MoE expert parallelism) crosses the same EFA + MNNVL fabrics and has the same dma-buf attach requirement. Also updates docs/user/validation.md with a 3-variant table documenting when each check is selected.

Mixin alternative, rejected. Initially tried extracting the shared GB200/EKS GPU-operator block into a mixin, but the mixin system is strictly additive — it can introduce new componentRef names but cannot extend a componentRef already declared upstream in the inheritance chain (gpu-operator comes from eks-training / eks-inference). Per-leaf duplication with a "keep in sync" comment is the pragmatic choice until the mixin system gains extension semantics; concretely that's ~10 duplicated lines across two files vs. a much larger refactor.

Testing

# Static verification
make qualify                                    # passes, 0 lint issues

# Recipe hydration
aicr query --service eks --accelerator gb200 --intent training --os ubuntu --platform kubeflow \
  --selector components.gpu-operator.values.driver            # → version 580.126.20, kernelModuleConfig.name nvidia-kernel-module-params
aicr query --service eks --accelerator gb200 --intent inference --os ubuntu \
  --selector components.gpu-operator.values.driver            # → same
aicr query --service oke --accelerator gb200 --intent training --os ubuntu \
  --selector components.gpu-operator.values.driver.version    # → 580.105.08 (unchanged)
aicr query --service eks --accelerator h100 --intent training --os ubuntu \
  --selector components.gpu-operator.values.driver.version    # → 580.105.08 (unchanged)

End-to-end on real GB200/EKS hardware (EKS 1.34, Ubuntu 24.04, 2× p6e-gb200.36xlarge, ASG-terminated before redeploy for a clean-boot driver rollout):

On-cluster verification of the NVreg self-fulfillment:

• nvidia-kernel-module-params ConfigMap present in gpu-operator namespace with options nvidia NVreg_GrdmaPciTopoCheckOverride=1
• Driver DaemonSet image: nvcr.io/nvidia/driver:580.126.20-ubuntu24.04; ConfigMap mounted at /drivers/nvidia.conf
• /proc/driver/nvidia/params on both GB200 nodes reports GrdmaPciTopoCheckOverride: 1 — flag is live in the loaded kernel module, not just declared

Risk Assessment

• Low — Isolated to two GB200/EKS overlays + one user-doc page. No CLI, API, or validator-engine code changes.

Rollout notes:

• The three behavior changes (driver 580.126.20, NVreg ConfigMap, NCCL variants) are scoped to GB200/EKS via overlay path, verified by aicr query against OKE/H100 recipes showing no drift.
• The driver bump redeploys the nvidia-driver DaemonSet on GB200/EKS clusters that regenerate their bundle; plan this alongside a rolling-replace of GPU nodes (ASG terminate with --no-should-decrement-desired-capacity) so the new driver lands on clean-boot replacements rather than reinstalling over running kernel state.
• No migration needed for non-GB200 or non-EKS recipes.

Checklist

• Tests pass locally (make test with -race)
• Linter passes (make lint — golangci-lint run ./... → 0 issues)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality (validator-side tests for the new NCCL variants landed in feat(performance): add GB200 EKS support for NCCL all-reduce bandwidth check #640; this PR is recipe/doc only)
• I updated docs if user-facing behavior changed (docs/user/validation.md 3-variant table)
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)