Summary

Adds GB200 OKE performance phases following the existing training/inference split: NCCL goals for training overlays and inference-perf throughput/TTFT goals for the Dynamo inference overlay. OKE training advertises NVLS only for now; OKE NET/RDMA stays out until the OCI testbed proves a non-Socket NCCL transport end to end.

Motivation / Context

The OKE GB200 leaves were missing the recommended performance phase. This adds conservative smoke goals by mirroring the established EKS/GKE overlay pattern while avoiding an unverified OKE NET contract.

Fixes: #1007
Related: N/A

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Refactoring (no functional changes)
• Build/CI/tooling

Component(s) Affected

• CLI (cmd/aicr, pkg/cli)
• API server (cmd/aicrd, pkg/server)
• Recipe engine / data (pkg/recipe)
• Bundlers (pkg/bundler, pkg/component/*)
• Collectors / snapshotter (pkg/collector, pkg/snapshotter)
• Validator (pkg/validator)
• Core libraries (pkg/errors, pkg/k8s)
• Docs/examples (docs/, examples/)
• Other: ____________

Implementation Notes

• Adds nccl-all-reduce-bw-nvls >= 500 at gb200-oke-training so gb200-oke-ubuntu-training and gb200-oke-ubuntu-training-kubeflow inherit the training performance goal.
• Registers only CriteriaServiceOKE + CriteriaAcceleratorGB200 for the NCCL NVLS variant and ships the OKE NVLS runtime template.
• Keeps the OKE NVLS runtime template on the IMEX/ComputeDomain claim path via resourceClaimTemplateName: nccl-all-reduce-imex.
• Leaves OKE NET/RDMA out of the support matrix and recipe goals until feat(validation): add performance-phase constraints to OKE overlays once OCI testbed lands #1007 or OCI testbed data proves pod RDMA exposure and non-Socket NCCL logs.
• Adds inference-perf to gb200-oke-ubuntu-inference-dynamo, mirroring the GB200 EKS Dynamo model/concurrency/throughput/TTFT goals.
• Removes the stale validation-floor comment that described OKE GB200 training performance as a pending gap.
• Gives OKE NCCL workers default scheduling in platformWorkerScheduling: a tolerate-all toleration (so workers schedule on tainted GB200 pools, not just the launcher) plus a nvidia.com/gpu.product node selector derived from the same accelerator-narrowed node set used for sizing — degrading to no selector on non-GFD installs. An explicit-but-empty --node-selector no longer clears that default asymmetrically (gated on len(), matching the counting path).
• Floors the three OKE GB200 training overlays at K8s.server.version >= 1.34, since the NVLS IMEX path waits on the GA resource.k8s.io/v1 ResourceClaimTemplate (Kubernetes 1.34+); a DRA-incapable cluster is now rejected at recipe resolution rather than failing the performance phase at runtime. The EKS GB200 training chain shares this floor and is tracked in EKS GB200 training recipe floor (>= 1.32.4) is below DRA resource.k8s.io/v1 requirement (1.34+) #1255.
• Documents OKE NVLS selection in docs/user/validation.md and the OKE default validator worker selector in docs/user/cli-reference.md.
• Adds regression coverage for resolved OKE performance goals, OKE NVLS support/template parsing, IMEX claim wiring, OKE worker scheduling (gpu.product selector + taint toleration, non-GFD fallback), commonGPUProduct, the EKS-only GB200 NET NVreg preflight, and the limited scope of runtime-template parse tests.

Performance-gate assumption: the bandwidth/throughput thresholds are conservative smoke gates that assume a full 8-GPU node. The inference-perf goal is mirrored from the EKS H100 8×GPU Dynamo measurement (Qwen/Qwen3-8B, 256 concurrency/GPU, >= 50000 tok/s, TTFT-p99 <= 2000 ms) and the nccl-all-reduce-bw-nvls >= 500 GB/s figure is sized for a full-node 2-node pair — neither is yet calibrated against OKE GB200 silicon. They should be re-baselined once OCI testbed data is available; until then a partially-populated node (fewer than 8 GPUs) is expected to under-shoot these gates.

Testing

go test ./validators/performance -count=1
go test ./pkg/recipe -count=1
golangci-lint run -c .golangci.yaml ./validators/performance/... ./pkg/recipe/...
env -u GITLAB_TOKEN make qualify

All checks passed locally on the rebased branch. make qualify completed the full gate; the vulnerability scan reported the existing k8s.io/kubernetes advisories and did not fail the gate.

Risk Assessment

• Low — Isolated change, well-tested, easy to revert
• Medium — Touches multiple components or has broader impact
• High — Breaking change, affects critical paths, or complex rollout

Rollout notes: OKE NET/RDMA is intentionally excluded until an OCI-specific runtime template and passing non-Socket NCCL logs are available from the testbed.

Checklist

• Tests pass locally (make test with -race)
• Linter passes (make lint)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality
• I updated docs if user-facing behavior changed
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S) — GPG signing info