Summary

An audit of every recipe declaring an nccl-all-reduce-bw* performance check
found four (service, accelerator) tuples where the recipe advertises a check
that the performance validator skips — a skipped check does not fail, so the
performance phase reports passed without ever measuring or enforcing the
threshold (false assurance). The root cause is the validator's
supportedNCCLCombinations support table: it is an exact (variant, service) → []accelerator lookup with no Any fallback, so any declared tuple absent from
the table silently no-ops.

These are pre-existing issues on main (independent of any in-flight PR).

Full audit

A100 training recipes (a100-{eks,oke,aks,gke}-…) correctly comment the check
out (no active performance: block); slurm leaves correctly set checks: []
(K8s-native NCCL dropped on Slinky). Those are not affected.

Evidence

The support table (validators/performance/nccl_all_reduce_bw_constraint.go):

variantDefault: {
    CriteriaServiceEKS: {CriteriaAcceleratorH100},          // no b200, no h200
    CriteriaServiceGKE: {CriteriaAcceleratorH100},          // no b200
    CriteriaServiceAny: {CriteriaAcceleratorB200, GB200},   // matches service=any only
    // no CriteriaServiceAKS entry at all
},
variantNVLS: { CriteriaServiceEKS: {GB200}, CriteriaServiceOKE: {GB200} },

Resolution is an exact service lookup with no Any fallback:

if supportedAccelerators, ok := supportedNCCLCombinations[variant][service]; ok {
    for _, a := range supportedAccelerators { if accelerator == a { supported = true } }
}
// gke+b200 -> byService[gke] = [H100] -> b200 absent -> skip
// aks+h100 -> byService[aks] absent entirely          -> skip
// eks+h200 -> byService[eks] = [H100] -> h200 absent  -> skip

Each falls through to "skipped - requires Service + Accelerator to be implemented", and the phase passes.

Why some are "remove" and one is "wire"

Enforcing a check requires the validator to actually run it, which needs a
testbed-validated, transport-specific NCCL runtime template — not just a table
entry. The three tuples differ in how cheaply that can be done:

• gke + b200 (A4) → remove. The only GKE template is TCPXO/FastRak-specific
  to a3-megagpu H100 (tcpxo-daemon sidecar, nccl-tcpxo-installer profile),
  and the GKE Go path is hardwired to TCPXO 8-NIC discovery. A4 uses GPUDirect
  RDMA over RoCE/CX-7 — a different transport that needs a new, testbed-
  validated template (nvcf-dgxc-k8s-gcp-azne1-prd7). Can't author/validate it
  without the testbed.
• aks + h100 → remove. No AKS template and no AKS scheduling path exist at
  all (AKS falls into the service=any bucket). Needs a validated AKS template +
  path.
• eks + h200 → wire. H200 is Hopper on EFA — electrically identical to H100
  for NCCL (NVLink4 intra-node, EFA inter-node). It reuses the existing
  testdata/h100/eks/runtime.yaml and the same calibrated >= 300 GB/s floor, so
  wiring is a table entry + a NVIDIA-H200- product matcher + the shared
  template. (Wants one EFA-testbed confirmation, but the floor is conservative.)

Resolution (this issue's PR)

1. Remove the unenforced nccl-all-reduce-bw check + constraint from
   b200-gke-cos-training.yaml (Kubeflow child inherits) and
   h100-aks-training.yaml (ubuntu child inherits), replacing each with a comment
   explaining why it is intentionally not declared and what wiring it would need.
2. Wire eks + h200: add H200 to variantDefault.EKS, add a NVIDIA-H200-
   product matcher, add testdata/h200/eks/runtime.yaml (mirrors the H100 EKS
   template), and keep the recipe's >= 300 floor. Tests updated.

Follow-ups (tracked separately, gated on testbed characterization): wire
gke + b200 once the B200 GKE A4 RoCE/RDMA template is validated; wire
aks + h100 once an AKS template + scheduling path exist.