Summary

nccl-all-reduce-bw (and -net, -nvls) orchestrate the NCCL all-reduce test via Kubeflow Trainer + TrainJob. When Kubeflow Trainer is not already present on the cluster, the validator installs Trainer v2.2.0 from GitHub (validators/performance/trainer_lifecycle.go), whose manifests pin the JobSet controller image to the Kubernetes staging registry:

us-central1-docker.pkg.dev/k8s-staging-images/jobset/jobset:v0.11.0

That staging tag has been garbage-collected (MANIFEST_UNKNOWN/404), so jobset-controller-manager enters ImagePullBackOff, its admission webhook (mjobset.kb.io) has no endpoints, the NCCL TrainJob cannot create launcher/worker pods, and the performance phase fails. The promoted image registry.k8s.io/jobset/jobset:v0.11.0 exists and is the correct replacement.

Trigger condition

The bug fires only when the validator installs Trainer itself — i.e. isTrainerInstalled returns false. That function is a bare existence Get on the trainjobs.trainer.kubeflow.org CRD (no health check), so:

• Trainer absent (clean cluster) → validator installs v2.2.0 → staging JobSet 404 → fail.
• Trainer present & healthy → install skipped, existing JobSet reused → pass.
• CRD present but controller unhealthy/gone → install skipped, then TrainJob creation fails fast against a dead/missing webhook.

Reproducibility: deterministic when the trigger condition is met (clean cluster / Trainer CRD absent). A healthy pre-installed Trainer or a patched live JobSet yields a pass, so it is not unconditional.

Scope

• All variants share the same runNCCLTrainJob Trainer/TrainJob path (validators/performance/nccl_all_reduce_bw_constraint.go:239); none is inherently immune.
• Supported combos (supportedNCCLCombinations): default → EKS H100/H200, GKE H100, B200/GB200 (any); -net → EKS GB200; -nvls → EKS GB200, OKE GB200. All affected on a clean cluster.
• Training only. Inference performance uses the separate ln/inference-perf path (no Trainer/JobSet); unaffected.

Evidence (live, GKE aicr-demo5, H100)

• crane manifest on the staging image → MANIFEST_UNKNOWN (3/3 probes); promoted registry.k8s.io/jobset/jobset:v0.11.0 → exists.
• jobset-controller-manager pod: ImagePullBackOff, Failed to pull … not found.
• TrainJob event: TrainJobResourcesCreationFailed … failed calling webhook "mjobset.kb.io": no endpoints available for service "jobset-webhook-service".
• NCCL validator: failed to find launcher pod: timeout → performance phase failed.
• Reproduced on 2 consecutive clean-cluster runs. After manually repointing the live JobSet Deployment image to registry.k8s.io/jobset/jobset:v0.11.0, JobSet became Ready, the TrainJob launched, and the test passed at 338.16 GB/s (threshold 225) — confirming the only defect is the image registry path.
• GB200-EKS run passed because Trainer was already installed there ("Kubeflow Trainer already installed, proceeding").

Root cause

Inherited staging-registry reference in upstream Kubeflow Trainer v2.2.0 manifests, activated by staging-registry garbage collection (bit-rot) — not a code regression. Adopting v2.2.0 introduced the reference; GC later armed it.

Fix

• AICR (preferred): rewrite the JobSet image …/k8s-staging-images/jobset/jobset → registry.k8s.io/jobset/jobset (same tag v0.11.0) before applying Trainer resources in trainer_lifecycle.go (kustomize images: transformer, or post-process the built resource map).
• Hardening: make isTrainerInstalled verify controller readiness (not just CRD presence) to avoid the skip-then-fail mode; and/or add a preflight that fails fast with a clear message when the JobSet image is not pullable (instead of a ~7-minute timeout).
• Upstream: file with kubeflow/trainer to stop referencing the ephemeral staging registry for JobSet.

Workaround

Pre-install a healthy Kubeflow Trainer (so isTrainerInstalled short-circuits), or patch the live JobSet Deployment image to registry.k8s.io/jobset/jobset:v0.11.0.