Summary

Retire the gb200-any-training.yaml criteria-wildcard overlay. NCCL bandwidth thresholds (the only content the wildcard carried) belong per-leaf, not on a cross-service wildcard — each service has a different network fabric (EFA on EKS, TCPXO on GKE, RoCE on OKE) and the same number is rarely correct for two fabrics.

The fabric-independent deployment-phase floor (gpu-operator version pin + 4 standard health checks) remains on gb200-any.yaml — that pattern (established by #1001) is correct because the version requirement applies across every service.

Motivation / Context

Per #1052: the wildcard pattern was useful before any service-specific GB200 leaves existed but is misleading now. EKS GB200 training leaves carry per-leaf NCCL thresholds; the wildcard was load-bearing only for the OKE GB200 leaves (which had no perf block of their own). After the OCI testbed work in #1007 lands, gb200-oke-training.yaml will ship its own measurement-anchored constraint and the warn-only data gap closes.

The companion b200-any-training.yaml was a stub — no concrete B200 service-bound leaves existed yet — and is retired by #1004 (PR #1053) as a natural consequence of adding the first concrete B200 leaf.

Fixes: #1052
Related: #1004 (PR #1053 — retires the B200 wildcard), #1007 (OCI testbed; supplies the OKE GB200 NCCL measurement that closes the warn-only gap)

Type of Change

• Refactoring (no functional changes for in-tree recipes today — gb200-oke-* training leaves lose their inherited NCCL constraint, which is currently a warn-only floor recommendation in non-strict mode)

Component(s) Affected

• Recipe engine / data (pkg/recipe) — floor-test header refresh only; no code change
• Docs/examples (docs/) — three pages updated to reflect the retirement

Implementation Notes

Deletion. recipes/overlays/gb200-any-training.yaml removed. It carried only validation.performance.checks/constraints for nccl-all-reduce-bw >= 720.

OKE GB200 training data gap. gb200-oke-training, gb200-oke-ubuntu-training, and gb200-oke-ubuntu-training-kubeflow no longer inherit a performance block. Took option (b) from the issue: defer the real OCI testbed measurement to #1007, leave the gap as a warn-only floor-test result in the meantime. TestOverlayValidationPhaseFloor treats missing performance as t.Log(WARN ...) in default (non-strict) mode, so a knownGaps entry would be stale-flagged by the hygiene guard — instead, the knownGaps header comment in pkg/recipe/validation_phase_floor_test.go documents the pending gap and notes that a strict-mode CI toggle would need three entries referencing #1007.

gb200-any.yaml comment refresh. Dropped the "Companion to gb200-any-training.yaml" intro; explains why the intent-scoped sibling was retired and points at #1052.

Doc updates (all three pages the issue called out):

• docs/integrator/recipe-development.md — switch the criteria-wildcard example from gb200-any-training.yaml to gb200-any.yaml (the accelerator-scoped deployment floor). Add the caveat that per-fabric values don't belong on wildcards.
• docs/contributor/data.md — refresh the wildcard explanation section, the resolver-tracing prose example, and the Mermaid flowchart to use gb200-any throughout. Update the naming convention to mention -any (deployment-floor pattern) alongside -any-<intent>.
• docs/design/005-overlay-refactoring.md — remove b200-any-training and gb200-any-training from the overlay tree; brief historical note explains the retirement (cross-referencing feat(recipes): add concrete GKE B200 service-bound overlays #1004 and refactor(recipes): retire *-any-training.yaml wildcards (NCCL thresholds per-leaf) #1052).

Out of scope.

• b200-any-training.yaml deletion: handled by PR feat(recipes): add concrete GKE B200 service-bound overlays #1053 (issue feat(recipes): add concrete GKE B200 service-bound overlays #1004). Independent of this PR per the coordination note in refactor(recipes): retire *-any-training.yaml wildcards (NCCL thresholds per-leaf) #1052.
• Adding accelerator-wide deployment-floor wildcards for a100 / l40: those accelerators have no overlays at all today (tracked in feat(recipes): add concrete A100 service-bound overlays #1002 / feat(recipes): add concrete L40 / L40S service-bound overlays #1003).
• Real gb200-oke-training perf measurements: blocked on feat(validation): add performance-phase constraints to OKE overlays once OCI testbed lands #1007 OCI testbed.

Testing

# Floor test
go test -count=1 -run TestOverlayValidationPhaseFloor ./pkg/recipe/
# ok  	github.com/NVIDIA/aicr/pkg/recipe	0.738s

# Acceptance: hydrate the OKE GB200 training recipe
go run ./cmd/aicr recipe --service oke --accelerator gb200 --intent training --format yaml
# resolves: 11 components, 6 overlays
# (base, monitoring-hpa, gb200-any, oke, oke-training, gb200-oke-training)
# deployment phase carries Deployment.gpu-operator.version >= v25.10.0 via gb200-any.yaml
# performance phase absent (warn-only floor gap — see Implementation Notes)

# Full gate
unset GITLAB_TOKEN && make qualify
# 22 chainsaw tests pass; no vulnerabilities; license check clean.

Risk Assessment

• Low — Single overlay deletion plus doc refreshes. Behavior delta is the loss of an inherited NCCL constraint on three OKE GB200 training leaves, which today is a warn-only floor recommendation (no CI failure, no production validator change). EKS GB200 leaves are unaffected (carry their own perf blocks). Inference recipes are unaffected (the wildcard was intent: training). Trivial to revert (one file restore + comment rollbacks).

Rollout notes: No flag, no opt-in. The next aicr recipe --service oke --accelerator gb200 --intent training hydration produces a recipe without the cross-service NCCL constraint; everything else is identical. When #1007 closes with real OCI numbers, gb200-oke-training.yaml ships its own performance block and the warn drains.

Checklist

• Tests pass locally (make test with -race) — via make qualify
• Linter passes (make lint) — via make qualify
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality — N/A (deletion + doc refresh; floor-test header comment refreshed)
• I updated docs if user-facing behavior changed — three doc pages refreshed
• Changes follow existing patterns in the codebase — mirrors the per-leaf perf model the EKS GB200 leaves already use
• Commits are cryptographically signed (git commit -S)