Summary

Cluster-side validation of the freshly-landed BCM service overlay (#feat/bcm-service-type merged to main) on a krusty test cluster with a real H200 NVL node surfaced a small set of recipe-layer gaps spanning shared component values, BCM overlays, and the criteria registry. None block usage today — aicr validate deployment+conformance phases pass end-to-end on real Hopper hardware — but they're worth tightening before BCM is called production-ready and before users with H200 hardware start asking why the recipe metadata says accelerator: h100.

Context

Validated against:

• aicr recipe --service bcm --accelerator h100 --os ubuntu --intent training (passing h100 because there's no h200 criteria value yet — see gap [Feature]: Configure copy-pr-bot #4 below)
• BCM-provisioned k8s 1.34.8, Ubuntu 24.04, kernel 6.8.0-71
• 2× NVIDIA H200 NVL (141GB HBM3e each) on node007
• gpu-operator v26.3.1, nvidia-dra-driver-gpu 25.12.0
• Driver: containerized (GPU operator's nvidia-driver-daemonset running)
• DRA ResourceSlice publishes both H200s correctly with the right capacity/arch attributes

Architectural decision

Keep the BCM service overlay containerized-driver-only. Earlier validate failures from the same cluster (DRA pod stuck Init:0/1, empty /run/nvidia/driver) happened because the BCM base image shipped a pre-installed driver while the AICR overlay assumes the containerized path. The right fix is to strip the driver from the BCM image (in progress with the BCM team), not to add a host-installed-driver toggle in the overlay. One mode, fewer footguns.

Gaps to fix

• Shared DRA component — recipes/components/nvidia-dra-driver-gpu/values.yaml:74,78: controller.priorityClassName: "" and kubeletPlugin.priorityClassName: "" neutralize the chart default (system-node-critical) without explanation. Either add an inline comment with the rationale, or restore the chart default. Neutralizing system-node-critical lets DRA pods be evicted under node pressure, which would break GPU allocation for running pods.

• BCM overlay — recipes/overlays/bcm.yaml: the BCM-master toleration is added to controller.tolerations only. Mirror onto kubeletPlugin.tolerations so DRA's kubelet plugin still schedules if BCM workers ever inherit the node-role.kubernetes.io/master taint (small BCM deployments combining control-plane+worker roles).

• BCM training overlay — recipes/overlays/bcm-training.yaml: consider gds.enabled: true for the gpu-operator componentRef. H200 NVL (141GB HBM3e) with BCM's typical local-NVMe layout is the strongest case for GPUDirect Storage; leaving it off forgoes a meaningful training-perf win.

• Criteria registry — H200 — pkg/recipe/criteria.go:116-118: defines h100, gb200, b200 as Hopper/Blackwell accelerator types but h200 is missing. Users with H200 hardware currently have to pass --accelerator h100, which works (same Hopper arch, same R570/R580 driver line, NVML auto-detects everything) but produces misleading recipe metadata. No new overlay is needed today — h200 resolves to the same hydrated values as h100 via shared base components. The scope is purely:

  • Add CriteriaAcceleratorH200 CriteriaAcceleratorType = "h200" and the parse case
  • Add to AllCriteriaAcceleratorTypes() at pkg/recipe/criteria.go:155
  • Convert existing OriginExternal test registrations in pkg/recipe/criteria_registry_parse_test.go:105,108,113 to verify the built-in path
  • Per .claude/CLAUDE.md "Adding a new enum value" rule, audit and update every surface that enumerates accelerator types: api/aicr/v1/server.yaml (every enum: block); doc pages docs/README.md, docs/user/cli-reference.md, docs/user/api-reference.md, docs/contributor/api-server.md, docs/contributor/cli.md, docs/contributor/data.md, docs/contributor/validations.md; package godoc in pkg/recipe/doc.go and field comments; urfave/cli Description/Usage strings in pkg/cli/recipe.go; .github/ISSUE_TEMPLATE/*.yml dropdowns.

  Add an h200-specific overlay file only when there's a real tuning delta (e.g., H200-specific MIG profiles, NVL-form-factor topology config).

Out of scope (filed separately)

• Add integration test: nvidiaDriverRoot and gpu-operator.hostPaths.driverInstallDir must stay in lockstep across overlays #1087 — Add integration test asserting nvidiaDriverRoot and gpu-operator.hostPaths.driverInstallDir stay in lockstep across overlays