Skip to content

[Feature]: Add bcm as a service type with recipe overlays #1059

Description

@mchmarny

Feature Summary

Add NVIDIA Base Command Manager (BCM) as a first-class service value in AICR criteria (alongside eks, gke, aks, oke, kind, lke, etc.) and ship a recipe overlay tree so users can generate validated Helmfile bundles for BCM-managed Kubernetes clusters. Argo CD validation is a follow-up.

Problem / Use Case

BCM (Bright Cluster Manager, currently cm-setup-11.x) is NVIDIA's on-prem cluster manager. With BCM 11.x, cm-kubernetes-setup provisions only the base platform — Kubernetes control plane, CNI, DNS — and leaves the NVIDIA runtime stack (GPU Operator, Network Operator, Prometheus, NIM Operator, LWS, etc.) for the operator to deliver. Today AICR has no service: bcm, so:

  • Operators cannot generate AICR recipes targeted at the BCM topology (K8s 1.34+, containerd, tigera-operator CNI, master/worker labels, MetalLB ingress, local-path default StorageClass).
  • There is no canonical AICR → Helmfile path for delivering the NVIDIA runtime stack onto a freshly provisioned BCM cluster.
  • Conformance validation has no way to recognize BCM-specific topology.

Reference baseline (BCM 11.0 / cm-kubernetes-setup): Kubernetes 1.34, containerd, tigera-operator CNI, MetalLB, local-path default StorageClass, control-plane label node-role.kubernetes.io/master.

Proposed Solution

  1. Criteria enum — add to pkg/recipe/criteria.go:

    • CriteriaServiceBCM CriteriaServiceType = "bcm"
    • Wire into ParseCriteriaServiceType, GetCriteriaServiceTypes, and every doc/OpenAPI/issue-template enumeration
  2. Service-level overlayrecipes/overlays/bcm.yaml:

    • Inherits base, so the full AICR runtime stack (NFD, cert-manager, GPU Operator, nvsentinel, DRA driver, Prometheus stack, etc.) is delivered onto the BCM cluster.
    • Adds BCM topology constraints: K8s.server.version >= 1.34, containerd runtime, control-plane label node-role.kubernetes.io/master (BCM uses the legacy master label in addition to control-plane).
    • Component overrides where needed for BCM topology — e.g., add node-role.kubernetes.io/master to existing tolerations / nodeAffinity so control-plane workloads schedule on BCM masters; point kube-prometheus-stack storage at the local-path StorageClass.
    • Conformance checks: platform-health, gpu-operator-health, dra-support, accelerator-metrics, ai-service-metrics.
  3. Intent overlaysrecipes/overlays/bcm-training.yaml, bcm-inference.yaml, following the EKS pattern.

  4. Accelerator + OS leaf overlays — at least h100-bcm-ubuntu-training.yaml (and an underlying h100-bcm-training.yaml); reuse the existing os-ubuntu mixin.

  5. Deployment path — Helmfile (initial) — document the canonical flow:

    aicr bundle -r recipe.yaml \
      --deployer helmfile \
      -o ./bundles
    helmfile -f ./bundles/helmfile.yaml apply
    • Argo CD (--deployer argocd / argocd-helm) is a planned follow-up once the Helmfile path is validated against a real BCM cluster.
  6. Docs & registry — update docs/user/component-catalog.md, docs/user/cli-reference.md, docs/user/api-reference.md, docs/contributor/data.md, and the OpenAPI enum blocks in api/aicr/v1/server.yaml.

Success Criteria

  • aicr recipe --service bcm --accelerator h100 --intent training --os ubuntu resolves a recipe without errors.
  • aicr bundle -r recipe.yaml --deployer helmfile -o ./bundles emits a helmfile.yaml release graph that the operator can apply against a BCM-provisioned cluster to deliver the full NVIDIA runtime stack.
  • aicr validate -r recipe.yaml -s <bcm-snapshot>.yaml --no-cluster evaluates constraints against a snapshot captured from a BCM 11.x cluster.
  • BCM appears in CLI --service help, API enum, and component catalog docs.
  • make qualify passes; make bom-docs regenerated and committed.

Alternatives Considered

  • Treating BCM as service: any with a custom mixin — loses first-class discovery, breaks the criteria-based registry lookup, and forces users to know undocumented mixin names.
  • Registering via --data only — works for an internal trial but not for users consuming the embedded OSS catalog; BCM is a publicly documented NVIDIA product and belongs in the static enum.
  • Starting with Argo CD — deferred to a follow-up issue. Helmfile is the simplest path to validate the overlay end-to-end without depending on a pre-provisioned Argo CD instance.

Component

Recipe engine / data

Priority

Important (would improve my workflow)

Compatibility / Breaking Changes

Additive only — new enum value, new overlay files, no changes to existing service types or recipes.

Operational Considerations

  • BCM uses node-role.kubernetes.io/master for the control-plane label; component tolerations/affinity in the overlay should accept both master and control-plane for portability.
  • BCM ships local-path as the default StorageClass; persistent components (e.g., Prometheus) must declare storageClassName: local-path explicitly.
  • Need a snapshot fixture from a real BCM cluster for E2E / KWOK coverage.
  • Argo CD (--deployer argocd / argocd-helm) validation is out of scope for this issue and will be tracked separately.

Willing to contribute

Yes, I can open a PR

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions