Skip to content

Integrate k8s-launch-kit profiles as a recipe Component generator #829

Description

@mchmarny

Part of #827.

Goal

aicr bundle produces a complete network-operator deployment whose Helm values and post-install CRs (NicClusterPolicy, SriovNetworkNodePolicy, IPPool, …) are derived from recipe inputs (fabric, deployment type) plus snapshot topology — by consuming k8s-launch-kit (l8k) profiles as a Go library.

Background

network-operator is already a registered Helm component in AICR (recipes/registry.yaml:108-124, recipes/components/network-operator/values.yaml, chart nvidia/[email protected]). Today its values are static; users hand-author the per-fabric / per-deployment-type CRs.

l8k encodes that per-profile knowledge as templated CR manifests:

  • Deployment types: sriov, rdma_shared, host_device
  • Fabrics: infiniband, ethernet
  • Special: spectrum-x, spectrum-x-ra2.1
  • Profile templates live in profiles/<name>/{10-nicclusterpolicy,11-nicnodepolicy,20-ippool,30-sriovnetworknodepolicy,40-sriovnetwork}.yaml and are rendered by (*NetworkOperatorPlugin).GenerateProfileDeploymentFiles.

Two structural problems

  1. No code-level Component hook in AICR. pkg/component exists but holds the declarative ComponentConfig struct + MakeBundle helper — it is not an interface for programmatic value computation. Bundler value flow (pkg/bundler/bundler.go:470-518) is: base recipe values → --set overrides → node-selector injection. There is no extension point where a component can compute values from snapshot data.
  2. Output shape mismatch. l8k's GenerateProfileDeploymentFiles returns map[filename]rendered-CR-YAML — these are CRs that assume the network-operator chart is already installed. AICR bundles produce Helm values.yaml. These don't merge.
  3. Snapshot data does not reach the bundler. bundler.Make(ctx, recipeResult, dir) — the snapshot is consumed earlier, in recipe resolution (metadata_store.go:752), only for constraint validation.

Options

Option A — Hybrid bundle (recommended for v1)

  • Keep network-operator as the existing Helm component with static values.yaml.

  • Add a new code-level extension interface in AICR (working name: pkg/bundler/plugin.Plugin):

    type Plugin interface {
        Name() string
        Generate(ctx context.Context, in PluginInput) (PluginOutput, error)
    }
    type PluginInput struct {
        Recipe   *recipe.RecipeResult
        Snapshot *measurement.Snapshot   // optional, may be nil
        Overrides map[string]any          // user `--set` values
    }
    type PluginOutput struct {
        Manifests map[string][]byte // filename → rendered manifest
    }
  • Implement network-operator-profile plugin that imports l8k as a library and calls GenerateProfileDeploymentFiles(profile, cfg). Output goes into the bundle as post-install resources (e.g., bundles/network-operator/post-install/*.yaml).

  • Bundler ordering: network-operator chart first, then network-operator-profile CRs. Use the existing DependencyRefs topo-sort (metadata.go:721-766); for ArgoCD deployer, emit sync-wave annotations.

  • Pros: minimal upstream l8k churn; reuses l8k's profile output unchanged; introduces one well-scoped extension interface AICR can grow into.

  • Cons: real design work in AICR (new interface, snapshot threading through bundler, ordering for non-Helm-managed CRs). Two-resource-class bundle (chart + bare manifests) is harder to roll back than a single Helm release.

Option B — Helm values translation

Translate LaunchKubernetesConfig directly into the network-operator chart's values.yaml schema. One Helm release, no post-install drift.

  • Pros: cleanest deployment story; fits existing AICR value-override flow.
  • Cons: the nvidia/network-operator chart's values schema does not fully expose SriovNetworkNodePolicy / NicNodePolicy / IPPool shapes — likely impossible without upstream chart changes. Translation logic doesn't exist in l8k today and is not their roadmap.

Option C — Full Kustomize component

Register network-operator-profile as a Kustomize component pointing at l8k-rendered manifests. AICR already supports Kustomize.

  • Pros: uses an existing component type.
  • Cons: still needs the snapshot→profile-input plumbing; needs a build step that runs l8k between recipe resolution and bundling; doesn't address the no-Component-interface gap.

Recommended: Option A.

Work breakdown (Option A)

  1. Design the plugin interface. Decide where it lives (pkg/bundler/plugin is the leading candidate — pkg/component is taken). Decide the input shape and whether snapshot is required or optional.
  2. Thread snapshot to the bundler. bundler.Make signature changes (or add a WithSnapshot option). Audit all callers.
  3. Map criteria → l8k profile selection. Two sub-decisions:
    • Are fabric and network (deployment type) new top-level recipe criteria fields, or component-config under network-operator's registry entry?
    • How much can be derived automatically from intent + snapshot vs. requiring explicit user input? E.g., intent: training + Mellanox CX-7 IB NICs in snapshot → default to fabric: infiniband, network: sriov?
  4. Implement network-operator-profile plugin under pkg/bundler/plugin/networkoperator (vendor l8k). Render the chosen profile via GenerateProfileDeploymentFiles.
  5. Bundle artifacts: decide on directory layout (bundles/<component>/post-install/*.yaml) and how the deployer (Helm script vs. ArgoCD) consumes them.
  6. Deployment ordering: use DependencyRefs to express "profile CRs depend on network-operator". For ArgoCD, emit sync-wave annotations (chart wave 1, CRs wave 2).
  7. RA1/RA2 / Spectrum-X gating: today l8k's ApplyNetworkOperatorRelease enforces release-line/profile compatibility. Decide whether AICR replicates this in recipe overlay constraints or delegates to a runtime check inside the plugin.
  8. Docs: component-catalog.md (network-operator-profile entry), recipe-development.md (new criteria fields if added), data.md (criteria reference).
  9. Tests: KWOK e2e covering (fabric × deployment-type) matrix; bundler unit tests for plugin invocation; values/manifest fixture tests for each profile.

Open questions

  • New criteria fields vs. component config: fabric: infiniband|ethernet and network: sriov|rdma-shared|host-device look like first-class criteria (they affect overlay/mixin selection), but they're network-operator-specific. Where do they belong?
  • Auto-derivation policy: how much should AICR derive vs. require explicit input? Per the original brief: "Deployment type (sriov / rdma shared / …) — think how to derive from user's intent + cluster capabilities".
  • Plugin interface scope: if we introduce pkg/bundler/plugin.Plugin for one use case, do we anticipate more (e.g., GPU-operator-profile)? Generalize now or stay narrow?
  • Single Helm release vs. hybrid: is there appetite to push Option B's chart-values translation upstream over time, or is hybrid the long-term answer?
  • Module path / pinning: same questions as #SUB1 — l8k has no v1.0.0; vendor by SHA?

Acceptance criteria

  • User can run aicr bundle -r recipe.yaml --set networkoperator:profile=sriov-ib-rdma -o ./bundles (or equivalent) and get both the Helm values and the rendered profile CRs in the output bundle.
  • Plugin reads snapshot when present (e.g., to populate subnet/MTU/PF list); falls back to defaults when not.
  • Deployment order is enforceable via Helm install scripts and ArgoCD sync waves.
  • All (fabric × deployment-type) profile combinations have KWOK e2e coverage.
  • make qualify passes; coverage on pkg/bundler/plugin/networkoperator ≥ 70%.
  • Documentation updated as listed above.

Metadata

Metadata

Assignees

Fields

No fields configured for Enhancement.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions