feat(recipes): update AKS H100 Dynamo recipe to match working cluster state by Jont828 · Pull Request #700 · NVIDIA/aicr

Jont828 · 2026-04-27T23:29:56Z

Summary

Updates the h100-aks-ubuntu-inference-dynamo recipe to reproduce the state of a working AKS ND-H100 cluster running Dynamo. Adds the full RDMA/InfiniBand stack and migrates Dynamo from v0.9 to v1.0.1.

Motivation / Context

The recipe was missing the RDMA stack required for GPUDirect on AKS ND-series nodes, had no version pins, and targeted Dynamo v0.9 (now v1.0.1 with a restructured values schema). This change makes aicr bundle produce a deployable configuration that matches a known-good cluster.

Fixes: N/A
Related: N/A

Type of Change

New feature (non-breaking change that adds functionality)

Component(s) Affected

Docs/examples (docs/, examples/)
Other: Recipes (recipes/)

Implementation Notes

RDMA stack (5 layers):

NFD NodeFeatureRule (nfd-network-rule.yaml, hook-weight 1) — labels ConnectX-7 nodes (PCI device IDs 101c/101e, vendor 15b3) with pci-15b3.present=true. Replaces the chart's generic Ethernet class rule (deployNodeFeatureRules: false in values-aks.yaml).
NicClusterPolicy (nic-cluster-policy-aks.yaml, hook-weight 5) — reconciled by network-operator to deploy: MOFED driver (doca3.2.0-25.10-1.2.8.0-2), RDMA shared device plugin (rdma/hca_shared_devices_a, rdmaHcaMax: 1000), DOCA telemetry. Folds the standalone rdma-shared-dp-ds DaemonSet that was manually deployed on the working cluster into the CR.
ib-node-config DaemonSet (ib-node-config-aks.yaml) — host-level: loads ib_umad/rdma_ucm kernel modules, sets LimitMEMLOCK=infinity on containerd/kubelet systemd units so container processes inherit unlimited memlock (no IPC_LOCK needed in pod specs). Node selector uses pci-15b3.present=true (portable) rather than the working cluster's pool-specific label. Includes a TODO to replace with Nodewright (formerly Skyhook) once AKS RDMA support lands upstream (NVIDIA/nodewright-packages#39).
driver.rdma.useHostMofed: true — tells GPU operator to use MOFED installed by network-operator instead of building nvidia-peermem from source. nfd.enabled: false prevents duplicate NFD DaemonSets.
Version pins — gpu-operator: v26.3.0, network-operator: v26.1.0, kube-prometheus-stack: 83.7.0 — match the working cluster.

Dynamo v0.9 → v1.0.1:

dynamo-crds componentRef removed — v1.0 platform chart manages CRDs itself.
dynamo-platform/values.yaml rewritten for v1.0 global.* subchart keys: grove.install: true (Dynamo manages grove), kai-scheduler.install: false / enabled: true (use AICR-managed external instance), etcd.install: false (k8s-native discovery).
nats.enabled: false removed — NATS is active in v1.0; operator injects natsAddress into DynamoGraphDeployment workloads.
Prometheus endpoint corrected: kube-prometheus-kube-prome-prometheus (derived from fullnameOverride: kube-prometheus in AICR's kube-prometheus-stack values).
NATS JetStream PVC override restored: storageClassName: managed-csi.
registry.yaml dynamo-platform defaultVersion: 0.9.1 → 1.0.1.

Testing

make test
make lint

Recipe compilation and bundle inspection against the working cluster confirmed correct values structure.

Risk Assessment

Low — Isolated change, well-tested, easy to revert

Rollout notes: AKS-only changes. Other cloud overlays (EKS, GKE) are untouched. Dynamo version bump only affects the AKS Dynamo overlay; dynamo-crds registry entry is unchanged for other overlays that may reference it.

Checklist

Tests pass locally (make test with -race)
Linter passes (make lint)
I did not skip/disable tests to make CI green
I added/updated tests for new functionality
I updated docs if user-facing behavior changed
Changes follow existing patterns in the codebase
Commits are cryptographically signed (git commit -S)

copy-pr-bot · 2026-04-27T23:30:00Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Jont828 · 2026-04-29T01:16:57Z

This branch has been rebased onto chore/bump-dynamo-platform-1.0.1 which handles the Dynamo version bump (0.9.x → 1.0.2), dynamo-crds removal, and grove component addition across all overlays. That PR should merge first so this one only carries the AKS-specific changes (RDMA stack, network-operator manifests, gpu-operator values).

Conflicts resolved during rebase:

recipes/components/dynamo-platform/values.yaml — took the base branch's structure (grove as separate AICR component with install: false / enabled: true, correct prometheus endpoint kube-prometheus-prometheus)
recipes/overlays/h100-aks-ubuntu-inference-dynamo.yaml — took the base branch's version (1.0.2, grove componentRef, no dead etcd/nats overrides)
recipes/registry.yaml — took 1.0.2 defaultVersion

github-actions · 2026-04-30T16:53:11Z

@Jont828 this PR now has merge conflicts with main. Please rebase to resolve them.

Signed-off-by: Jont828 <[email protected]>

Plan to update the h100-aks-ubuntu-inference-dynamo recipe to match the working AKS ND-H100 cluster state. Covers version bumps (GPU operator v26.3.0, network operator v26.1.0, dynamo v1.0.1), NicClusterPolicy manifest for MOFED + RDMA device plugin, NFD NodeFeatureRule for Mellanox IB detection, and dynamo values rewrite for the 1.0 schema. Signed-off-by: Jont828 <[email protected]>

… state Brings h100-aks-ubuntu-inference-dynamo in sync with a working AKS ND-H100 cluster running Dynamo. Changes span the full RDMA stack and Dynamo v0.9 → v1.0.1 migration. RDMA / InfiniBand (aks.yaml + network-operator manifests): - Add NFD NodeFeatureRule manifest (hook-weight 1): labels ConnectX-7 nodes (device IDs 101c/101e) with pci-15b3.present=true; replaces chart's generic Ethernet class rule (deployNodeFeatureRules: false) - Add NicClusterPolicy manifest (hook-weight 5): MOFED driver doca3.2.0-25.10-1.2.8.0-2, RDMA shared device plugin (hca_shared_devices_a, rdmaHcaMax 1000, vendor 15b3/mlx5_core), DOCA telemetry; folds standalone rdma-shared-dp-ds DaemonSet into CR - Add ib-node-config DaemonSet manifest: loads IB kernel modules and sets LimitMEMLOCK=infinity on containerd/kubelet systemd units; node selector changed from pool-specific label to pci-15b3.present for portability; TODO comment for Skyhook replacement - Add network-operator/values-aks.yaml with deployNodeFeatureRules: false - Version pins: gpu-operator v26.3.0, network-operator v26.1.0, kube-prometheus-stack 83.7.0 - gpu-operator/values-aks.yaml: add driver.rdma.useHostMofed: true (use MOFED from network-operator, skip peermem source build), nfd.enabled: false (avoid duplicate NFD DaemonSets) Dynamo v0.9 → v1.0.1: - Remove dynamo-crds componentRef (v1.0 platform manages CRDs itself) - Rewrite dynamo-platform/values.yaml for v1.0 global.* subchart keys: grove.install: true, kai-scheduler.install: false / enabled: true (use AICR-managed external instance), etcd.install: false - Remove nats.enabled: false (NATS is active in v1.0; operator injects natsAddress into DynamoGraphDeployments) - Fix Prometheus endpoint: kube-prometheus-kube-prome-prometheus (derived from fullnameOverride: kube-prometheus in AICR values) - Restore NATS JetStream PVC override: storageClassName: managed-csi - registry.yaml: dynamo-platform defaultVersion 0.9.1 → 1.0.1 Docs: replace implementation plan with post-implementation summary covering RDMA stack layers, Dynamo migration rationale, and version table. Signed-off-by: Jont828 <[email protected]>

Skyhook is being renamed to Nodewright. Nodewright is adding AKS support (NVIDIA/nodewright-packages#39) but RDMA/MEMLOCK setup is not yet included. Update the TODO comment to track the right upstream project. Signed-off-by: Jont828 <[email protected]>

Signed-off-by: Jont828 <[email protected]>

The AKS recipe unconditionally deploys the full RDMA/InfiniBand stack via network-operator. Users running H100s without InfiniBand can now opt out at bundle time with: --set networkoperator:enabled=false \ --set gpuoperator:driver.rdma.useHostMofed=false Add CheckHostMofedWithoutNetworkOperator validation that warns when network-operator is disabled but useHostMofed is not overridden. Add a ConfigMap kill switch to the ib-node-config DaemonSet so operators can disable host-level RDMA setup at runtime without redeploying.

The assert-recipe.yaml had grove listed twice in componentRefs, causing a slice length mismatch (18 expected vs 17 actual) in the E2E test.

mchmarny

/lgtm

The container-images BOM renders each chart at its registry defaultVersion, but at resolution that default is only a fallback: a componentRef version/tag pin in base.yaml, an overlay, or a mixin overrides it. So the BOM reflects what recipes install only when no pin diverges. A pin that drifts from the default (most dangerously on a component whose sole consumer diverges) makes the BOM advertise a version no recipe installs -- the #1418 aws-efa class of bug. Add TestOverlayVersionPinsMatchRegistry: every base/overlay/mixin componentRefs version/tag must equal the component's registry default unless declared in versionPinExemptions with a justification. Runs under make test -> make qualify. Fails on undeclared drift, stale exemptions, and vacuous discovery. Each exemption binds a single divergence via expectedPin/expectedDefault: if either the recipe pin or the registry default later moves, the guard fails so the divergence is re-reviewed rather than a new, unvetted version silently inheriting the old blessing. Fix the one live divergence it caught: base.yaml pinned nvsentinel v1.3.0 while the registry default (and BOM) advertised v1.9.0 -- every recipe shipped v1.3.0. Align base to v1.9.0 so reality matches the BOM. Record the intentional aks kube-prometheus-stack 83.7.0 divergence (#700) as an exemption. Document the single-source version model in docs/contributor/recipe.md. Fixes: #1424 Signed-off-by: Mark Chmarny <[email protected]>

The container-images BOM renders each chart at its registry defaultVersion, but at resolution that default is only a fallback: a componentRef version/tag pin in base.yaml, an overlay, or a mixin overrides it. So the BOM reflects what recipes install only when no pin diverges. A pin that drifts from the default (most dangerously on a component whose sole consumer diverges) makes the BOM advertise a version no recipe installs -- the #1418 aws-efa class of bug. Enforce both halves of #1424's acceptance: - TestOverlayVersionPinsMatchRegistry (pkg/recipe): every base/overlay/mixin componentRefs version/tag must equal the component's registry default unless declared in versionPinExemptions with a justification. Each exemption binds one divergence via expectedPin/expectedDefault, so a later move of either the pin or the default re-triggers review. Fails on undeclared drift, stale exemptions, and vacuous discovery. - TestCommittedBOMVersionsMatchRegistry (tools/bom): the committed docs/user/container-images.md version column must match the registry pins. This catches a coordinated bump (registry + pins moved together) that forgot make bom-docs -- which the recipe guard alone would pass. Version-only diff, no Helm rendering, runs under make test. Both run under make test -> make qualify. Fix the one live divergence caught: base.yaml pinned nvsentinel v1.3.0 while the registry default (and BOM) advertised v1.9.0 -- every recipe shipped v1.3.0. Align base to v1.9.0 so reality matches the BOM. Record the intentional aks kube-prometheus-stack 83.7.0 divergence (#700) as an exemption. Document the single-source version model and the two gates in docs/contributor/recipe.md. Fixes: #1424 Signed-off-by: Mark Chmarny <[email protected]>

The container-images BOM renders each chart at its registry defaultVersion, but at resolution that default is only a fallback: a componentRef version/tag pin in base.yaml, an overlay, or a mixin overrides it. So the BOM reflects what recipes install only when no pin diverges. A pin that drifts from the default (most dangerously on a component whose sole consumer diverges) makes the BOM advertise a version no recipe installs -- the #1418 aws-efa class of bug. Enforce both halves of #1424's acceptance: - TestOverlayVersionPinsMatchRegistry (pkg/recipe): every base/overlay/mixin componentRefs version/tag must equal the component's registry default unless declared in versionPinExemptions with a justification. Each exemption binds one divergence via expectedPin/expectedDefault, so a later move of either the pin or the default re-triggers review. It also resolves every overlay and fails if an exemption's registry default is installed by no recipe (the sole-consumer policy). Fails on undeclared drift, stale/unsafe exemptions, and vacuous discovery. - TestCommittedBOMVersionsMatchRegistry (tools/bom): the committed docs/user/container-images.md version column must match the registry pins. This catches a coordinated bump (registry + pins moved together) that forgot make bom-docs -- which the recipe guard alone would pass. Version-only diff, no Helm rendering, runs under make test. Both run under make test -> make qualify. Fix the one live divergence caught: base.yaml pinned nvsentinel v1.3.0 while the registry default (and BOM) advertised v1.9.0 -- every recipe shipped v1.3.0. Align base to v1.9.0 so reality matches the BOM. Record the intentional aks kube-prometheus-stack 83.7.0 divergence (#700) as an exemption. Realign stale pins in examples/recipes/{eks-training,aks-training, eks-gb200-ubuntu-training-with-validation}.yaml to resolved output (gpu-operator v26.3.2, nvsentinel v1.9.0, AKS kube-prometheus-stack 83.7.0), preserving their curated form. Document the single-source version model and the two gates in docs/contributor/recipe.md. Fixes: #1424 Signed-off-by: Mark Chmarny <[email protected]>

Jont828 requested review from a team as code owners April 27, 2026 23:29

github-actions Bot added area/recipes area/docs size/XL labels Apr 27, 2026

This comment was marked as resolved.

Sign in to view

mchmarny assigned Jont828 Apr 27, 2026

ayuskauskas reviewed Apr 28, 2026

View reviewed changes

Comment thread recipes/components/dynamo-platform/values.yaml Outdated

Jont828 force-pushed the jont828/aks-h100-dynamo-recipe-update branch from 9617e29 to 539d299 Compare April 29, 2026 00:16

github-actions Bot added size/L and removed area/docs size/XL labels Apr 29, 2026

This comment was marked as resolved.

Sign in to view

Jont828 force-pushed the jont828/aks-h100-dynamo-recipe-update branch from 539d299 to 9bf84c2 Compare April 29, 2026 01:16

github-actions Bot added area/ci area/tests area/docs area/bundler labels Apr 29, 2026

github-actions Bot added size/XL and removed size/L labels Apr 29, 2026

This comment was marked as resolved.

Sign in to view

github-actions Bot added the needs-rebase label Apr 30, 2026

Jont828 added 3 commits April 30, 2026 16:39

chore(recipe): bump dynamo-platform from 0.9.x to 1.0.2

8ee0a8f

Signed-off-by: Jont828 <[email protected]>

Jont828 added 4 commits April 30, 2026 16:39

chore: remove implementation plan doc from tracked files

d2a4e14

Signed-off-by: Jont828 <[email protected]>

Coderabbit fixes

87899d0

Signed-off-by: Jont828 <[email protected]>

Jont828 force-pushed the jont828/aks-h100-dynamo-recipe-update branch from fc54e87 to e5a85bb Compare April 30, 2026 20:39

github-actions Bot removed area/ci area/docs labels Apr 30, 2026

fix(tests): remove duplicate grove entry in ai-conformance assertion

2482c8a

The assert-recipe.yaml had grove listed twice in componentRefs, causing a slice length mismatch (18 expected vs 17 actual) in the E2E test.

ayuskauskas approved these changes May 1, 2026

View reviewed changes

Merge branch 'main' into jont828/aks-h100-dynamo-recipe-update

387b1bc

mchmarny enabled auto-merge (squash) May 4, 2026 22:57

mchmarny approved these changes May 4, 2026

View reviewed changes

mchmarny merged commit 4e602f2 into NVIDIA:main May 4, 2026
79 checks passed

Jont828 mentioned this pull request May 5, 2026

fix: AKS H100 RDMA sets network operator as dependency and fix chart values/metrics #738

Merged

23 tasks

coderabbitai Bot mentioned this pull request Jun 11, 2026

recipes/dynamo: bump platform to 1.2 #1308

Merged

25 tasks

mchmarny mentioned this pull request Jul 1, 2026

fix(recipes): guard overlay version pins against registry defaults #1572

Merged

12 tasks

yuanchen8911 mentioned this pull request Jul 6, 2026

chore(recipes): represent explicit per-source version variants in the BOM #1611

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(recipes): update AKS H100 Dynamo recipe to match working cluster state#700

feat(recipes): update AKS H100 Dynamo recipe to match working cluster state#700
mchmarny merged 9 commits into
NVIDIA:mainfrom
Jont828:jont828/aks-h100-dynamo-recipe-update

Jont828 commented Apr 27, 2026

Uh oh!

copy-pr-bot Bot commented Apr 27, 2026

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Jont828 commented Apr 29, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

mchmarny left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Jont828 commented Apr 27, 2026

Summary

Motivation / Context

Type of Change

Component(s) Affected

Implementation Notes

Testing

Risk Assessment

Checklist

Uh oh!

copy-pr-bot Bot commented Apr 27, 2026

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Jont828 commented Apr 29, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

mchmarny left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants