Summary

Adds a Dynamo inference performance phase to the H100 AKS Ubuntu inference overlay, mirroring the current H100 EKS/GKE inference-perf smoke gate, and documents a known limitation of the existing performance gates: the inference throughput and NCCL bus-bandwidth floors are fixed absolute full-node values calibrated on 8-GPU H100 nodes, not normalized for GPU count / fabric class.

⚠️ Known limitation — the current performance gates assume a full 8-GPU H100 node. The inference-throughput >= 50000 floor (and the analogous NCCL nccl-all-reduce-bw busbw floors) was calibrated on a full 8-GPU H100 node (GB200 on a 4-GPU node) and is not normalized for GPU count or fabric class. Because the recipes are SKU-agnostic (service + accelerator only), the same recipe lands on smaller / different-fabric H100 SKUs (p5.4xlarge, a3-highgpu-1g/2g/4g, AKS NC80adis/NC40ads) where a healthy run can false-fail. This PR documents the 8-GPU happy path and the workaround (TTFT-only on smaller SKUs); the behavioral fixes are tracked in #1254 (inference, per-GPU normalization) and #1256 (NCCL, fabric/transport-class floors).

Motivation / Context

The AKS Dynamo inference leaf was missing the recommended performance phase. This adds the same inference throughput and TTFT goals used by the current H100 EKS/GKE Dynamo overlays.

While adding the AKS gate it became clear the shared >= 50000 throughput floor (and the analogous NCCL busbw floors) is a full-node value: on a smaller H100 SKU — which AKS commonly is (NC80adis = 2 GPUs, NC40ads = 1 GPU), and which exists on every provider (p5.4xlarge, a3-highgpu-1g/2g/4g) — it can false-fail a healthy run. Rather than ship that silently, this PR adds explicit NOTE comments to the affected overlays and a node-shape-assumption note to the validation docs, cross-referencing the normalization follow-ups.

Fixes: #1006
Related: #1254 (inference throughput per-GPU normalization), #1256 (NCCL fabric/transport-class floors)

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Refactoring (no functional changes)
• Build/CI/tooling

Component(s) Affected

• CLI (cmd/aicr, pkg/cli)
• API server (cmd/aicrd, pkg/server)
• Recipe engine / data (pkg/recipe)
• Bundlers (pkg/bundler, pkg/component/*)
• Collectors / snapshotter (pkg/collector, pkg/snapshotter)
• Validator (pkg/validator)
• Core libraries (pkg/errors, pkg/k8s)
• Docs/examples (docs/, examples/)
• Other: ____________

Implementation Notes

• Adds performance.checks: [inference-perf] to h100-aks-ubuntu-inference-dynamo with the same model, concurrency, throughput, and TTFT goals as the current H100 EKS/GKE Dynamo overlays: Qwen/Qwen3-8B, 256 concurrency per GPU, >= 50000 throughput, <= 2000 TTFT p99.
• Adds a resolved-recipe regression test so the AKS Dynamo leaf follows the inference pattern and does not pick up training NCCL checks.
• Documents the 8-GPU ND H100 supported path for AKS: docs/integrator/aks-gpu-setup.md now uses Standard_ND96isr_H100_v5 (8-GPU ND H100 v5) for the example GPU nodepool instead of the 2-GPU Standard_NC80adis_H100_v5, with a note that the performance gates are full-node floors and that smaller NCads SKUs should gate on inference-ttft-p99 only until inference-perf throughput floor is a fixed absolute full-node value applied to SKU-agnostic recipes → false-fails smaller node shapes #1254. The AKS overlay NOTE is framed the same way (assumes 8-GPU ND H100; NCads SKUs drop the throughput constraint / TTFT-only until inference-perf throughput floor is a fixed absolute full-node value applied to SKU-agnostic recipes → false-fails smaller node shapes #1254).
• Full-node-assumption notes (no behavior change):
  • docs/user/validation.md — adds a "Node-shape assumption" note to both the training (NCCL) and inference performance sections, explaining that the floors are fixed absolute full-node values and which SKUs can false-fail, linking inference-perf throughput floor is a fixed absolute full-node value applied to SKU-agnostic recipes → false-fails smaller node shapes #1254 / nccl-all-reduce-bw training gate is a fixed absolute fabric-specific busbw value applied to SKU-agnostic recipes → false-fails EKS/H100 small SKUs #1256. (Uses the file's bold-label-paragraph convention, matching **Warm-up:** / **Determinism:**.)
  • Inference Dynamo overlays (h100-aks, h100-eks, h100-gke, gb200-eks, rtx-pro-6000-eks) — adds a NOTE on the throughput floor. Corrects the EKS/GKE comments that implied the evaluator scales across node sizes (it only down-scales for partial occupancy, not across node shapes).
  • Active H100 training overlays (h100-eks-training, h100-gke-cos-training) — adds a NOTE that the busbw floor is fabric-class dependent, not GPU-count scaled.
• inference-ttft-p99 is called out as a per-request latency at fixed concurrency-per-GPU that does not need GPU-count normalization.

Scope note: the AKS recipe criteria remain generic (service: aks, accelerator: h100), so a user on NC40ads/NC80adis can still generate/validate this recipe and hit the throughput false-fail. This PR documents the 8-GPU ND H100 happy path and the NCads workaround (TTFT-only); the behavioral fix (per-GPU normalization) is tracked in #1254 and intentionally out of scope here.

Testing

go test ./pkg/recipe -count=1
env AICR_VALIDATION_FLOOR_STRICT=1 go test ./pkg/recipe -run 'TestOverlayValidationPhaseFloor/h100-aks-ubuntu-inference-dynamo$|TestAKSDynamoInferencePerformanceGoalsFollowPattern' -count=1
yamllint <changed overlays>          # clean
golangci-lint run -c .golangci.yaml ./pkg/recipe/...
unset GITLAB_TOKEN && make qualify

The added overlay/doc changes are comments and prose only (no semantic recipe change), so resolution is unchanged — TestAKSDynamoInferencePerformanceGoalsFollowPattern and TestOverlayValidationPhaseFloor still pass, and yamllint is clean on all changed overlays. make qualify completes the full gate; the vulnerability scan reports the existing k8s.io/kubernetes advisories and does not fail the gate.

Risk Assessment

• Low — Isolated change, well-tested, easy to revert
• Medium — Touches multiple components or has broader impact
• High — Breaking change, affects critical paths, or complex rollout

Rollout notes: The AKS goals are provisional smoke-floor values mirrored from the current H100 EKS/GKE Dynamo overlays; recalibrate once Azure AKS H100 testbed data is available. The full-node-assumption notes are documentation only — the actual normalization fixes are tracked separately in #1254 (inference) and #1256 (NCCL).

Checklist

• Tests pass locally (make test with -race)
• Linter passes (make lint)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality
• I updated docs if user-facing behavior changed
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S) — GPG signing info