Skip to content

fix(ci): make GPU training and inference symmetric#1

Closed
yuanchen8911 wants to merge 1 commit into
codex/issue-554-gpu-conformance-dedupfrom
codex/gpu-ci-symmetry
Closed

fix(ci): make GPU training and inference symmetric#1
yuanchen8911 wants to merge 1 commit into
codex/issue-554-gpu-conformance-dedupfrom
codex/gpu-ci-symmetry

Conversation

@yuanchen8911

@yuanchen8911 yuanchen8911 commented Apr 15, 2026

Copy link
Copy Markdown
Owner

Summary

Dependency: This PR is intentionally stacked on top of NVIDIA#577 and should be reviewed/merged after NVIDIA#577.

Make the H100 training and inference GPU workflows symmetric by moving training to a kind+Kubeflow recipe, moving inference to H100 x2, and aligning their shared conformance coverage while keeping the inference smoke-test tail.

Motivation / Context

This is the follow-up to the GPU conformance dedup work in NVIDIA#577.
It removes the remaining drift between training and inference GPU CI so both workflows exercise the same core conformance checks, with only controller/gateway coverage and the inference smoke test differing.

Fixes: N/A
Related: NVIDIA#554, NVIDIA#577

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • Build/CI/tooling

Component(s) Affected

  • CLI (cmd/aicr, pkg/cli)
  • API server (cmd/aicrd, pkg/api, pkg/server)
  • Recipe engine / data (pkg/recipe)
  • Bundlers (pkg/bundler, pkg/component/*)
  • Collectors / snapshotter (pkg/collector, pkg/snapshotter)
  • Validator (pkg/validator)
  • Core libraries (pkg/errors, pkg/k8s)
  • Docs/examples (docs/, examples/)
  • Other: CI workflows and chainsaw coverage

Implementation Notes

  • Adds recipes/overlays/h100-kind-training-kubeflow.yaml and updates the training workflow to install platform: kubeflow.
  • Extends training conformance coverage with robust-controller and secure-accelerator-access, and adds Kubeflow chainsaw/assert coverage for the trainer controller, webhook, and TrainJob CRD.
  • Moves inference to H100 x2, adds gang-scheduling to the kind inference recipes, and removes dead deployment phase plumbing from the inference workflow.
  • Aligns timeout budgets and path filters so the remaining workflow differences are the deployed platform-specific components and the inference smoke-test tail.
  • This PR is intentionally stacked on top of fix(ci): deduplicate conformance coverage in GPU CI NVIDIA/aicr#577; it should be retargeted to main after fix(ci): deduplicate conformance coverage in GPU CI NVIDIA/aicr#577 merges.

Testing

go test ./pkg/recipe/...
git diff --check -- .github/workflows/gpu-h100-training-test.yaml .github/workflows/gpu-h100-inference-test.yaml pkg/recipe/conformance_test.go pkg/recipe/deployment_order_guard_test.go recipes/overlays/h100-kind-training.yaml recipes/overlays/h100-kind-training-kubeflow.yaml recipes/overlays/h100-kind-inference.yaml recipes/overlays/h100-kind-inference-dynamo.yaml tests/chainsaw/ai-conformance/cluster/assert-kubeflow-trainer.yaml tests/chainsaw/ai-conformance/kind-training-kubeflow
make lint
  • go test ./pkg/recipe/... passed.
  • git diff --check -- ... passed.
  • make lint was green on this branch before the local Homebrew Go toolchain upgrade, but current reruns are blocked locally by a stuck parallel golangci-lint lock after that environment change.
  • make qualify is not reliable locally right now because the upgraded Homebrew Go 1.26.2 install has a separate stdlib resolution issue (go list sync fails).

Risk Assessment

  • Medium — Touches multiple components or has broader impact
  • High — Breaking change, affects critical paths, or complex rollout
  • Low — Isolated change, well-tested, easy to revert

Rollout notes: This changes the inference GPU job to require H100 x2 and assumes the corresponding runner class remains available. Retarget the PR to main after NVIDIA#577 merges.

Checklist

  • Tests pass locally (make test with -race)
  • Linter passes (make lint)
  • I did not skip/disable tests to make CI green
  • I added/updated tests for new functionality
  • I updated docs if user-facing behavior changed
  • Changes follow existing patterns in the codebase
  • Commits are cryptographically signed (git commit -S) — GPG signing info

@github-actions

Copy link
Copy Markdown

Welcome to AICR, @yuanchen8911! Thanks for your first pull request.

Before review, please ensure:

  • All commits are signed off per the DCO
  • CI checks pass (tests, lint, security scan)
  • The PR description explains the why behind your changes

A maintainer will review this soon.

yuanchen8911 added a commit that referenced this pull request May 14, 2026
…outing

kgateway v2.2 removed InferencePool routing from its Envoy data plane
(PR kgateway-dev/kgateway#12689, deprecated in v2.1). The Gateway API
Inference Extension support that AICR uses for CNCF AI Conformance
Advanced Ingress moved entirely to the separate agentgateway project,
which ships its own Helm charts, GatewayClass, controller, and
AgentgatewayParameters CRD.

AICR's only kgateway consumer was the inference Gateway resource — zero
HTTPRoutes, no TrafficPolicy/BackendConfigPolicy/etc. — so this PR
replaces kgateway entirely rather than running both side-by-side.

Changes:
- registry + mixin: kgateway/kgateway-crds (v2.0.0, cr.kgateway.dev)
  -> agentgateway/agentgateway-crds (v2.2.1, cr.agentgateway.dev)
- inference-gateway.yaml: GatewayParameters (gateway.kgateway.dev) ->
  AgentgatewayParameters (agentgateway.dev/v1alpha1, strategic-merge
  patch on Deployment spec); gatewayClassName: kgateway -> agentgateway;
  namespace kgateway-system -> agentgateway-system
- health check + component dirs renamed under git mv to preserve history
- conformance validator (validators/conformance/inference_gateway_check.go)
  updated to gate on `agentgateway` component and query the new
  GatewayClass / namespace / Deployment names
- evidence collector script + requirement title updated
- GPU H100 inference workflow path filter follows the renamed component
  directories so PRs touching only agentgateway* still trigger CI
- Go test fixtures, undeploy.sh.tmpl + golden files, chainsaw assertions,
  UAT recipe snapshots regenerated against the new component graph
- docs/user (component-catalog, container-images, api-reference,
  cli-reference) refreshed; container-images.md regenerated via
  `make bom-docs`
- demos/cuj2-* + demos/query.md + demos/images/meta.md updated so
  kubectl commands and architecture diagrams match the new namespace
  and component names

Conformance-equivalence: AICR's `ai_inference` requirement is data-plane
agnostic — it asserts a GatewayClass is Accepted, a Gateway is
Programmed, and the InferencePool CRDs exist. Swapping to agentgateway
preserves all five evidence checks; only the names in the captured
output change.

Historical evidence snapshots under docs/conformance/cncf/v1.35/nim-eks/,
demos/examples/CUJ2-Test-Report.md, design doc 005, and CHANGELOG
entries are intentionally left referencing kgateway as frozen records.

Closes follow-up #1 from issue NVIDIA#698.

Validation:
- make qualify (Go unit tests, golangci-lint, yamllint, 20/20 chainsaw
  including cli-bundle-agentgateway-templates, vulnerability scan,
  license headers): all green
- Bundle generation verified end-to-end (recipe -> bundle -> deploy.sh
  layout under /tmp); rendered post chart confirmed to emit
  AgentgatewayParameters + Gateway with class agentgateway in namespace
  agentgateway-system
- validators/conformance package tests pass (22.6s)
yuanchen8911 added a commit that referenced this pull request Jun 9, 2026
Deployment-phase health-check validation regressed in the recent
in-process executor + deepened-check stack (NVIDIA#1235/NVIDIA#1245/NVIDIA#1252):

1. inprocess.go: runAssertWithRetry/runErrorWithRetry retried ANY
   non-nil error, so a permanent JMESPath evaluation error was retried
   for the entire assert window (6m) instead of failing fast like the
   old chainsaw binary did. Classify assertion-engine errors as
   ErrCodeInvalidRequest (terminal) and short-circuit the retry loops
   on them. Adds a regression test asserting terminal eval errors
   fail fast (< AssertRetryInterval) rather than retrying to deadline.

2. recipes/checks/*: the (init)containerStatuses[?...] | length(@)
   projection throws 'invalid type for: <nil>' on any pod without
   (init) containers (the common case), feeding defect #1. Coalesce to
   an empty array across all 22 affected component health checks.

3. recipes/checks/dynamo-platform: validate-deployment-exists asserted
   a Deployment named 'dynamo-operator', but the chart prefixes it with
   the release name (dynamo-platform-dynamo-operator-controller-manager),
   so the assert never matched and retried to the deadline. Match by the
   stable app.kubernetes.io/name=dynamo-operator label instead.

Verified on an EKS H100 cluster: deployment phase goes from an 8m
timeout (status=other) to PASSED in ~21s; full deployment 4/4 and
conformance 11/11 green.
yuanchen8911 added a commit that referenced this pull request Jun 9, 2026
Deployment-phase health-check validation regressed in the recent
in-process executor + deepened-check stack (NVIDIA#1235/NVIDIA#1245/NVIDIA#1252):

1. inprocess.go: runAssertWithRetry/runErrorWithRetry retried ANY
   non-nil error, so a permanent JMESPath evaluation error was retried
   for the entire assert window (6m) instead of failing fast like the
   old chainsaw binary did. Classify assertion-engine errors as
   ErrCodeInvalidRequest (terminal) and short-circuit the retry loops
   on them. Adds a regression test asserting terminal eval errors
   fail fast (< AssertRetryInterval) rather than retrying to deadline.

2. recipes/checks/*: the (init)containerStatuses[?...] | length(@)
   projection throws 'invalid type for: <nil>' on any pod without
   (init) containers (the common case), feeding defect #1. Coalesce to
   an empty array across all 22 affected component health checks.

3. recipes/checks/dynamo-platform: validate-deployment-exists asserted
   a Deployment named 'dynamo-operator', but the chart prefixes it with
   the release name (dynamo-platform-dynamo-operator-controller-manager),
   so the assert never matched and retried to the deadline. Match by the
   stable app.kubernetes.io/name=dynamo-operator label instead.

Verified on an EKS H100 cluster: deployment phase goes from an 8m
timeout (status=other) to PASSED in ~21s; full deployment 4/4 and
conformance 11/11 green.
yuanchen8911 added a commit that referenced this pull request Jun 9, 2026
Deployment-phase health-check validation regressed in the recent
in-process executor + deepened-check stack (NVIDIA#1235/NVIDIA#1245/NVIDIA#1252):

1. inprocess.go: runAssertWithRetry/runErrorWithRetry retried ANY
   non-nil error, so a permanent JMESPath evaluation error was retried
   for the entire assert window (6m) instead of failing fast like the
   old chainsaw binary did. Classify assertion-engine errors as
   ErrCodeInvalidRequest (terminal) and short-circuit the retry loops
   on them. Adds a regression test asserting terminal eval errors
   fail fast (< AssertRetryInterval) rather than retrying to deadline.

2. recipes/checks/*: the (init)containerStatuses[?...] | length(@)
   projection throws 'invalid type for: <nil>' on any pod without
   (init) containers (the common case), feeding defect #1. Coalesce to
   an empty array across all 22 affected component health checks.

3. recipes/checks/dynamo-platform: validate-deployment-exists asserted
   a Deployment named 'dynamo-operator', but the chart prefixes it with
   the release name (dynamo-platform-dynamo-operator-controller-manager),
   so the assert never matched and retried to the deadline. Match by the
   stable app.kubernetes.io/name=dynamo-operator label instead.

Verified on an EKS H100 cluster: deployment phase goes from an 8m
timeout (status=other) to PASSED in ~21s; full deployment 4/4 and
conformance 11/11 green.
yuanchen8911 added a commit that referenced this pull request Jun 9, 2026
Deployment-phase health-check validation regressed in the recent
in-process executor + deepened-check stack (NVIDIA#1235/NVIDIA#1245/NVIDIA#1252):

1. inprocess.go: runAssertWithRetry/runErrorWithRetry retried ANY
   non-nil error, so a permanent JMESPath evaluation error was retried
   for the entire assert window (6m) instead of failing fast like the
   old chainsaw binary did. Classify assertion-engine errors as
   ErrCodeInvalidRequest (terminal) and short-circuit the retry loops
   on them. Adds a regression test asserting terminal eval errors
   fail fast (< AssertRetryInterval) rather than retrying to deadline.

2. recipes/checks/*: the (init)containerStatuses[?...] | length(@)
   projection throws 'invalid type for: <nil>' on any pod without
   (init) containers (the common case), feeding defect #1. Coalesce to
   an empty array across all 22 affected component health checks.

3. recipes/checks/dynamo-platform: validate-deployment-exists asserted
   a Deployment named 'dynamo-operator', but the chart prefixes it with
   the release name (dynamo-platform-dynamo-operator-controller-manager),
   so the assert never matched and retried to the deadline. Match by the
   stable app.kubernetes.io/name=dynamo-operator label instead.

Verified on an EKS H100 cluster: deployment phase goes from an 8m
timeout (status=other) to PASSED in ~21s; full deployment 4/4 and
conformance 11/11 green.
yuanchen8911 added a commit that referenced this pull request Jun 9, 2026
Deployment-phase health-check validation regressed in the recent
in-process executor + deepened-check stack (NVIDIA#1235/NVIDIA#1245/NVIDIA#1252):

1. inprocess.go: runAssertWithRetry/runErrorWithRetry retried ANY
   non-nil error, so a permanent JMESPath evaluation error was retried
   for the entire assert window (6m) instead of failing fast like the
   old chainsaw binary did. Classify assertion-engine errors as
   ErrCodeInvalidRequest (terminal) and short-circuit the retry loops
   on them. Adds a regression test asserting terminal eval errors
   fail fast (< AssertRetryInterval) rather than retrying to deadline.

2. recipes/checks/*: the (init)containerStatuses[?...] | length(@)
   projection throws 'invalid type for: <nil>' on any pod without
   (init) containers (the common case), feeding defect #1. Coalesce to
   an empty array across all 22 affected component health checks.

3. recipes/checks/dynamo-platform: validate-deployment-exists asserted
   a Deployment named 'dynamo-operator', but the chart prefixes it with
   the release name (dynamo-platform-dynamo-operator-controller-manager),
   so the assert never matched and retried to the deadline. Match by the
   stable app.kubernetes.io/name=dynamo-operator label instead.

Verified on an EKS H100 cluster: deployment phase goes from an 8m
timeout (status=other) to PASSED in ~21s; full deployment 4/4 and
conformance 11/11 green.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant