Summary

Bring real-cluster inference to CUJ1 parity (DC3): a phase_serve step deploys a Dynamo DynamoGraphDeployment, hits its OpenAI-compatible endpoint, and asserts a completion; and the nightly batch now runs both training and inference on both clouds, serialized through the shared reservation lease.

Motivation / Context

Inference was only a --no-cluster dry-run that no workflow invoked — no inference column existed on any cloud. This makes inference a real cluster phase, the counterpart of phase_train, and wires it end-to-end so nightly runs exercise a served endpoint and emit a signed bundle whose criteria map to inference-dynamo.

Fixes: #1276
Related: #1596 (follow-up: surface the build commit on the corroboration dashboard)

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Refactoring (no functional changes)
• Build/CI/tooling

Component(s) Affected

• CLI (cmd/aicr, pkg/cli)
• API server (cmd/aicrd, pkg/server)
• Recipe engine / data (pkg/recipe)
• Bundlers (pkg/bundler, pkg/component/*)
• Collectors / snapshotter (pkg/collector, pkg/snapshotter)
• Validator (pkg/validator)
• Core libraries (pkg/errors, pkg/k8s)
• Docs/examples (docs/, examples/)
• Other: UAT runner + workflows (tests/uat/**, .github/workflows/uat-*), reservation broker (pkg/uatbroker, tools/uat-broker)

Implementation Notes

• phase_serve (tests/uat/{aws,gcp}/run): deploys the KAI queue + DRA ResourceClaim + Frontend/decode-Worker DynamoGraphDeployment (inlined from demos/workloads/inference/vllm-agg.yaml), waits for pod readiness, port-forwards the frontend, POST /v1/chat/completions, asserts a non-empty completion. Fails closed (captured pod logs/events under serve-logs/) on a non-ready deployment or invalid completion, mirroring phase_train. Node scheduling/model are SERVE_*-overridable; tolerations are a portable superset across both clouds.
• Intent selection: run all picks train vs serve from the config's recipe intent; the workflows add an intent-gated UAT - serve step mirroring train.
• Platform cross-check in phase_conformance: the declared platform must match the deployed workload CRD (dynamographdeployments.nvidia.com / trainjobs.trainer.kubeflow.org), since the TestGrid coordinate is author-declared and otherwise cluster-unverifiable.
• Nightly cadence decision: both intents run nightly on both clouds, folded into the version matrix as serialized cells (version outer-loop / intent inner-loop) through the shared per-reservation lease — no second cron, no contention, and main lands both intents before any release cell. Data-driven via a new nightly-intents list in infra/uat/reservations.yaml (pkg/uatbroker). Trade-off: this ~doubles a reservation's cell count; the batch time-box drops oldest cells first (tune previous_n / deadline_offset_hours), documented in docs/contributor/uat.md.
• Version stamping: the main-cell binary is stamped version=main + commit + date via ldflags so evidence/results read main instead of the un-stamped dev. Because a real commit would otherwise redirect validator images :latest → :sha-<commit> (which may not exist for a feature-branch ref), main-cell validator images are pinned to the run's own :uat-<run_id> build — mirroring the existing agent pin. Release cells are unaffected (empty override → released :vX.Y.Z).

Testing

# Targeted gates (Go changes are scoped to the broker):
go test -race ./pkg/uatbroker/... ./tools/uat-broker/...   # ok, 96.9% / 91.5%
golangci-lint run -c .golangci.yaml ./pkg/uatbroker/... ./tools/uat-broker/...  # 0 issues
yamllint -c .yamllint.yaml .github/workflows/uat-*.yaml infra/uat/reservations.yaml  # clean
shellcheck tests/uat/aws/run tests/uat/gcp/run   # clean
# ldflags verified: `aicr --version` -> "main (commit: <sha>, date: ...)"

Full make qualify (e2e/grype) was not run — its e2e leg provisions real infrastructure. Per the issue's test plan, the real-hardware inference run is a manual workflow_dispatch intent=inference during bring-up; the --no-cluster chainsaw dry-run continues to validate the inference AICRConfig resolves.

Risk Assessment

• Low — Isolated change, well-tested, easy to revert
• Medium — Touches multiple components or has broader impact
• High — Breaking change, affects critical paths, or complex rollout

Rollout notes: No breaking changes. One behavior change to flag: main-cell validator images now resolve to the run's :uat-<run_id> build instead of :latest (consistent with the agent pin; removes known :latest staleness). nightly-intents defaults to [training] when absent, so the change is backward-compatible. The nightly cell count roughly doubles — see the time-box tuning note in docs/contributor/uat.md.

Checklist

• Tests pass locally (make test with -race) — ran -race on the changed Go packages (pkg/uatbroker, tools/uat-broker)
• Linter passes (make lint) — golangci-lint (affected pkgs, 0 issues) + yamllint + shellcheck
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality
• I updated docs if user-facing behavior changed
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)