Parent: #1264 · Labels: enhancement, area/tests (auto), area/ci
(auto), area/docs (auto) · Size: L · Depends on: DC2

Goal. Bring real-cluster inference to CUJ1 parity: deploy a Dynamo
DynamoGraphDeployment, hit the OpenAI-compatible endpoint, capture conformance
evidence, and emit a signed bundle — the inference counterpart of the existing
phase_train. DC3 is the gate on the inference-dynamo half of the launch scope:
no inference column exists on any cloud until DC3 lands.

Scope.

• Add phase_serve to tests/uat/{aws,gcp}/run, parallel to phase_train
  (tests/uat/aws/run:193-256): apply the Dynamo deployment, wait for readiness,
  issue a sample OpenAI-compatible request, assert a valid completion, capture
  logs. Select it via the test config / intent, so run all runs
  prep → install → conformance → serve → verify for inference and
  …→ train →… for training. The committed cuj2-inference/chainsaw-test.yaml
  is today a --no-cluster simulation (cuj2-inference/chainsaw-test.yaml:86-92)
  and is not invoked by any workflow — this child makes inference a real
  cluster phase, not a dry run.
• Wire the inference path into the workflows so a workflow_dispatch
intent=inference (DC2) reaches phase_serve. Resolve the cron-scheduling
  decision explicitly (not as a parenthetical): either add a daily inference cron
  lane or alternate the single existing cron between intents. Do not schedule two
  daily crons against the single AWS reservation (cr-0cbe491320188dfa6) unless
  the superseded-run surfacing from DC1 is in place — two crons + a human dispatch on
  one reservation is a routine three-contender case, and the loser is silently
  cancelled. Because GCP draws from a separate reservation, alternating intents
  across clouds (e.g. training cron on one, inference cron on the other) is the
  safer launch posture. Make "two crons (or cron + dispatch) on one reservation queue
  cleanly, third is reported superseded" the documented validation scenario for the
  broker, and record the chosen cadence in docs/contributor/uat.md.
• Conformance + evidence emission already exist (phase_conformance,
  tests/uat/aws/run:175-191); confirm the inference recipe's criteria
  (intent: inference, platform: dynamo) flow into the emitted bundle so its
  TestGrid coordinate resolves to the inference-dynamo tab (under h100-ubuntu for
  AWS, h100-cos for GCP). Optionally cross-check, in phase_conformance, that the
  emitted bundle's declared platform matches the deployed component set (dynamo for
  inference, kubeflow-trainer for training) as a first-party sanity assertion — the
  Tab coordinate is author-declared and otherwise cluster-unverifiable
  (match.go:70-71).
• Docs same PR: document the inference UAT path in docs/contributor/uat.md
  and reference demos/cuj2.md (the operational CUJ2 definition the chainsaw test
  cites at cuj2-inference/chainsaw-test.yaml:23). If this child is the first to
  create docs/contributor/uat.md, register it in docs/index.yml.

Out of scope. TestGrid publish itself (the TestGrid epic TG2/TG5); Azure inference (DC7
decides Azure's fate).

Key files. tests/uat/{aws,gcp}/run (phase_train :193-256,
phase_conformance :175-191, dispatch case :289-301),
tests/uat/{aws,gcp}/tests/cuj2-inference/ (existing assets),
tests/uat/{aws,gcp}/tests/h100-inference-config.yaml (from DC2),
.github/workflows/uat-aws.yaml / uat-gcp.yaml, docs/contributor/uat.md,
docs/index.yml.

Tests. A real-hardware inference run (manual workflow_dispatch during
bring-up) reaches a served endpoint and emits a verifiable bundle; a --no-cluster
chainsaw dry-run keeps validating the inference AICRConfig resolves; phase_serve
fails closed (non-zero exit, captured logs) on a non-ready deployment, mirroring
phase_train's Failed=True handling (tests/uat/aws/run:233-240).

Acceptance. Inference runs end-to-end on real hardware at CUJ1 parity, emits a
signed bundle whose criteria map to inference-dynamo, the cron cadence is decided
and documented (avoiding two crons on the single AWS reservation), and the runner
cleanly selects train-vs-serve by intent.

Child of #1264 · staged from docs/design/011-uat-dynamic-clusters.md (DC3).