Parent: #1264 · Labels: enhancement, area/ci (auto), area/tests (auto,
tests/**) · Size: M · Depends on: DC1

Goal. Acquire a cluster from a reservation through the broker and shape it
for the requested intent (training vs inference), replacing AWS's hard-fail-on-busy
pre-flight with queue-aware acquisition — and decide what the GCP side does, since
GCP has no pre-flight capacity gate today. DC2 also owns the two lifecycles a
cluster can have on a reservation: the per-run nightly-batch cluster (provision →
CUJ → teardown, the existing behavior) and the long-lived daytime deployment (the
morning-handoff cluster DC8 hands to humans), plus the enforced cleanup boundary
between them.

Scope.

• AWS: convert the existing capacity step (uat-aws.yaml:122-147) from a
  race-and-fail check into a post-lease assertion: once the broker grants the
  lease, assert the reservation has the desired count (a genuinely exhausted
  reservation still fails; contention no longer does, because the lease was the
  gate).
• GCP: there is nothing to convert — uat-gcp.yaml has no capacity-check
  step. Explicitly decide between (a) adding a symmetric GCP-side
  capacity/quota assertion (post-lease, parallel to the AWS one) or (b)
  documenting that GCP relies on the GKE actuator failing at provision time.
  Record the decision in docs/contributor/uat.md. Do not imply a symmetric edit
  that does not exist.
• Select the test config and recipe criteria by an intent input rather than the
  hard-coded TEST_CONFIG: …/h100-training-config.yaml (uat-aws.yaml:55,
  uat-gcp.yaml:56). Introduce tests/uat/{aws,gcp}/tests/h100-inference-config.yaml
  as a sibling of the training config (same AICRConfig shape as
  tests/uat/aws/tests/h100-training-config.yaml), with
  spec.validate.recipe.criteria.intent: inference / platform: dynamo. For the
  push target, set spec.validate.evidence.attestation.out + .push to a stable
  per-intent prefix (e.g. ghcr.io/nvidia/aicr-uat-eks-h100-inference, mirroring the
  training config's :98); the runner's inject_push_target (tests/uat/aws/run:69-76)
  appends -${RUN_ID} at runtime for per-run OCI isolation, so the config sets only
  the stable prefix.
• System/CPU pools stay per-run dynamic on both clouds (GCP autoscales, AWS fixed
  desired: 3); the GPU pool count comes from the reservation row. No second
  reservation is hardcoded.
• Daytime long-lived deployment lifecycle. Add a provision-and-hold mode (distinct
  from the nightly per-run provision-and-destroy) the morning handoff uses to stand up
  the DC8 daytime cluster: provision the cluster sized for its intent, deploy the
  workload, and leave it up (no teardown at job end). The daytime cluster's name is
  stable / lease-tagged rather than aicr-uat-${{ github.run_id }}
  (uat-aws.yaml:53) so the evening teardown and the pre-batch guard can find it. DC8
  owns what is deployed and how access is shared; DC2 owns the provision-and-hold /
  teardown mechanics and the cluster lifecycle.
• Enforced cleanup-before-UAT boundary. Add an explicit teardown step that releases
  the daytime cluster (and its DC8 lease) before the nightly batch opens, plus a
  pre-batch guard: the nightly batch refuses to start against a reservation that
  still holds an un-torn-down daytime cluster (detect by the lease tag / cluster name),
  failing fast with a clear message rather than racing the reservation. A missed
  teardown must surface as a blocked batch, never as silent contention. Tie the guard
  to DC1's reservation lease so "daytime lease still held" is observable.

Out of scope. The inference workload steps themselves (DC3 owns phase_serve); the
daytime deployment's workload content, served-endpoint exposure, and out-of-band
access distribution (DC8); the broker scheduling mechanism and lease (DC1).

Key files. .github/workflows/uat-aws.yaml (capacity :122-147, TEST_CONFIG
env :55, DEPLOYMENT_ID/cluster name :53, Destroy Cluster :343),
.github/workflows/uat-gcp.yaml (TEST_CONFIG env :56; no capacity step — GCP
decision lands here), tests/uat/{aws,gcp}/tests/h100-inference-config.yaml
(new), tests/uat/{aws,gcp}/cluster-config.yaml, tests/uat/{aws,gcp}/run
(inject_push_target :69-76), docs/contributor/uat.md.

Tests. Dispatch the workflow with intent=inference and confirm it selects the
inference config and provisions; confirm a busy-AWS-reservation dispatch queues (DC1)
rather than exiting 1; confirm the GCP capacity decision is reflected (assertion
present and green, or the actuator-time-failure behavior documented). Confirm
provision-and-hold leaves the daytime cluster up at job end and that the pre-batch
guard fails fast when a daytime cluster on the target reservation has not been torn
down (rather than provisioning into contention). Chainsaw/--no-cluster dry-runs
validate the new inference AICRConfig resolves.

Acceptance. A run selects its cluster shape from {reservation, intent}; busy
AWS reservations queue; the GCP capacity posture is decided and implemented; the
inference config provisions a cluster the same way training does; the daytime cluster
provisions-and-holds and is torn down before the batch, with a pre-batch guard that
blocks (not races) when an un-torn-down daytime cluster still holds the reservation.

Child of #1264 · staged from docs/design/011-uat-dynamic-clusters.md (DC2).