Parent: #1264 · Labels: enhancement, area/ci (auto, .github/**),
area/tests (auto, tests/**), area/docs (auto) · Size: M · Depends on:
DC1 (the day/night scheduler + daytime lease) and DC2 (provision-and-hold lifecycle +
enforced cleanup boundary)

Goal. At the morning handoff, stand up one long-lived deployment per cloud that
humans interact with outside CI through the working day, then tear it down before
the nightly batch. The two flavors are split across the two clouds — AWS hosts a
training cluster, GCP hosts an inference cluster (≈1 training + 1 inference total) —
so the day's scarce reservation time is divided between human use (DC8) and the nightly
version matrix (DC5). The inference deployment exposes a served (OpenAI-compatible)
endpoint humans can hit; the training deployment lets humans submit jobs / demo. This
is not a CI lane: nothing here emits a TestGrid column, and access is distributed
out-of-band.

Scope.

• Morning-handoff provisioning. After DC1's nightly batch drains a reservation,
  acquire the daytime lease (DC1) and use DC2's provision-and-hold mode to stand up
  the daytime cluster sized for its intent, deploy the workload (training stack on AWS,
  Dynamo inference on GCP), and leave it up — no per-run teardown. Use a stable,
  lease-tagged cluster name (not aicr-uat-${{ github.run_id }}, uat-aws.yaml:53) so
  the evening teardown and DC2's pre-batch guard can find it.
• Inference served endpoint. The GCP inference deployment exposes its
  OpenAI-compatible endpoint (the same Dynamo DynamoGraphDeployment DC3 deploys for
  phase_serve) for humans to send completions to; the AWS training deployment exposes
  a way to submit Kubeflow TrainJobs / run a demo. Reuse the DC3 serve assets and the
  DC2 inference config — DC8 does not invent a new serving stack.
• Out-of-band access. Distribute the kubeconfig / endpoint URL out-of-band (a
  team channel / shared secret store), never through the CI path, the evidence
  bundle, or the TestGrid dashboard. The daytime cluster produces no TestGrid
  column and emits no evidence bundle — it is human-facing, not a UAT cell.
• Evening teardown before the batch. Tear the daytime deployment down and release
  its DC1 lease before the nightly UAT batch opens, freeing the reservation. This is
  the enforced cleanup boundary DC2 implements; DC8 schedules the teardown at the
  evening edge of the day/night cycle and relies on DC2's pre-batch guard to block
  (not race) the batch if a teardown was missed.
• Cloud split is configurable, not hardcoded forever. Which cloud hosts which flavor
  is a data choice (alongside the reservation registry, DC1), defaulting to AWS=training
  / GCP=inference. Both flavors per cloud during the day is out of scope at launch
  (one reservation per cloud cannot host both a held daytime cluster and the nightly
  batch simultaneously) — it is a Deferred pull-trigger once more infra lands.
• Docs same PR. Document the daytime human-access deployment in
  docs/contributor/uat.md (the day/night-cycle page DC1 creates): how to reach each
  cluster, when it is up (working day only) and down (overnight), and the out-of-band
  access path. If DC8 is somehow the first to create the page, register it in
  docs/index.yml.

Out of scope. The day/night scheduler and lease mechanism (DC1); the
provision-and-hold / teardown / pre-batch-guard mechanics (DC2); the phase_serve
deployment logic and evidence emission (DC3 — DC8 reuses the serve assets but emits no
bundle); both flavors per cloud during the day (Deferred); any CI/TestGrid surfacing
of the daytime clusters.

Key files. infra/uat/reservations.yaml (DC1, the cloud→flavor default lives
alongside it), the shared uat-run.yaml / a daytime-deployment workflow,
.github/workflows/uat-aws.yaml / uat-gcp.yaml (provision-and-hold + teardown
scheduling), tests/uat/{aws,gcp}/run (reuse phase_serve from DC3 / training deploy),
tests/uat/gcp/tests/h100-inference-config.yaml (from DC2), docs/contributor/uat.md.

Tests. A manual workflow_dispatch provisions the daytime training cluster on AWS
and the daytime inference cluster on GCP, leaves both up, and the GCP inference endpoint
answers a sample OpenAI-compatible request; the evening teardown removes both and
releases the DC1 lease; a deliberately-skipped teardown causes DC2's pre-batch guard to
block the next nightly batch (not race it); confirm no evidence bundle / TestGrid column
is emitted for the daytime clusters.

Acceptance. At the morning handoff one training (one cloud) + one inference (other
cloud) long-lived deployment come up for human use outside CI; the inference deployment
serves a reachable OpenAI-compatible endpoint; access is shared out-of-band and nothing
daytime appears as a TestGrid column; the deployment is torn down and its lease released
before the nightly batch, with DC2's pre-batch guard blocking the batch if a teardown is
missed.

Child of #1264 · staged from docs/design/011-uat-dynamic-clusters.md (DC8).