Summary
Make AICR's real-hardware UAT comprehensive and contention-safe by time-sharing the
single per-cloud reservation across a day/night operating model. Introduce a
simple day/night scheduler + capacity request/queue broker that arbitrates the
scarce reservation-backed GPU time (replacing today's AWS hard-fail-on-busy): night
runs a main-first, time-boxed version matrix — the training and inference CUJ on
AWS and GCP for main + the previous N stable AICR releases — sequentially per cloud;
morning hands each freed reservation to a long-lived daytime human-access
deployment (one training cluster on one cloud, one inference cluster on the other,
for human use outside CI); day is untouched-by-CI human use; evening tears the
daytime deployment down — an enforced cleanup boundary — before the next night
batch. Make the per-run cluster shape per intent (training vs inference) from the
same reservation, and version-parameterize the install (released aicr/validator
images at version X for each release row, main at tip) so the matrix is
cross-version regression. Wire the existing CUJ2 inference assets into the UAT runner
at CUJ1 parity, broaden CLI-boundary coverage (query, diff, evidence verify/publish, real-cluster snapshot, more bundle deployers), and drive resolvable
recipes across the dynamic clusters (and KWOK where hardware-independent) so the matrix
fills out beyond the two launch recipes. Every UAT run emits an ADR-007 evidence bundle
handed to the TestGrid epic's TestGrid ingestion (authenticating as the TestGrid epic's
dedicated write-only publish SA, never the shared UAT actuator SA); the
dynamic-clusters epic does not write the bucket, and the daytime human-access clusters
emit no bundle and produce no column. Launch is phased: training-kubeflow on both
clouds first (eks/h100-ubuntu, gke/h100-cos), inference-dynamo once DC3
lands. Designed for more reservations, clouds, and accelerators to come (and eventually
both daytime flavors per cloud). Cost governance is deferred (the only variable cost is
the CPU/system node pool — autoscaled on GCP, fixed on AWS).
Children
Epic acceptance criteria
- The single per-cloud reservation is time-shared on a day/night cycle: the
nightly batch runs a main-first, time-boxed version matrix sequentially
per cloud (provision→CUJ→evidence→publish→teardown→next); the morning handoff
grants the daytime human-access lease; the evening teardown is enforced and the
nightly batch refuses to start (pre-batch guard) against an un-torn-down daytime
cluster rather than racing the reservation.
- Two UAT runs contending for the same reservation (AWS cron + a human AWS
workflow_dispatch, or two requesters sharing a future shared reservation)
queue rather than one hard-failing; a third contender is explicitly
reported as superseded (not silently cancelled); the reservation set is
data-driven (adding a row needs no broker code change).
training-kubeflow runs end-to-end on real hardware on AWS H100
(eks/h100-ubuntu) and GCP H100 (gke/h100-cos) at launch, and
inference-dynamo runs end-to-end on both once DC3 lands, each emitting a
signed ADR-007 evidence bundle whose criteria map to the correct per-cloud
TestGrid coordinate.query, diff, evidence verify/publish, real-cluster snapshot, and at
least one additional bundle deployer beyond helmfile are exercised by
real-or-simulated tests that emit a CTRF/JUnit-shaped result (the end-to-end
"appears as a TestGrid column" check depends on the TestGrid epic TG2/TG5).- At least the full set of resolvable launch-scope recipes (and KWOK-coverable
recipes) run through a matrix that produces per-recipe results keyed by overlay
metadata.name, one column per run, across the AICR-version axis (main +
the previous N stable releases, time-boxed and main-first; each release row
installs the released aicr/validator images at that version and tags its
bundle/column aicr_version).
- At the morning handoff, one training (one cloud) + one inference (other cloud)
long-lived deployment come up for human use outside CI (DC8); the inference one
serves a reachable OpenAI-compatible endpoint; access is shared out-of-band; the
daytime clusters emit no evidence bundle and produce no TestGrid column; both are
torn down before the nightly batch.
- The Azure stub has a recorded revive-or-retire decision (implemented, DC7), and the
orphaned tests/chainsaw/snapshot/deploy-agent test (plus its stale README
cross-reference) is wired or removed.
make qualify green on every child; any child touching Go packages clears the
75% coverage floor (.settings.yaml:107); docs updated in the same PR as
behavior; no new required merge-gate check that depends on GPUs.
Out of scope (deferred — see Deferred)
Cost governance / budget caps; a standing broker service (vs lease + GH
concurrency); MI300/B200/GB200 shapes at launch (data rows added when capacity
exists); per-PR GPU runs as a required gate; both daytime flavors (training +
inference) on a single cloud during the working day (one reservation per cloud
cannot hold a daytime cluster and run the nightly batch at once — needs more infra);
more than the bounded N stable releases on the version axis (time-box drops the
oldest first); alerting on UAT regressions (dropped — the regression signal is
TestGrid itself, per the recipe-quality epic brief).
Staged from docs/design/011-uat-dynamic-clusters.md.
Summary
Make AICR's real-hardware UAT comprehensive and contention-safe by time-sharing the
single per-cloud reservation across a day/night operating model. Introduce a
simple day/night scheduler + capacity request/queue broker that arbitrates the
scarce reservation-backed GPU time (replacing today's AWS hard-fail-on-busy): night
runs a
main-first, time-boxed version matrix — the training and inference CUJ onAWS and GCP for
main+ the previous N stable AICR releases — sequentially per cloud;morning hands each freed reservation to a long-lived daytime human-access
deployment (one training cluster on one cloud, one inference cluster on the other,
for human use outside CI); day is untouched-by-CI human use; evening tears the
daytime deployment down — an enforced cleanup boundary — before the next night
batch. Make the per-run cluster shape per intent (training vs inference) from the
same reservation, and version-parameterize the install (released
aicr/validatorimages at version X for each release row,
mainat tip) so the matrix iscross-version regression. Wire the existing CUJ2 inference assets into the UAT runner
at CUJ1 parity, broaden CLI-boundary coverage (
query,diff,evidence verify/publish, real-clustersnapshot, more bundle deployers), and drive resolvablerecipes across the dynamic clusters (and KWOK where hardware-independent) so the matrix
fills out beyond the two launch recipes. Every UAT run emits an ADR-007 evidence bundle
handed to the TestGrid epic's TestGrid ingestion (authenticating as the TestGrid epic's
dedicated write-only publish SA, never the shared UAT actuator SA); the
dynamic-clusters epic does not write the bucket, and the daytime human-access clusters
emit no bundle and produce no column. Launch is phased:
training-kubeflowon bothclouds first (
eks/h100-ubuntu,gke/h100-cos),inference-dynamoonce DC3lands. Designed for more reservations, clouds, and accelerators to come (and eventually
both daytime flavors per cloud). Cost governance is deferred (the only variable cost is
the CPU/system node pool — autoscaled on GCP, fixed on AWS).
Children
Epic acceptance criteria
nightly batch runs a
main-first, time-boxed version matrix sequentiallyper cloud (provision→CUJ→evidence→publish→teardown→next); the morning handoff
grants the daytime human-access lease; the evening teardown is enforced and the
nightly batch refuses to start (pre-batch guard) against an un-torn-down daytime
cluster rather than racing the reservation.
workflow_dispatch, or two requesters sharing a future shared reservation)queue rather than one hard-failing; a third contender is explicitly
reported as superseded (not silently cancelled); the reservation set is
data-driven (adding a row needs no broker code change).
training-kubeflowruns end-to-end on real hardware on AWS H100(
eks/h100-ubuntu) and GCP H100 (gke/h100-cos) at launch, andinference-dynamoruns end-to-end on both once DC3 lands, each emitting asigned ADR-007 evidence bundle whose criteria map to the correct per-cloud
TestGrid coordinate.
query,diff,evidence verify/publish, real-clustersnapshot, and atleast one additional bundle deployer beyond
helmfileare exercised byreal-or-simulated tests that emit a CTRF/JUnit-shaped result (the end-to-end
"appears as a TestGrid column" check depends on the TestGrid epic TG2/TG5).
recipes) run through a matrix that produces per-recipe results keyed by overlay
metadata.name, one column per run, across the AICR-version axis (main+the previous N stable releases, time-boxed and
main-first; each release rowinstalls the released
aicr/validator images at that version and tags itsbundle/column
aicr_version).long-lived deployment come up for human use outside CI (DC8); the inference one
serves a reachable OpenAI-compatible endpoint; access is shared out-of-band; the
daytime clusters emit no evidence bundle and produce no TestGrid column; both are
torn down before the nightly batch.
orphaned
tests/chainsaw/snapshot/deploy-agenttest (plus its stale READMEcross-reference) is wired or removed.
make qualifygreen on every child; any child touching Go packages clears the75% coverage floor (
.settings.yaml:107); docs updated in the same PR asbehavior; no new required merge-gate check that depends on GPUs.
Out of scope (deferred — see Deferred)
Cost governance / budget caps; a standing broker service (vs lease + GH
concurrency); MI300/B200/GB200 shapes at launch (data rows added when capacity
exists); per-PR GPU runs as a required gate; both daytime flavors (training +
inference) on a single cloud during the working day (one reservation per cloud
cannot hold a daytime cluster and run the nightly batch at once — needs more infra);
more than the bounded N stable releases on the version axis (time-box drops the
oldest first); alerting on UAT regressions (dropped — the regression signal is
TestGrid itself, per the recipe-quality epic brief).
Staged from
docs/design/011-uat-dynamic-clusters.md.