Parent: #1264 · Labels: enhancement, area/ci (auto), area/tests
(auto) · Size: M (launch deliverable is the KWOK matrix; hardware-matrix breadth
is latent until more reservation rows exist) · Depends on: KWOK-matrix
portion — only the existing KWOK harness (no DC2/DC3); hardware-matrix portion —
DC2 (training columns) and DC3 (inference columns). For end-to-end TestGrid
visibility: the TestGrid epic TG2/TG5.

Goal. Drive resolvable recipes across the dynamic clusters (and KWOK where
hardware-independent) so the TestGrid grid fills beyond the two launch recipes,
keyed by overlay metadata.name, across an AICR-version axis (main + the
previous N stable releases). DC5 owns the version axis of the nightly matrix and the
version-parameterized install that produces each release-row cell.

Scope.

• Enumerate resolvable recipes by scanning recipes/overlays/*.yaml and reading
  .spec.criteria.* with yq — reuse the existing KWOK classifier precedent
  (kwok-recipes.yaml:96-116), do not invent a new enumerator and do not depend on
  an aicr recipe list subcommand (none exists at HEAD).
• AICR-version axis. Add aicr_version as a matrix dimension over the set
  (main, rel-1, rel-2, …) — main at tip plus the previous N stable releases — so the
  hardware matrix is {aicr version} × {intent (training, inference)} × {cloud (eks/aws, gke/gcp)} and each cell is one TestGrid column tagged with its aicr_version,
  intent, cloud, and observed k8s major.minor. This is cross-version regression:
  does an older stable AICR still produce a working, conformant recipe against today's
  cluster? Ordering and time-box (matching DC1's batch schedule): main first, then
  the most-recent releases descending; the batch is time-boxed to the nightly window
  and drops the oldest release row first when the window is tight, so main and the
  freshest releases always land. N is a small bounded number (data-configured
  alongside the reservation registry), not "all history".
• Version-parameterized install. For a release row at version X, install + validate
  using the released aicr binary + validator images pinned to version X (and
  main builds from tip) — not the workflow's checked-out tip binary for every cell.
  The per-cell aicr_version (threaded from DC1's schedule / the uat-run.yaml
aicr_version input) selects which released aicr binary and validator image tag the
  install step pulls; the recipe content and cluster shape are otherwise identical to
  the main cell. Tag the emitted bundle / TestGrid column with aicr_version so the
  cells are sliceable by version (the TestGrid epic's TG4a/TG4b expose
  aicr_version as a column dimension/facet, parallel to the k8s facet).
• Partition the matrix by what each recipe needs: hardware-independent dimensions
  (scheduling/bundle-render) run in KWOK (already the Tier1/2/3 model) and depend on
  neither DC2 nor DC3, so the KWOK breadth can land in parallel with DC3; hardware
  dimensions (real GPU deploy/perf/conformance) run on the dynamic clusters via the
  broker (DC1) at the launch shapes (AWS H100 / h100-ubuntu, GCP H100 / h100-cos)
  — the training columns need DC2, the inference columns need DC3. A recipe whose
  accelerator/os has no reservation row runs KWOK-only until capacity exists.
• Launch reality: at launch there are exactly two reservation rows (AWS H100,
  GCP H100), so DC5's recipe-breadth contribution at launch is just the two launch
  recipes DC2/DC3 already produce — recipe breadth adds nothing until a third
  reservation row lands. The AICR-version axis multiplies those two launch recipes
  across main + the previous N releases, so the launch hardware matrix is wider on
  the version axis (more columns per recipe coordinate) even before a new accelerator
  arrives, bounded by the nightly time-box. DC5's launch deliverable is therefore the
  KWOK matrix (independent of DC2/DC3, mergeable early) plus the version axis over
  the two launch recipes; hardware recipe breadth is latent scope tied to the same
  reservation-row pull-trigger the dynamic-clusters epic uses for new accelerators.
• Produce a per-recipe result keyed by metadata.name for every executed recipe,
  tagged with its aicr_version, in a shape the TestGrid epic ingests (one column
  per run; column identity incorporates run identity per the TG2 contract in
  Decision/Approach §3, so distinct nightly runs of the same recipe+version are distinct
  columns while a re-ingested bundle is idempotent; main + each release is a distinct
  column under the same recipe coordinate, sliceable by aicr_version). The matrix is
  the producer of the grid the TestGrid epic serves.
• Keep the GPU runs off the merge gate (they stay cron/dispatch; see DC6 for the
  optional opt-in and the non-blocking smoke).

Out of scope. The TestGrid config-gen/taxonomy inference (the TestGrid epic TG3); the public
UI (the TestGrid epic TG4); offline structural health (#1224 owns that surface — see
Interaction).

Key files. .github/workflows/kwok-recipes.yaml (classifier :96-132), a new
or extended matrix workflow, tests/uat/{aws,gcp}/run (version-parameterized install
step), the shared uat-run.yaml (aicr_version input, DC1), recipes/overlays/*.yaml
(criteria source of truth), infra/uat/reservations.yaml (DC1, gates which shapes
run on hardware; the version-set N is data-configured alongside it).

Tests. The enumerator lists the expected resolvable recipes for a given
service/accelerator; a KWOK matrix run produces a result row per recipe (developable
against make kwok-test-all with no hardware); a launch-shape hardware run produces
a column for each of the two launch recipes (per intent); a release-row cell installs
the pinned released aicr/validator images for version X (not tip) and emits a bundle
tagged aicr_version=X; the version axis is main-first and drops the oldest release
first when the time-box is exceeded.

Acceptance. Resolvable recipes run across KWOK + dynamic clusters; each emits a
result keyed by metadata.name and tagged with aicr_version, one column per run;
the version axis spans main + the previous N stable releases, time-boxed and
main-first; release rows install the released aicr/validator images at that
version; recipes without a reservation row degrade to KWOK-only cleanly; the KWOK
portion is mergeable independent of DC3, and the hardware recipe breadth is correctly
latent until a third reservation row lands.

Child of #1264 · staged from docs/design/011-uat-dynamic-clusters.md (DC5).