Skip to content

Epic: TG — Deploy & operate the AICR TestGrid instance #1263

Description

@njhensley

Summary

Stand up a new, public, AICR-native TestGrid in the eidosx UAT GCP project. A
tools/ CI publisher converts AICR
evidence bundles (CTRF → deterministic jUnit) into a TestGrid-wire-compatible
GCS feed laid out CSP/service-first (group=service, dashboard=accelerator-os,
tab=intent[-platform], k8s-in-column). Upstream Kubernetes TestGrid
updater/tabulator/summarizer workers roll it up; an AICR-native config-gen, API,
and public UI replace the upstream config-gen, API, aggregator, and web UI. Both first-party UAT
runs and accepted external-contributor ADR-007 bundles are ingested by an
AICR-owned CI job — contributors never write the bucket. The write identity reuses
the existing GitHub-Actions→GCP WIF federation on a dedicated, write-only
publish service account (not the shared high-privilege UAT SA). Launch covers AWS
H100 (eks/h100-ubuntu) and GCP H100 (gke/h100-cos) — two dashboards,
because the clouds run different OS families.

Children

Epic acceptance criteria

  1. A nightly UAT run produces an evidence bundle that is ingested and published to
    the AICR TestGrid GCS feed at the correct CSP-first coordinate, and the build
    appears as a column in the public UI within one worker cycle — AWS H100 lands
    under eks / h100-ubuntu; GCP H100 lands under gke / h100-cos.
  2. The publish path is deterministic (same evidence bundle → byte-identical
    jUnit + started/finished metadata, with started/finished timestamps sourced
    from the bundle predicate, never time.Now()), is idempotent (re-ingesting
    a bundle for the same run does not create a duplicate column), preserves
    time-series ordering (distinct nightly runs of the same recipe are distinct,
    chronologically-ordered columns), and uses a dedicated, write-only WIF
    identity that cannot read, list, or delete the bucket and cannot touch
    GKE/compute.
  3. The taxonomy is CSP-first everywhere (group=service, dashboard=accelerator-os,
    tab=intent[-platform]); k8s version appears only in column metadata (never
    in a path segment), and is instead incorporated via observed major.minor +
    declared constraint in column metadata, a first-class UI facet/filter (TG4b)
    and API grouping dimension (TG4a); aicr_version is likewise a first-class UI
    facet/filter (TG4b) and API grouping dimension (TG4a), parallel to the k8s facet,
    so the multiple per-coordinate columns from main + the previous N stable AICR
    releases can be sliced, grouped, and compared by AICR version; a recipe's
    (group, dashboard, tab)
    coordinate is stable across a k8s major.minor change; the single shared
    recipe→coordinate function (one Go package, golden-tested for both launch
    clouds) is the only mapping used by TG2, TG3, TG4a, and the recipe-quality
    epic RQ1.
  4. Column metadata carries aicr version (aicr_version), observed k8s version
    (k8s_version, major.minor), the recipe-declared k8s constraint (k8s_constraint),
    verified signer identity + issuer, source (uat/community, derived from
    the verified signer
    ), and the evidence bundle digest, under a pinned key
    schema
    that TG3's config-gen Header.extra references; the UI surfaces
    signer/source and the declared-vs-observed k8s pairing, and distinguishes
    source=community (self-asserted signer, provenance-not-correctness) from
    source=uat (workflow-OIDC-pinned).
  5. An accepted external-contributor ADR-007 bundle (referenced by a
    recipes/evidence/<recipe>.yaml pointer) is ingested and published by the
    AICR-owned CI job after a real signer-policy verify (pinned issuer + in-tree
    identity allowlist) and a trusted-registry allow-list check; at no point does
    the contributor hold bucket-write credentials.
  6. TG6's canonical recipe→coordinate mapping is documented, ships as the shared Go
    function, and emits a stable URL that the recipe-quality epic's Evidence column
    and link-integrity bot can consume via TG4a's coordinate-presence endpoint; the
    docs page reconciles with ADR-009's deferred verify-gated evidence axis and
    cross-links docs/user/recipe-health.md without duplicating its data.
  7. The running instance is operable: the GKE host cluster, workers, and API expose
    liveness/readiness; a publish-path failure and a produced-but-unpublished
    bundle (the dynamic-clusters side succeeds, TestGrid ingestion fails / no column
    materializes within N worker cycles) surface as visible failures (not silent
    swallows); a runbook covers a stuck queue / wedged worker / failed publish /
    host-cluster outage.
  8. make qualify green; 75% coverage floor cleared on every new Go package
    (.settings.yaml:107 coverage_threshold: '75'; the floor, not the stale 70%
    in CLAUDE.md).

Out of scope

Making UAT comprehensive (CUJ2 inference, more CLI boundaries) and turning the
static reservation clusters dynamic — that is the dynamic-clusters epic (the
inference-dynamo tab specifically depends on DC3). Replacing the pending
Evidence column in recipe-health.md and the CUJ/CLI coverage matrix — that is
the recipe-quality epic. Regression
alerting (dropped for now; UAT failure signal comes from TestGrid itself). UAT
cluster cost governance (deferred — but the always-on TestGrid GKE footprint is an
in-scope standing cost, see Deferred). KMS-backed signing (ADR-007 deferred). A
junit/ directory inside the evidence bundle (that is the parallel
ADR-007-extension epic; TG2's jUnit is rendered into the GCS feed, not the OCI
bundle).


Staged from docs/design/010-aicr-testgrid.md.

Metadata

Metadata

Type

Fields

No fields configured for Epic.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions