Skip to content

TG2 — Publish path (tools/testgrid-publish: OCI bundle → CTRF → jUnit → GCS feed) #1267

Description

@njhensley

Parent: #1263 · Labels: enhancement · Size: L · Depends on: TG1
(no auto area label today — tools/** matches no glob in .github/labeler.yml;
this child MUST add a tools/** (or tools/testgrid-publish/**) glob mapping to
a new/existing area, e.g. area/ci or a new area/tools.)

Goal. A deterministic, least-privilege CI Go tool that consumes an AICR
evidence OCI bundle, converts its per-phase CTRF to canonical jUnit XML, assembles
started.json/finished.json, and writes the 4-level TestGrid-wire-compatible
GCS feed at the CSP-first coordinate with the pinned column-metadata key schema.

Scope.

  • New tools/testgrid-publish/ Go tool (GOFLAGS=-mod=vendor go run ./tools/testgrid-publish), modeled on the tools/bom Go-tool precedent (flags,
    -deterministic posture, README).
  • Input = OCI evidence bundle. Pull via ORAS (the same transport ADR-007's
    --push/verifier paths use), materialize the summary-bundle/, read
    recipe.yaml (for criteria → coordinate) and ctrf/{deployment,performance, conformance}.json. The publisher's job: OCI bundle → CTRF → jUnit →
    started/finished → GCS.
  • Coordinate via the shared mapping function. Derive (group, dashboard, tab)
    by calling the shared pkg/recipe coordinate helper (the single source of
    truth; TG6 owns the spec), taking the recipe's resolved Criteria as input —
    never by parsing metadata.name. The same helper is imported by TG3/TG4a/RQ1.
  • Deterministic CTRF→jUnit converter. Map CTRF TestResult
    (pkg/validator/ctrf/types.go: Name, Status ∈ {passed,failed,skipped,
    pending,other}, Duration int ms, Suite, Message, Stdout) to jUnit:
    failed<failure>, other<error>, skipped/pending<skipped/>,
    passed→bare <testcase>; time = strconv.FormatFloat(ms/1000, 'f', 3, 64);
    suite totals = sum-of-rounded; preserve CTRF order; omit timestamp/hostname;
    fixed XML declaration + indentation + encoding/xml escaping; rune-safe
    line-boundary <system-out> cap with a fixed sentinel. Byte-deterministic
    per the pkg/serializer.MarshalYAMLDeterministic / tools/bom -deterministic
    precedent. Rows are named <phase>/<check>. Multi-CTRF merge: all
    ctrf/*.json (deployment/performance/conformance, plus any phases the dynamic-clusters epic DC4
    adds) merge into one deterministic junit.xml with phase-namespaced row names so
    the per-build column never collides duplicate test-case names into one flaky row.
  • Tab sanitize — empty platform yields bare intent. When platform is absent,
    the tab is the bare intent (training, inference) with no trailing
    hyphen
    and no unknown substitution — the generic empty→unknown rule
    applies only to a required segment that resolves empty, never to an
    intentionally-omitted optional sub-segment. Path-construction tests include a
    no-platform recipe so TG2 and RQ1 are provably consistent.
  • build-id (load-bearing for idempotency AND time-series). Define {build-id}
    so that (a) distinct first-party UAT runs of the same recipe are distinct,
    chronologically-ordered columns
    and (b) re-ingesting a bundle for the same
    run
    is idempotent. The idempotency key is {recipe-coordinate + run identity},
    not raw bundle bytes (a re-signed/re-pushed bundle of the same run gets a new
    digest because the predicate's AttestedAt is a Truncate(time.Second)
    wall-clock, distinct from BundleHash). Confirm against the upstream updater
    whether columns sort by started.json timestamp or by build-id string; if the
    latter, prefix the build-id with a zero-padded/ISO timestamp (timestamp + short-digest) so it is both monotonic and idempotent. For community
    re-ingestion of an identical bundle, the digest makes it idempotent. A column
    ordering golden/integration check lives in TG3's seeded-feed smoke.
  • started.json / finished.json — timestamps from the bundle, not the clock.
    started.json = {timestamp, metadata}; finished.json = {timestamp, passed(bool), result: SUCCESS|FAILURE, metadata}. Both timestamps are sourced
    from the bundle predicate (AttestedAt / phase timestamps), never
    time.Now() at publish — otherwise determinism and same-run idempotency break.
    passed is binary, derived from the CTRF summary (any phase with
    Summary.Failed > 0 or Summary.Other > 0 → FAILURE); nuance stays in the
    jUnit. Upload order started → junit → finished so a partial failure never
    shows a result without evidence.
  • GCS path = the contract. Write
    gs://<bucket>/groups/{group}/{dashboard}/{tab}/{build-id}/ with started.json,
    artifacts/junit.xml, finished.json. group=service, dashboard=accelerator-os,
    tab=intent[-platform], all from the bundle's recipe criteria via the shared
    helper. 4-level depth is load-bearing — never emit a 5th level. Sanitize every
    required segment (lowercase; collapse /\:;[]{}, +whitespace to -; dedupe;
    trim; empty→unknown; k8s truncated to major.minor for column metadata).
  • Column metadata under a pinned key schema. Emit the started.json metadata
    with the exact key strings, count, and order defined once in TG6's taxonomy
    spec and referenced verbatim by TG3's config-gen: the aicr version
    (aicr_version, the released/main-tip AICR version that produced the run); the
    observed runtime k8s version (k8s_version, truncated to major.minor, from
    the bundle predicate / snapshot); the recipe-declared k8s constraint
    (k8s_constraint,
    the K8s.server.version range read from the recipe — e.g. ">= 1.32.4",
    recipes/overlays/gb200-eks-training.yaml:32; cf. ">= 1.30" at
    aks-training.yaml:31 / gke-cos-inference.yaml:35 — read for display only,
    never as a path segment); verified signer identity + issuer (the
    verified SAN from aicr evidence verify, not the raw bundle-asserted string);
    source (uat iff the verified SAN matches the UAT-workflow identity regexp,
    else community — derived, not a TG5 free flag); and the evidence bundle
    digest
    . These map onto the column-header extra array the upstream workers and
    API understand (Header.extra; e.g. the upstream convention uses
    [release_stream, release_tag, runtime_release, operations_release],
    test-result-ingestion-reference.md §3, §6).
  • Least-privilege write. Authenticate via GCP WIF using the dedicated
    write-only publish identity from TG1; the tool only creates objects (no
    read/list/delete). Idempotent (replace-on-write at the same build-id blob path),
    so re-runs are safe.
  • Add a tools/** labeler glob + CODEOWNERS. tools/ has no labeler.yml
    glob today, so a tools/testgrid-publish/** PR gets no area label until this
    child adds one (e.g. area/ci or a new area/tools). A narrower
    tools/testgrid-publish/** CODEOWNERS entry is optional (precedent: ADR-009's
    planned pkg/health/** self-labeling); the * @nvidia/aicr-write catch-all
    already owns the path.

Out of scope. Deploying the workers (TG3); the config-gen (TG3); the API/UI
(TG4a/TG4b); deciding which bundles to publish or the signer/registry policy (TG5);
a junit/ directory inside the OCI evidence bundle (the parallel
ADR-007-extension epic).

Key files. New tools/testgrid-publish/ (main.go, converter, gcs writer,
README); the shared coordinate helper under pkg/recipe; pkg/validator/ctrf/types.go
(CTRF model), pkg/validator/ctrf/builder.go:56 (ExitCodeToCTRFStatus);
pkg/validator/phases.go:34-45 (PhaseDeployment/PhasePerformance/
PhaseConformance, PhaseOrder); determinism precedent pkg/serializer/yaml.go:31,
tools/bom/main.go:63,145; pointer/signer shape pkg/evidence/attestation/pointer.go,
signer.go; verified-SAN provenance from aicr evidence verify
(pkg/cli/evidence_verify.go:78-84). .github/labeler.yml.

Tests. Golden-file converter tests (each CTRF status incl. pending<skipped/>
and other<error>; multi-CTRF phase-namespaced merge with no row collision;
mixed-suite aggregate counts; rounding-boundary time; over-cap rune-safe
<system-out>; byte-identical across two marshals; XML escaping). A determinism
test that runs the publisher twice with clock skew between runs and asserts
byte-identical output (proving timestamps come from the bundle, not time.Now()).
GCS path-construction table tests (sanitization, 4-level depth, empty→unknown for
required segments, bare-intent tab for no-platform, k8s truncation, build-id
monotonic+idempotent). started/finished assembly tests (binary passed from CTRF
summary; pinned metadata keys present; timestamps from predicate). A WIF/GCS write
test against a fake or emulated bucket asserting create-only behavior, that two
distinct runs produce two columns, and that re-ingesting the same run replaces
rather than duplicates. 75% floor on the new package (pkg=tools/testgrid-publish
if measured as a package, else the converter sub-package). The shared coordinate
helper's golden test (both launch clouds) lives in pkg/recipe.

Acceptance. tools/testgrid-publish consumes an OCI evidence bundle, emits a
byte-deterministic 4-level CSP-first GCS feed via the shared coordinate helper
(timestamps from the bundle predicate; build-id monotonic+idempotent so distinct
runs are distinct ordered columns), with the pinned column-metadata key schema and
verified provenance, writes with the dedicated create-only identity, and is covered
by golden + path + determinism + write tests; tools/** self-labels.


Child of #1263 · staged from docs/design/010-aicr-testgrid.md (TG2).

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions