Parent: #1263 · Labels: enhancement · Size: L · Depends on: TG1
(no auto area label today — tools/** matches no glob in .github/labeler.yml;
this child MUST add a tools/** (or tools/testgrid-publish/**) glob mapping to
a new/existing area, e.g. area/ci or a new area/tools.)
Goal. A deterministic, least-privilege CI Go tool that consumes an AICR
evidence OCI bundle, converts its per-phase CTRF to canonical jUnit XML, assembles
started.json/finished.json, and writes the 4-level TestGrid-wire-compatible
GCS feed at the CSP-first coordinate with the pinned column-metadata key schema.
Scope.
- New
tools/testgrid-publish/ Go tool (GOFLAGS=-mod=vendor go run ./tools/testgrid-publish), modeled on the tools/bom Go-tool precedent (flags,
-deterministic posture, README).
- Input = OCI evidence bundle. Pull via ORAS (the same transport ADR-007's
--push/verifier paths use), materialize the summary-bundle/, read
recipe.yaml (for criteria → coordinate) and ctrf/{deployment,performance, conformance}.json. The publisher's job: OCI bundle → CTRF → jUnit →
started/finished → GCS.
- Coordinate via the shared mapping function. Derive
(group, dashboard, tab)
by calling the shared pkg/recipe coordinate helper (the single source of
truth; TG6 owns the spec), taking the recipe's resolved Criteria as input —
never by parsing metadata.name. The same helper is imported by TG3/TG4a/RQ1.
- Deterministic CTRF→jUnit converter. Map CTRF
TestResult
(pkg/validator/ctrf/types.go: Name, Status ∈ {passed,failed,skipped,
pending,other}, Duration int ms, Suite, Message, Stdout) to jUnit:
failed→<failure>, other→<error>, skipped/pending→<skipped/>,
passed→bare <testcase>; time = strconv.FormatFloat(ms/1000, 'f', 3, 64);
suite totals = sum-of-rounded; preserve CTRF order; omit timestamp/hostname;
fixed XML declaration + indentation + encoding/xml escaping; rune-safe
line-boundary <system-out> cap with a fixed sentinel. Byte-deterministic
per the pkg/serializer.MarshalYAMLDeterministic / tools/bom -deterministic
precedent. Rows are named <phase>/<check>. Multi-CTRF merge: all
ctrf/*.json (deployment/performance/conformance, plus any phases the dynamic-clusters epic DC4
adds) merge into one deterministic junit.xml with phase-namespaced row names so
the per-build column never collides duplicate test-case names into one flaky row.
- Tab sanitize — empty platform yields bare intent. When
platform is absent,
the tab is the bare intent (training, inference) with no trailing
hyphen and no unknown substitution — the generic empty→unknown rule
applies only to a required segment that resolves empty, never to an
intentionally-omitted optional sub-segment. Path-construction tests include a
no-platform recipe so TG2 and RQ1 are provably consistent.
- build-id (load-bearing for idempotency AND time-series). Define
{build-id}
so that (a) distinct first-party UAT runs of the same recipe are distinct,
chronologically-ordered columns and (b) re-ingesting a bundle for the same
run is idempotent. The idempotency key is {recipe-coordinate + run identity},
not raw bundle bytes (a re-signed/re-pushed bundle of the same run gets a new
digest because the predicate's AttestedAt is a Truncate(time.Second)
wall-clock, distinct from BundleHash). Confirm against the upstream updater
whether columns sort by started.json timestamp or by build-id string; if the
latter, prefix the build-id with a zero-padded/ISO timestamp (timestamp + short-digest) so it is both monotonic and idempotent. For community
re-ingestion of an identical bundle, the digest makes it idempotent. A column
ordering golden/integration check lives in TG3's seeded-feed smoke.
- started.json / finished.json — timestamps from the bundle, not the clock.
started.json = {timestamp, metadata}; finished.json = {timestamp, passed(bool), result: SUCCESS|FAILURE, metadata}. Both timestamps are sourced
from the bundle predicate (AttestedAt / phase timestamps), never
time.Now() at publish — otherwise determinism and same-run idempotency break.
passed is binary, derived from the CTRF summary (any phase with
Summary.Failed > 0 or Summary.Other > 0 → FAILURE); nuance stays in the
jUnit. Upload order started → junit → finished so a partial failure never
shows a result without evidence.
- GCS path = the contract. Write
gs://<bucket>/groups/{group}/{dashboard}/{tab}/{build-id}/ with started.json,
artifacts/junit.xml, finished.json. group=service, dashboard=accelerator-os,
tab=intent[-platform], all from the bundle's recipe criteria via the shared
helper. 4-level depth is load-bearing — never emit a 5th level. Sanitize every
required segment (lowercase; collapse /\:;[]{}, +whitespace to -; dedupe;
trim; empty→unknown; k8s truncated to major.minor for column metadata).
- Column metadata under a pinned key schema. Emit the started.json metadata
with the exact key strings, count, and order defined once in TG6's taxonomy
spec and referenced verbatim by TG3's config-gen: the aicr version
(aicr_version, the released/main-tip AICR version that produced the run); the
observed runtime k8s version (k8s_version, truncated to major.minor, from
the bundle predicate / snapshot); the recipe-declared k8s constraint
(k8s_constraint,
the K8s.server.version range read from the recipe — e.g. ">= 1.32.4",
recipes/overlays/gb200-eks-training.yaml:32; cf. ">= 1.30" at
aks-training.yaml:31 / gke-cos-inference.yaml:35 — read for display only,
never as a path segment); verified signer identity + issuer (the
verified SAN from aicr evidence verify, not the raw bundle-asserted string);
source (uat iff the verified SAN matches the UAT-workflow identity regexp,
else community — derived, not a TG5 free flag); and the evidence bundle
digest. These map onto the column-header extra array the upstream workers and
API understand (Header.extra; e.g. the upstream convention uses
[release_stream, release_tag, runtime_release, operations_release],
test-result-ingestion-reference.md §3, §6).
- Least-privilege write. Authenticate via GCP WIF using the dedicated
write-only publish identity from TG1; the tool only creates objects (no
read/list/delete). Idempotent (replace-on-write at the same build-id blob path),
so re-runs are safe.
- Add a
tools/** labeler glob + CODEOWNERS. tools/ has no labeler.yml
glob today, so a tools/testgrid-publish/** PR gets no area label until this
child adds one (e.g. area/ci or a new area/tools). A narrower
tools/testgrid-publish/** CODEOWNERS entry is optional (precedent: ADR-009's
planned pkg/health/** self-labeling); the * @nvidia/aicr-write catch-all
already owns the path.
Out of scope. Deploying the workers (TG3); the config-gen (TG3); the API/UI
(TG4a/TG4b); deciding which bundles to publish or the signer/registry policy (TG5);
a junit/ directory inside the OCI evidence bundle (the parallel
ADR-007-extension epic).
Key files. New tools/testgrid-publish/ (main.go, converter, gcs writer,
README); the shared coordinate helper under pkg/recipe; pkg/validator/ctrf/types.go
(CTRF model), pkg/validator/ctrf/builder.go:56 (ExitCodeToCTRFStatus);
pkg/validator/phases.go:34-45 (PhaseDeployment/PhasePerformance/
PhaseConformance, PhaseOrder); determinism precedent pkg/serializer/yaml.go:31,
tools/bom/main.go:63,145; pointer/signer shape pkg/evidence/attestation/pointer.go,
signer.go; verified-SAN provenance from aicr evidence verify
(pkg/cli/evidence_verify.go:78-84). .github/labeler.yml.
Tests. Golden-file converter tests (each CTRF status incl. pending→<skipped/>
and other→<error>; multi-CTRF phase-namespaced merge with no row collision;
mixed-suite aggregate counts; rounding-boundary time; over-cap rune-safe
<system-out>; byte-identical across two marshals; XML escaping). A determinism
test that runs the publisher twice with clock skew between runs and asserts
byte-identical output (proving timestamps come from the bundle, not time.Now()).
GCS path-construction table tests (sanitization, 4-level depth, empty→unknown for
required segments, bare-intent tab for no-platform, k8s truncation, build-id
monotonic+idempotent). started/finished assembly tests (binary passed from CTRF
summary; pinned metadata keys present; timestamps from predicate). A WIF/GCS write
test against a fake or emulated bucket asserting create-only behavior, that two
distinct runs produce two columns, and that re-ingesting the same run replaces
rather than duplicates. 75% floor on the new package (pkg=tools/testgrid-publish
if measured as a package, else the converter sub-package). The shared coordinate
helper's golden test (both launch clouds) lives in pkg/recipe.
Acceptance. tools/testgrid-publish consumes an OCI evidence bundle, emits a
byte-deterministic 4-level CSP-first GCS feed via the shared coordinate helper
(timestamps from the bundle predicate; build-id monotonic+idempotent so distinct
runs are distinct ordered columns), with the pinned column-metadata key schema and
verified provenance, writes with the dedicated create-only identity, and is covered
by golden + path + determinism + write tests; tools/** self-labels.
Child of #1263 · staged from docs/design/010-aicr-testgrid.md (TG2).
Parent: #1263 · Labels:
enhancement· Size: L · Depends on: TG1(no auto area label today —
tools/**matches no glob in.github/labeler.yml;this child MUST add a
tools/**(ortools/testgrid-publish/**) glob mapping toa new/existing area, e.g.
area/cior a newarea/tools.)Goal. A deterministic, least-privilege CI Go tool that consumes an AICR
evidence OCI bundle, converts its per-phase CTRF to canonical jUnit XML, assembles
started.json/finished.json, and writes the 4-level TestGrid-wire-compatibleGCS feed at the CSP-first coordinate with the pinned column-metadata key schema.
Scope.
tools/testgrid-publish/Go tool (GOFLAGS=-mod=vendor go run ./tools/testgrid-publish), modeled on thetools/bomGo-tool precedent (flags,-deterministicposture, README).--push/verifier paths use), materialize thesummary-bundle/, readrecipe.yaml(for criteria → coordinate) andctrf/{deployment,performance, conformance}.json. The publisher's job: OCI bundle → CTRF → jUnit →started/finished → GCS.
(group, dashboard, tab)by calling the shared
pkg/recipecoordinate helper (the single source oftruth; TG6 owns the spec), taking the recipe's resolved
Criteriaas input —never by parsing
metadata.name. The same helper is imported by TG3/TG4a/RQ1.TestResult(
pkg/validator/ctrf/types.go:Name,Status∈ {passed,failed,skipped,pending,other},Durationint ms,Suite,Message,Stdout) to jUnit:failed→<failure>,other→<error>,skipped/pending→<skipped/>,passed→bare<testcase>;time=strconv.FormatFloat(ms/1000, 'f', 3, 64);suite totals = sum-of-rounded; preserve CTRF order; omit timestamp/hostname;
fixed XML declaration + indentation +
encoding/xmlescaping; rune-safeline-boundary
<system-out>cap with a fixed sentinel. Byte-deterministicper the
pkg/serializer.MarshalYAMLDeterministic/tools/bom -deterministicprecedent. Rows are named
<phase>/<check>. Multi-CTRF merge: allctrf/*.json(deployment/performance/conformance, plus any phases the dynamic-clusters epic DC4adds) merge into one deterministic
junit.xmlwith phase-namespaced row names sothe per-build column never collides duplicate test-case names into one flaky row.
platformis absent,the tab is the bare
intent(training,inference) with no trailinghyphen and no
unknownsubstitution — the genericempty→unknownruleapplies only to a required segment that resolves empty, never to an
intentionally-omitted optional sub-segment. Path-construction tests include a
no-platform recipe so TG2 and RQ1 are provably consistent.
{build-id}so that (a) distinct first-party UAT runs of the same recipe are distinct,
chronologically-ordered columns and (b) re-ingesting a bundle for the same
run is idempotent. The idempotency key is
{recipe-coordinate + run identity},not raw bundle bytes (a re-signed/re-pushed bundle of the same run gets a new
digest because the predicate's
AttestedAtis aTruncate(time.Second)wall-clock, distinct from
BundleHash). Confirm against the upstream updaterwhether columns sort by
started.jsontimestamp or by build-id string; if thelatter, prefix the build-id with a zero-padded/ISO timestamp (
timestamp + short-digest) so it is both monotonic and idempotent. For communityre-ingestion of an identical bundle, the digest makes it idempotent. A column
ordering golden/integration check lives in TG3's seeded-feed smoke.
started.json = {timestamp, metadata};finished.json = {timestamp, passed(bool), result: SUCCESS|FAILURE, metadata}. Both timestamps are sourcedfrom the bundle predicate (
AttestedAt/ phase timestamps), nevertime.Now()at publish — otherwise determinism and same-run idempotency break.passedis binary, derived from the CTRF summary (any phase withSummary.Failed > 0orSummary.Other > 0→ FAILURE); nuance stays in thejUnit. Upload order started → junit → finished so a partial failure never
shows a result without evidence.
gs://<bucket>/groups/{group}/{dashboard}/{tab}/{build-id}/withstarted.json,artifacts/junit.xml,finished.json. group=service, dashboard=accelerator-os,tab=
intent[-platform], all from the bundle's recipe criteria via the sharedhelper. 4-level depth is load-bearing — never emit a 5th level. Sanitize every
required segment (lowercase; collapse
/\:;[]{},+whitespace to-; dedupe;trim; empty→
unknown; k8s truncated to major.minor for column metadata).with the exact key strings, count, and order defined once in TG6's taxonomy
spec and referenced verbatim by TG3's config-gen: the aicr version
(
aicr_version, the released/main-tip AICR version that produced the run); theobserved runtime k8s version (
k8s_version, truncated tomajor.minor, fromthe bundle predicate / snapshot); the recipe-declared k8s constraint
(
k8s_constraint,the
K8s.server.versionrange read from the recipe — e.g.">= 1.32.4",recipes/overlays/gb200-eks-training.yaml:32; cf.">= 1.30"ataks-training.yaml:31/gke-cos-inference.yaml:35— read for display only,never as a path segment); verified signer identity + issuer (the
verified SAN from
aicr evidence verify, not the raw bundle-asserted string);source (
uatiff the verified SAN matches the UAT-workflow identity regexp,else
community— derived, not a TG5 free flag); and the evidence bundledigest. These map onto the column-header
extraarray the upstream workers andAPI understand (
Header.extra; e.g. the upstream convention uses[release_stream, release_tag, runtime_release, operations_release],test-result-ingestion-reference.md§3, §6).write-only publish identity from TG1; the tool only creates objects (no
read/list/delete). Idempotent (replace-on-write at the same build-id blob path),
so re-runs are safe.
tools/**labeler glob + CODEOWNERS.tools/has nolabeler.ymlglob today, so a
tools/testgrid-publish/**PR gets no area label until thischild adds one (e.g.
area/cior a newarea/tools). A narrowertools/testgrid-publish/**CODEOWNERS entry is optional (precedent: ADR-009'splanned
pkg/health/**self-labeling); the* @nvidia/aicr-writecatch-allalready owns the path.
Out of scope. Deploying the workers (TG3); the config-gen (TG3); the API/UI
(TG4a/TG4b); deciding which bundles to publish or the signer/registry policy (TG5);
a
junit/directory inside the OCI evidence bundle (the parallelADR-007-extension epic).
Key files. New
tools/testgrid-publish/(main.go, converter, gcs writer,README); the shared coordinate helper under
pkg/recipe;pkg/validator/ctrf/types.go(CTRF model),
pkg/validator/ctrf/builder.go:56(ExitCodeToCTRFStatus);pkg/validator/phases.go:34-45(PhaseDeployment/PhasePerformance/PhaseConformance,PhaseOrder); determinism precedentpkg/serializer/yaml.go:31,tools/bom/main.go:63,145; pointer/signer shapepkg/evidence/attestation/pointer.go,signer.go; verified-SAN provenance fromaicr evidence verify(
pkg/cli/evidence_verify.go:78-84)..github/labeler.yml.Tests. Golden-file converter tests (each CTRF status incl.
pending→<skipped/>and
other→<error>; multi-CTRF phase-namespaced merge with no row collision;mixed-suite aggregate counts; rounding-boundary
time; over-cap rune-safe<system-out>; byte-identical across two marshals; XML escaping). A determinismtest that runs the publisher twice with clock skew between runs and asserts
byte-identical output (proving timestamps come from the bundle, not
time.Now()).GCS path-construction table tests (sanitization, 4-level depth, empty→
unknownforrequired segments, bare-intent tab for no-platform, k8s truncation, build-id
monotonic+idempotent). started/finished assembly tests (binary
passedfrom CTRFsummary; pinned metadata keys present; timestamps from predicate). A WIF/GCS write
test against a fake or emulated bucket asserting create-only behavior, that two
distinct runs produce two columns, and that re-ingesting the same run replaces
rather than duplicates. 75% floor on the new package (
pkg=tools/testgrid-publishif measured as a package, else the converter sub-package). The shared coordinate
helper's golden test (both launch clouds) lives in
pkg/recipe.Acceptance.
tools/testgrid-publishconsumes an OCI evidence bundle, emits abyte-deterministic 4-level CSP-first GCS feed via the shared coordinate helper
(timestamps from the bundle predicate; build-id monotonic+idempotent so distinct
runs are distinct ordered columns), with the pinned column-metadata key schema and
verified provenance, writes with the dedicated create-only identity, and is covered
by golden + path + determinism + write tests;
tools/**self-labels.Child of #1263 · staged from
docs/design/010-aicr-testgrid.md(TG2).