Parent: #1400 · Labels: enhancement (adds a tools/<generator>/** glob to .github/labeler.yml, e.g. area/ci or a new area/tools — tools/** matches no glob today) · Size: L · Depends on: GP3 (bucket to read from); soft on GP2 (real data to render — can develop against fixtures)
Goal. A deterministic Go generator (pkg/<corroborate> model + tools/<generator> CLI, modeled on tools/bom) that pulls all CTRFs + metadata from GCS, computes the consensus model counting distinct verified signers, never builds, and emits a deterministic index.json + per-recipe series/*.json + a self-contained static HTML/CSS/JS renderer that reads them.
Scope.
- Consensus model package (
pkg/<corroborate>): given the set of (signer, run, phase, row, result, aicr_version, k8s_version, attested_at) tuples for a recipe coordinate, compute per row: S_pass/S_fail over distinct allowlisted signers (latest-per-signer in scope), then the cell state (CONFIRMED / SINGLE / CONTESTED / FAILING / UNTESTED) per the Decision rules. Bucket each signer's latest CTRF status totally and explicitly (pkg/validator/ctrf/types.go:26-39): passed→S_pass; failed/other→S_fail; skipped/pending→NOT-RUN for that signer+row (a coverage gap, never a passing corroboration — so a skipped/pending row can neither promote to CONFIRMED nor suppress a CONTESTED). CONFIRMED requires ≥ 2 distinct allowlisted signers and empty S_fail; an unknown-verified community signer is a non-corroborating "reported" dot. Phase rollup CONTESTED > FAILING > UNTESTED > SINGLE > CONFIRMED. The row set is the union of CTRF Names across all sources (ctrf/types.go:114); a missing Name is NOT-RUN for that signer.
- Source class from the verified signer + the GP1 allowlist (first-party issuer pin / community / partner sections); never a free flag.
- Recency/version drift: default scope latest-per-signer; aicr-version and k8s
major.minor (from meta.json, sourced from the bundle predicate, never the clock) carried per dot and exposed as facets in the emitted site.
- Coordinate via the shared mapping helper. Derive
(group, dashboard, tab) by calling the shared pkg/recipe coordinate helper (the TestGrid epic's single source of truth, TG6 spec / TG2 helper), taking resolved Criteria as input — never parsing metadata.name. Imported, not re-derived.
tools/<generator> CLI modeled on tools/bom (-deterministic-style flag, -out, read-from-GCS or read-from-dir): emits index.json + per-recipe series/*.json (the machine contract) + the static HTML/CSS/JS renderer (catalog overview, per-recipe consensus grid with the dot-strip-by-class, per-source drilldown; CSP-first nav; k8s + aicr-version facets). The renderer embeds no data — it fetches index.json on boot and lazy-loads series/<recipe>.json on drilldown. Self-contained (inline/asset-bundled), precedent demos/*.html (+ the testgrid-pages/ prototype).
- Determinism (load-bearing). The JSON (
index.json + series/*.json) + HTML must be byte-identical across two runs (incl. clock skew between them): no time.Now()/random/UUID on the emit path; all timestamps from the bundle predicate; fixed ordering (coordinate sort, PhaseOrder from pkg/validator/phases.go:45, CTRF name within phase, signer-set sort by id-hash, JSON key order). Follow MarshalYAMLDeterministic (pkg/serializer/yaml.go:31) / tools/bom -deterministic (tools/bom/main.go:64).
- Add the
tools/<generator>/** labeler glob (no glob exists for tools/** today; .github/labeler.yml:51-53 only globs infra/**).
Out of scope. The ingest (GP2); the Pages publish workflow (GP5); the bucket/SA (GP3); the docs explainer (GP6). Authoring the mapping function (the TestGrid epic owns it; imported here).
Key files. New pkg/<corroborate>/ (consensus model) + tools/<generator>/ (CLI, index.json + series/*.json + HTML emit, README); pkg/validator/ctrf/types.go:26-39 (the five status constants), :85-131 (Summary + TestResult); pkg/validator/phases.go:34-45 (PhaseDeployment/Performance/Conformance + PhaseOrder); the shared pkg/recipe coordinate helper (imported); the GP1 allowlist; determinism precedent pkg/serializer/yaml.go:31, tools/bom/main.go:64; HTML precedent demos/*.html; .github/labeler.yml.
Tests. Table-driven consensus tests covering every state: CONFIRMED (≥2 allowlisted distinct), SINGLE (1 signer), CONTESTED (≥1 pass + ≥1 fail, asserted surfaced not averaged), FAILING (all fail), UNTESTED (no signer); a CTRF status-bucketing test asserting each of the five statuses maps to the intended bucket (passed→S_pass; failed/other→S_fail; skipped/pending→NOT-RUN), including a skipped-does-not-corroborate case (a skipped latest never promotes a row to CONFIRMED) and a skipped-does-not-suppress-CONTESTED case (a skipped row alongside one pass + one fail stays CONTESTED); sybil tests — N runs from one signer = SINGLE not CONFIRMED; two unknown community signers never reach CONFIRMED; one first-party + one unknown-community = SINGLE with a reported dot; an over-broad allowlist entry is rejected (the no-wildcard lint from GP1). Source-class derivation tests (first-party from issuer pin; community/partner from allowlist; unknown = reported). Recency tests (latest-per-signer wins; an older aicr-version run does not fuse with a newer one; facet scoping). Determinism: byte-identical JSON (index.json + series/*.json) + HTML across two runs with injected clock skew (proving timestamps come from the predicate). Mapping golden reuse (the canonical recipe→coordinate golden lives in the shared pkg/recipe helper, exercised for both launch clouds — h100-eks-ubuntu-training-kubeflow → eks/h100-ubuntu/training-kubeflow; h100-gke-cos-... → gke/h100-cos/...); enumerate criteria via the runtime union ((*CriteriaRegistry).Values(field), criteria_registry.go:179, via GetCriteriaRegistryFor, :254; static set GetCriteriaServiceTypes() etc., criteria.go:98), never a closed enum. 75% floor on pkg/<corroborate> and the generator package.
Acceptance. The generator pulls CTRFs + metadata, computes the consensus model counting distinct verified signers (sybil-resistant: unknown signers never manufacture CONFIRMED; CONTESTED first-class), emits byte-deterministic index.json + per-recipe series/*.json + a self-contained static renderer (catalog overview → consensus grid → dot-strip-by-class → per-source drilldown, CSP-first nav, k8s + aicr-version facets) via the shared mapping helper, and clears the 75% floor; tools/<generator>/** self-labels.
Child of #1400 · staged from docs/design/013-interim-evidence-dashboard.md (GP4).
Parent: #1400 · Labels:
enhancement(adds atools/<generator>/**glob to.github/labeler.yml, e.g.area/cior a newarea/tools—tools/**matches no glob today) · Size: L · Depends on: GP3 (bucket to read from); soft on GP2 (real data to render — can develop against fixtures)Goal. A deterministic Go generator (
pkg/<corroborate>model +tools/<generator>CLI, modeled ontools/bom) that pulls all CTRFs + metadata from GCS, computes the consensus model counting distinct verified signers, never builds, and emits a deterministicindex.json+ per-recipeseries/*.json+ a self-contained static HTML/CSS/JS renderer that reads them.Scope.
pkg/<corroborate>): given the set of(signer, run, phase, row, result, aicr_version, k8s_version, attested_at)tuples for a recipe coordinate, compute per row:S_pass/S_failover distinct allowlisted signers (latest-per-signer in scope), then the cell state (CONFIRMED / SINGLE / CONTESTED / FAILING / UNTESTED) per the Decision rules. Bucket each signer's latest CTRF status totally and explicitly (pkg/validator/ctrf/types.go:26-39):passed→S_pass;failed/other→S_fail;skipped/pending→NOT-RUN for that signer+row (a coverage gap, never a passing corroboration — so askipped/pendingrow can neither promote to CONFIRMED nor suppress a CONTESTED). CONFIRMED requires ≥ 2 distinct allowlisted signers and emptyS_fail; an unknown-verified community signer is a non-corroborating "reported" dot. Phase rollupCONTESTED > FAILING > UNTESTED > SINGLE > CONFIRMED. The row set is the union of CTRF Names across all sources (ctrf/types.go:114); a missing Name is NOT-RUN for that signer.major.minor(frommeta.json, sourced from the bundle predicate, never the clock) carried per dot and exposed as facets in the emitted site.(group, dashboard, tab)by calling the sharedpkg/recipecoordinate helper (the TestGrid epic's single source of truth, TG6 spec / TG2 helper), taking resolvedCriteriaas input — never parsingmetadata.name. Imported, not re-derived.tools/<generator>CLI modeled ontools/bom(-deterministic-style flag,-out, read-from-GCS or read-from-dir): emitsindex.json+ per-recipeseries/*.json(the machine contract) + the static HTML/CSS/JS renderer (catalog overview, per-recipe consensus grid with the dot-strip-by-class, per-source drilldown; CSP-first nav; k8s + aicr-version facets). The renderer embeds no data — itfetchesindex.jsonon boot and lazy-loadsseries/<recipe>.jsonon drilldown. Self-contained (inline/asset-bundled), precedentdemos/*.html(+ thetestgrid-pages/prototype).index.json+series/*.json) + HTML must be byte-identical across two runs (incl. clock skew between them): notime.Now()/random/UUID on the emit path; all timestamps from the bundle predicate; fixed ordering (coordinate sort,PhaseOrderfrompkg/validator/phases.go:45, CTRF name within phase, signer-set sort by id-hash, JSON key order). FollowMarshalYAMLDeterministic(pkg/serializer/yaml.go:31) /tools/bom -deterministic(tools/bom/main.go:64).tools/<generator>/**labeler glob (no glob exists fortools/**today;.github/labeler.yml:51-53only globsinfra/**).Out of scope. The ingest (GP2); the Pages publish workflow (GP5); the bucket/SA (GP3); the docs explainer (GP6). Authoring the mapping function (the TestGrid epic owns it; imported here).
Key files. New
pkg/<corroborate>/(consensus model) +tools/<generator>/(CLI,index.json+series/*.json+ HTML emit, README);pkg/validator/ctrf/types.go:26-39(the five status constants),:85-131(Summary+TestResult);pkg/validator/phases.go:34-45(PhaseDeployment/Performance/Conformance+PhaseOrder); the sharedpkg/recipecoordinate helper (imported); the GP1 allowlist; determinism precedentpkg/serializer/yaml.go:31,tools/bom/main.go:64; HTML precedentdemos/*.html;.github/labeler.yml.Tests. Table-driven consensus tests covering every state: CONFIRMED (≥2 allowlisted distinct), SINGLE (1 signer), CONTESTED (≥1 pass + ≥1 fail, asserted surfaced not averaged), FAILING (all fail), UNTESTED (no signer); a CTRF status-bucketing test asserting each of the five statuses maps to the intended bucket (
passed→S_pass;failed/other→S_fail;skipped/pending→NOT-RUN), including a skipped-does-not-corroborate case (askippedlatest never promotes a row to CONFIRMED) and a skipped-does-not-suppress-CONTESTED case (askippedrow alongside one pass + one fail stays CONTESTED); sybil tests — N runs from one signer = SINGLE not CONFIRMED; two unknown community signers never reach CONFIRMED; one first-party + one unknown-community = SINGLE with a reported dot; an over-broad allowlist entry is rejected (the no-wildcard lint from GP1). Source-class derivation tests (first-party from issuer pin; community/partner from allowlist; unknown = reported). Recency tests (latest-per-signer wins; an older aicr-version run does not fuse with a newer one; facet scoping). Determinism: byte-identical JSON (index.json+series/*.json) + HTML across two runs with injected clock skew (proving timestamps come from the predicate). Mapping golden reuse (the canonical recipe→coordinate golden lives in the sharedpkg/recipehelper, exercised for both launch clouds —h100-eks-ubuntu-training-kubeflow → eks/h100-ubuntu/training-kubeflow;h100-gke-cos-... → gke/h100-cos/...); enumerate criteria via the runtime union ((*CriteriaRegistry).Values(field),criteria_registry.go:179, viaGetCriteriaRegistryFor,:254; static setGetCriteriaServiceTypes()etc.,criteria.go:98), never a closed enum. 75% floor onpkg/<corroborate>and the generator package.Acceptance. The generator pulls CTRFs + metadata, computes the consensus model counting distinct verified signers (sybil-resistant: unknown signers never manufacture CONFIRMED; CONTESTED first-class), emits byte-deterministic
index.json+ per-recipeseries/*.json+ a self-contained static renderer (catalog overview → consensus grid → dot-strip-by-class → per-source drilldown, CSP-first nav, k8s + aicr-version facets) via the shared mapping helper, and clears the 75% floor;tools/<generator>/**self-labels.Child of #1400 · staged from
docs/design/013-interim-evidence-dashboard.md(GP4).