Skip to content

GP4 — Consensus model + deterministic generator (the viz) #1404

Description

@njhensley

Parent: #1400 · Labels: enhancement (adds a tools/<generator>/** glob to .github/labeler.yml, e.g. area/ci or a new area/toolstools/** matches no glob today) · Size: L · Depends on: GP3 (bucket to read from); soft on GP2 (real data to render — can develop against fixtures)

Goal. A deterministic Go generator (pkg/<corroborate> model + tools/<generator> CLI, modeled on tools/bom) that pulls all CTRFs + metadata from GCS, computes the consensus model counting distinct verified signers, never builds, and emits a deterministic index.json + per-recipe series/*.json + a self-contained static HTML/CSS/JS renderer that reads them.

Scope.

  • Consensus model package (pkg/<corroborate>): given the set of (signer, run, phase, row, result, aicr_version, k8s_version, attested_at) tuples for a recipe coordinate, compute per row: S_pass/S_fail over distinct allowlisted signers (latest-per-signer in scope), then the cell state (CONFIRMED / SINGLE / CONTESTED / FAILING / UNTESTED) per the Decision rules. Bucket each signer's latest CTRF status totally and explicitly (pkg/validator/ctrf/types.go:26-39): passedS_pass; failed/otherS_fail; skipped/pendingNOT-RUN for that signer+row (a coverage gap, never a passing corroboration — so a skipped/pending row can neither promote to CONFIRMED nor suppress a CONTESTED). CONFIRMED requires ≥ 2 distinct allowlisted signers and empty S_fail; an unknown-verified community signer is a non-corroborating "reported" dot. Phase rollup CONTESTED > FAILING > UNTESTED > SINGLE > CONFIRMED. The row set is the union of CTRF Names across all sources (ctrf/types.go:114); a missing Name is NOT-RUN for that signer.
  • Source class from the verified signer + the GP1 allowlist (first-party issuer pin / community / partner sections); never a free flag.
  • Recency/version drift: default scope latest-per-signer; aicr-version and k8s major.minor (from meta.json, sourced from the bundle predicate, never the clock) carried per dot and exposed as facets in the emitted site.
  • Coordinate via the shared mapping helper. Derive (group, dashboard, tab) by calling the shared pkg/recipe coordinate helper (the TestGrid epic's single source of truth, TG6 spec / TG2 helper), taking resolved Criteria as input — never parsing metadata.name. Imported, not re-derived.
  • tools/<generator> CLI modeled on tools/bom (-deterministic-style flag, -out, read-from-GCS or read-from-dir): emits index.json + per-recipe series/*.json (the machine contract) + the static HTML/CSS/JS renderer (catalog overview, per-recipe consensus grid with the dot-strip-by-class, per-source drilldown; CSP-first nav; k8s + aicr-version facets). The renderer embeds no data — it fetches index.json on boot and lazy-loads series/<recipe>.json on drilldown. Self-contained (inline/asset-bundled), precedent demos/*.html (+ the testgrid-pages/ prototype).
  • Determinism (load-bearing). The JSON (index.json + series/*.json) + HTML must be byte-identical across two runs (incl. clock skew between them): no time.Now()/random/UUID on the emit path; all timestamps from the bundle predicate; fixed ordering (coordinate sort, PhaseOrder from pkg/validator/phases.go:45, CTRF name within phase, signer-set sort by id-hash, JSON key order). Follow MarshalYAMLDeterministic (pkg/serializer/yaml.go:31) / tools/bom -deterministic (tools/bom/main.go:64).
  • Add the tools/<generator>/** labeler glob (no glob exists for tools/** today; .github/labeler.yml:51-53 only globs infra/**).

Out of scope. The ingest (GP2); the Pages publish workflow (GP5); the bucket/SA (GP3); the docs explainer (GP6). Authoring the mapping function (the TestGrid epic owns it; imported here).

Key files. New pkg/<corroborate>/ (consensus model) + tools/<generator>/ (CLI, index.json + series/*.json + HTML emit, README); pkg/validator/ctrf/types.go:26-39 (the five status constants), :85-131 (Summary + TestResult); pkg/validator/phases.go:34-45 (PhaseDeployment/Performance/Conformance + PhaseOrder); the shared pkg/recipe coordinate helper (imported); the GP1 allowlist; determinism precedent pkg/serializer/yaml.go:31, tools/bom/main.go:64; HTML precedent demos/*.html; .github/labeler.yml.

Tests. Table-driven consensus tests covering every state: CONFIRMED (≥2 allowlisted distinct), SINGLE (1 signer), CONTESTED (≥1 pass + ≥1 fail, asserted surfaced not averaged), FAILING (all fail), UNTESTED (no signer); a CTRF status-bucketing test asserting each of the five statuses maps to the intended bucket (passedS_pass; failed/otherS_fail; skipped/pending→NOT-RUN), including a skipped-does-not-corroborate case (a skipped latest never promotes a row to CONFIRMED) and a skipped-does-not-suppress-CONTESTED case (a skipped row alongside one pass + one fail stays CONTESTED); sybil tests — N runs from one signer = SINGLE not CONFIRMED; two unknown community signers never reach CONFIRMED; one first-party + one unknown-community = SINGLE with a reported dot; an over-broad allowlist entry is rejected (the no-wildcard lint from GP1). Source-class derivation tests (first-party from issuer pin; community/partner from allowlist; unknown = reported). Recency tests (latest-per-signer wins; an older aicr-version run does not fuse with a newer one; facet scoping). Determinism: byte-identical JSON (index.json + series/*.json) + HTML across two runs with injected clock skew (proving timestamps come from the predicate). Mapping golden reuse (the canonical recipe→coordinate golden lives in the shared pkg/recipe helper, exercised for both launch clouds — h100-eks-ubuntu-training-kubeflow → eks/h100-ubuntu/training-kubeflow; h100-gke-cos-... → gke/h100-cos/...); enumerate criteria via the runtime union ((*CriteriaRegistry).Values(field), criteria_registry.go:179, via GetCriteriaRegistryFor, :254; static set GetCriteriaServiceTypes() etc., criteria.go:98), never a closed enum. 75% floor on pkg/<corroborate> and the generator package.

Acceptance. The generator pulls CTRFs + metadata, computes the consensus model counting distinct verified signers (sybil-resistant: unknown signers never manufacture CONFIRMED; CONTESTED first-class), emits byte-deterministic index.json + per-recipe series/*.json + a self-contained static renderer (catalog overview → consensus grid → dot-strip-by-class → per-source drilldown, CSP-first nav, k8s + aicr-version facets) via the shared mapping helper, and clears the 75% floor; tools/<generator>/** self-labels.


Child of #1400 · staged from docs/design/013-interim-evidence-dashboard.md (GP4).

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions