Skip to content

GP2 — Ingest CI (pointers → verify-before-count → source-keyed GCS) #1402

Description

@njhensley

Parent: #1400 · Labels: enhancement, area/ci (auto, .github/**) · Size: L · Depends on: GP1 (pointers + allowlist), GP3 (bucket + write SA)

Goal. An AICR-owned CI job — triggered both on merge-to-main over recipes/evidence/** (community/partner pointers) and directly from each first-party UAT run (by bundle ref, no repo commit) — that ORAS-mirrors the bundle to a durable registry, unpacks CTRF + metadata via MaterializeBundle, verifies the signature/issuer/identity and the OCI registry before counting, and writes a source-keyed GCS layout — so contributors never write the bucket and unverified evidence never reaches the model.

Scope.

  • Two triggers. (a) On merge to recipes/evidence/**: discover per-source pointer files by glob recipes/evidence/<slug>/<source>/*.yaml (Make evidence pointers per-source (per contributing party) #1347 Option A) and read each via LoadAndValidatePointer (pointer.go:37) — reusing/extending the PR-time discovery harness in .github/scripts/recipe-evidence-check.sh. (b) Invoked directly from each first-party UAT run with the run's bundle OCI ref (uat-gcp.yaml:291), so first-party evidence ingests without a per-run repo commit. Both paths converge on the same verify→write steps below.
  • ORAS-mirror each referenced bundle (PointerBundle.OCI + .Digest, types.go:211) to a durable registry as a backup against the source registry going dark — an ORAS copy (pkg/oci/push.go oras.Copy/PushFromStore, :277,:294,:342), not aicr mirror (which is image discovery, pkg/cli/mirror.go:61-64).
  • Pull + unpack via MaterializeBundle (pkg/evidence/verifier/fetch.go:80materializeOCIRefRequireDigest, :195), reusing the existing ORAS transport; read ctrf/<phase>.json (deployment/performance/conformance, written by builder.go:136-155) and predicate metadata (AttestedAt, criteria, aicr version, k8s, digest).
  • Verify before counting in a step holding no bucket-write credentials: re-verify via the same verification engine the evidence verify CLI uses (verifier.Verify, pkg/evidence/verifier/verify.go:52), pinning both --expected-issuer (allowed issuer set) and --expected-identity-regexp (the GP1 in-tree allowlist — empty pins are a no-op; these are the pin flags defined at evidence_verify.go:78-84, uat-aws.yaml:258-260), reject tag-only refs (--allow-unpinned-tag=false, evidence_verify.go:103), and enforce a trusted-registry allowlist on the OCI ref (closing the open follow-up documented verbatim at .github/workflows/recipe-evidence.yaml:37 / .github/scripts/recipe-evidence-check.sh:46), bounding layer/bundle size against SSRF/exfil/DoS. The verified signer (PointerSigner identity+issuer, types.go:223) and source class are recorded from the verified SAN, never the raw pointer string.
  • Write source-keyed GCS in a separate step holding the dedicated write-only publish identity (GP3): results/<recipe-coordinate>/<signer-id-hash>/<run-id>/{ctrf/<phase>.json, meta.json}, where <recipe-coordinate> is from the shared mapping helper and meta.json carries verified signer id+issuer+class, aicr version, k8s major.minor, evidence digest, and AttestedAt. Create-only writes; same-bundle re-ingest is idempotent on the digest (PointerBundle.Digest).
  • Add a tools/** labeler glob if the ingest reuses a shared Go helper under tools/; if it is pure workflow + aicr CLI invocations, .github/**area/ci already covers it.

Out of scope. The bucket/SA Terraform (GP3); the generator/consensus model (GP4); Pages publish (GP5). Deciding corroboration weight (GP4 reads the GP1 allowlist).

Key files. New .github/workflows/evidence-ingest.yaml (+ any shared Go helper under tools/ or pkg/); pkg/evidence/verifier/fetch.go:80,:195 (MaterializeBundle); pkg/evidence/verifier/verify.go:52 (verifier.Verify, the verification engine); pkg/oci/push.go:277,:294,:342 (ORAS copy); pkg/cli/evidence_verify.go:78-84,:103 (issuer/identity/unpinned-tag pin flags), :123 (CLI's Verify call); pointer loader + single-attestation constraint pkg/evidence/verifier/pointer.go:37,:74-83; pointer schema pkg/evidence/attestation/types.go:188-226; verify-leg prior art (PR-time workflow + pointer discovery) .github/workflows/recipe-evidence.yaml:30-48 / .github/scripts/recipe-evidence-check.sh:40-52, registry-allowlist follow-up recipe-evidence.yaml:37 / recipe-evidence-check.sh:46; WIF auth uat-gcp.yaml:53-54,:109,:115; the shared pkg/recipe coordinate helper (imported).

Tests. A fixture/dry-run test that the ingest constructs the right verify + mirror + write invocations; an assertion the verify step uses non-empty issuer + identity pins and rejects tag-only refs; an assertion the registry-allowlist rejects out-of-list refs; a fork-safety assertion (no bucket credentials exposed to a PR/fork context, per hippo:reviewer-cicd norms) and that the write step uses the dedicated objectCreator-only SA (not the shared github-actions@eidosx); an idempotency test (same digest re-ingest replaces, does not duplicate); a source-keying test (two distinct signers land under two <signer-id-hash> prefixes). 75% floor on any new Go helper.

Acceptance. On merge to recipes/evidence/**, the job ORAS-mirrors the bundle, unpacks CTRF via MaterializeBundle, verifies signer + issuer + identity + registry in a credential-free step before counting, and writes a source-keyed GCS entry with verified metadata using the dedicated write-only identity; unverified or out-of-registry bundles are rejected; contributors hold no bucket credentials.


Child of #1400 · staged from docs/design/013-interim-evidence-dashboard.md (GP2).

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions