Skip to content

feat(evidence): GP2 ingest — verify bundles to source-keyed GCS#1484

Merged
njhensley merged 9 commits into
NVIDIA:mainfrom
njhensley:ci/evidence-ingest
Jun 26, 2026
Merged

feat(evidence): GP2 ingest — verify bundles to source-keyed GCS#1484
njhensley merged 9 commits into
NVIDIA:mainfrom
njhensley:ci/evidence-ingest

Conversation

@njhensley

Copy link
Copy Markdown
Member

Summary

Add the GP2 evidence-ingest producer: verify published, signed recipe-evidence bundles and synthesize the source-keyed tree the GP4 corroborate generator consumes, then publish it to the corroboration GCS bucket.

Motivation / Context

Nothing on main turns the per-run signed evidence bundles (from aicr validate --emit-attestation --push) into the aggregated, source-keyed layout the corroboration dashboard reads. GP2 is that bridge — with verify-before-count: signature, issuer, identity, and source registry are all checked in a credential-free step before any results are recorded, so unverified evidence never reaches the model and contributors never hold bucket credentials.

Fixes: #1402
Related: #1400 (parent), #1404 / #1483 (GP4 consumer)

Type of Change

  • New feature (non-breaking change that adds functionality)
  • Build/CI/tooling

Component(s) Affected

  • Docs/examples (docs/)
  • Other: evidence ingest (pkg/evidence/project, tools/evidence-project), CI (.github/workflows/evidence-ingest.yaml, uat-{aws,gcp}.yaml, composite action)

Implementation Notes

  • pkg/evidence/project — pure, offline synthesis library. Derives the recipe coordinate + signer id-hash and writes results/<group>/<dashboard>/<tab>/<idHash>/<runId>/{meta.json, ctrf/<phase>.json} (schema aicr-corroboration-meta/v1). Deterministic, idempotent-replace; fails closed when no verified signer is present.
  • tools/evidence-project — ingest CLI: enforce a trusted-registry allowlist, materialize once, verify on the unpacked dir with non-empty issuer/identity pins (pkg/evidence/verifier), classify, then synthesize. Holds no bucket credentials.
  • evidence-ingest.yaml (+ evidence-ingest.sh) — two triggers (push to recipes/evidence/**; workflow_call/dispatch with bundle_ref). Credential-free verify job; separate fork-gated publish job that gcloud storage rsyncs to gs://aicr-testgrid-staging/results via the eidosx WIF SA from uat-gcp.yaml.
  • UAT wiringuat-aws.yaml/uat-gcp.yaml call the ingest workflow as a dependent ingest-evidence job on a successful run, via the shared export-evidence-ref composite action; runId is set from the GHA run ID (the same one embedded in the OCI artifact), keeping the entry unique and traceable.
  • Allowlist — loader/classifier shares the GP1/GP4 recipes/evidence/allowlist.yaml schema (firstParty/community/partner, exact-or-^…$-regex identity, over-broad + overlap lint) so producer and consumer classify a signer identically. A documented interim first-party heuristic applies only while that file is absent.
  • idHash = first 12 hex of sha256(issuer + "\n" + identity); the GP4 consumer treats it as opaque.

Known / forward-looking limitations

  • Statement-only community bundles whose signer is verified against Rekor by digest are not yet ingestable — today's verifier yields a verified signer only from an in-artifact attestation.intoto.jsonl, so such bundles fail closed.
  • The publish job uses the shared github-actions@eidosx SA pending GP3's dedicated objectCreator-only identity.
  • Discovers the flat recipes/evidence/<recipe>.yaml layout today; forward-compatible with GP1's nested per-source glob.

Testing

# Run locally on the rebased base:
go build ./...                                                  # ok
go test ./pkg/evidence/project/... ./tools/evidence-project/... ./pkg/evidence/verifier/...
#   project 88.7%  cli 67.4%  verifier 67.0%
go vet ./pkg/evidence/project/... ./tools/evidence-project/...  # clean
golangci-lint run -c .golangci.yaml ./pkg/evidence/project/... ./tools/evidence-project/...  # 0 issues
actionlint + yamllint (3 workflows + composite action + labeler) # clean
check-docs-filenames / check-docs-mdx / check-agents-sync       # ok

-race, the project-wide make test-coverage floor, and the e2e/grype legs of make qualify were not run locally (no C compiler / Docker in the dev sandbox); the new Go code is goroutine-free and CI runs the full gate.

Risk Assessment

  • Medium — Touches the AWS/GCP UAT workflows (adds an output + a dependent job) and introduces a CI job that writes to a shared GCS bucket.

Rollout notes: The new publish job is gated to the canonical repo (never forks) and writes only under a fresh results/ prefix that does not collide with the bucket's existing paths (config, grid/, groups/, queue/, summary/, tabs/). The UAT ingest-evidence job only runs when a bundle was produced; it is additive and skips cleanly otherwise.

Checklist

  • Tests pass locally (make test-race deferred to CI; no C compiler in dev sandbox)
  • Linter passes (make lint — pinned golangci-lint 2.12.2, yamllint, actionlint)
  • I did not skip/disable tests to make CI green
  • I added/updated tests for new functionality
  • I updated docs if user-facing behavior changed
  • Changes follow existing patterns in the codebase
  • Commits are cryptographically signed (git commit -S)

…IA#1402)

Add the GP2 producer that turns published, signed recipe-evidence
bundles into the source-keyed tree the GP4 corroborate generator
consumes, and publishes it to the corroboration bucket.

- pkg/evidence/project: pure, offline synthesis library — derive the
  recipe coordinate + signer id-hash and write
  results/<group>/<dashboard>/<tab>/<idHash>/<runId>/{meta.json,
  ctrf/<phase>.json}. Deterministic, idempotent-replace; fails closed
  when no verified signer is present. Allowlist loader/classifier shares
  the GP1/GP4 recipes/evidence/allowlist.yaml schema so producer and
  consumer classify a signer identically.
- tools/evidence-project: ingest CLI — enforce a trusted-registry
  allowlist, materialize once, verify on the unpacked dir with non-empty
  issuer/identity pins, classify, then synthesize.
- .github/workflows/evidence-ingest.yaml (+ evidence-ingest.sh): two
  triggers (push to recipes/evidence/**; workflow_call/dispatch with a
  bundle_ref). Credential-free verify job; separate fork-gated publish
  job that rsyncs to gs://aicr-testgrid-staging/results via the eidosx
  WIF SA.
- uat-aws.yaml / uat-gcp.yaml: call the ingest workflow as a dependent
  job on a successful run, via the export-evidence-ref composite action.
- docs/contributor/evidence-ingest.md; labeler tools/evidence-project.

Forward-compatible with GP1 (flat pointers today; allowlist schema
matched). Statement-only/Rekor-by-digest community bundles are not yet
ingestable — the verifier lacks that path; producer fails closed.
@njhensley njhensley requested review from a team as code owners June 26, 2026 07:54
@njhensley njhensley added the theme/supply-chain SLSA, SBOM, Sigstore, and provenance verification label Jun 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@njhensley njhensley self-assigned this Jun 26, 2026
@coderabbitai

coderabbitai Bot commented Jun 26, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR adds a signed-evidence ingest pipeline. It introduces signer classification and signer ID hashing, deterministic evidence metadata and synthesis of source-keyed result directories, a CLI that verifies bundles then synthesizes outputs, and tests for these behaviors. It also adds an export-ref action, ingestion shell script/workflow, UAT wiring, and docs.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related PRs

  • NVIDIA/aicr#1416: Produces the UAT conformance evidence bundle and pointer that this PR exports and ingests.
  • NVIDIA/aicr#1479: Adjusts the UAT conformance evidence push layout that determines the bundle pointer consumed here.

Suggested labels

area/cli, area/tests, theme/ci-dx

Suggested reviewers

  • mchmarny
  • lalitadithya
🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning Most GP2 requirements are implemented, but the durable ORAS mirror/backup step from the linked issue is not present in the described changes. Add an ORAS copy/mirror step to durable storage for each bundle before verification, and wire it into the ingest workflow/CLI.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the GP2 evidence ingest change.
Description check ✅ Passed The description is directly about verifying signed evidence bundles and publishing a source-keyed ingest flow.
Out of Scope Changes check ✅ Passed The changes stay within evidence ingest, supporting CI/workflow wiring, docs, and related labeler updates.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 12

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/actions/export-evidence-ref/action.yml:
- Around line 41-49: The bundle reference handling in the export-evidence-ref
action should distinguish between a truly missing bundle and a partial one.
Update the logic around the `oci`/`digest` checks so `exit 0` only happens when
both `bundle.oci` and `bundle.digest` are absent; if `bundle.oci` exists but
`bundle.digest` is missing or malformed, fail the step instead of skipping
ingest. Keep the output formatting in the same action flow (`oci`, `digest`,
`repo`, and the `GITHUB_OUTPUT` write) but add a validation path that rejects
unpinned/tag-only bundle refs.

In @.github/workflows/evidence-ingest.yaml:
- Around line 185-194: The Sync source-keyed tree to GCS step uses root-level
gcloud storage rsync without removing stale objects, so re-ingesting the same
run can leave old ctrf/*.json files behind. Update this workflow step so the
per-run directory is replaced in place during upload, using a delete-aware sync
strategy scoped to the run/output tree while preserving other sources and
prefixes; keep the change localized to the Sync source-keyed tree to GCS block
and the rsync invocation.
- Around line 96-100: The checkout in the evidence ingest workflow only fetches
the last two commits, which can miss the real push range when the event spans
more history. Update the Checkout step to either fetch full history or
explicitly fetch the commit referenced by github.event.before before the ingest
script runs, so the push-mode diff logic can compare against the correct base.
Use the existing Checkout step and the downstream ingest script trigger as the
key places to adjust.

In `@docs/contributor/evidence-ingest.md`:
- Around line 18-35: Add a language tag to the fenced block(s) in the
contributor evidence ingest doc so markdownlint stops flagging them; update the
diagram/code fence around the pipeline illustration to use a text fence, and
make the same change for the other referenced fence in this section. Use the
existing fenced-block content in the document and keep the formatting otherwise
unchanged.

In `@pkg/evidence/project/classify.go`:
- Around line 56-60: The nil-allowlist fallback in classify.go is too broad
because firstPartyIdentity only matches the repository prefix, so Classify(nil,
...) can incorrectly mark any NVIDIA/aicr workflow path as first-party. Tighten
the interim heuristic in Classify and/or firstPartyIdentity so it only
recognizes the intended canonical workflow identity (not arbitrary paths or refs
under the repo), and keep the fallback behavior aligned with the documented
allowlist replacement used by the ingest path.
- Around line 153-159: The overlap check in overlaps() is too weak because it
only compares representativeIdentity() samples, so Validate() can miss
intersecting regex allowlist entries and later Classify() depends on class
order. Update the overlap detection to either reject unsupported regexp
constructs in allowlist patterns or perform a real intersection test over the
parsed regexp AST instead of relying on a single synthesized identity. Keep the
fix centered around overlaps(), representativeIdentity(), Validate(), and the
classification path used by Classify().

In `@pkg/evidence/project/idhash_test.go`:
- Around line 22-56: Add a golden assertion to the SignerIDHash tests so the
issuer/identity pair in TestSignerIDHash is checked against one fixed expected
hash value, not just shape and relative behavior. Keep the existing determinism
and collision checks, but pin a single known output for the SignerIDHash
function to lock the persisted contract against changes to idHashLen, the
separator, or the encoding.

In `@pkg/evidence/project/idhash.go`:
- Around line 22-26: The signer hash truncation in idHashLen is too short for a
durable bucket/dedup key and can merge distinct verified signers downstream.
Update the signer hashing logic in idHashLen and any helper in idhash.go to
retain at least 128 bits, or use the full digest, so the results/.../<idHash>/
subtree and GP4 dedup key stay unique and collision-resistant.

In `@pkg/evidence/project/synthesize.go`:
- Around line 186-206: The idempotent replace flow in synthesize should be made
atomic instead of deleting the existing runDir up front. Update the run
directory setup in synthesize to stage all output into a sibling temporary
directory, run copyCTRFReports and write meta.json there, then replace runDir
with a single os.Rename only after the full tree is complete. Keep the existing
cleanup/error handling around the staging dir so a cancellation or I/O failure
cannot remove the last good result.
- Around line 215-218: The bundle-local file reads in readRecipeView and
copyCTRFReports can still follow symlinks outside bundleDir, so harden both
paths before opening anything. Add a symlink/realpath check for the recipe and
each ctrf phase file, and reject any target that resolves outside the verified
bundle directory before calling readBoundedFile, os.Stat, or copying. Use the
existing readRecipeView and copyCTRFReports helpers to centralize the guard so
all bundle file accesses stay confined to bundleDir.

In `@tools/evidence-project/main.go`:
- Around line 117-121: The ref validation in main.go is letting
--allow-unpinned-tag bypass checkTrustedRegistry for all remote refs, including
digest-pinned refs. Update the ref handling so the allow-unpinned escape hatch
only skips the unpinned-tag restriction, while pinned refs still go through
checkTrustedRegistry in the existing ref and checkTrustedRegistry flow. Add a
regression test around the main/o trust check path to confirm a digest-pinned
oci:// ref is still rejected when its registry is not on the allowlist, even if
--allow-unpinned-tag is set.
- Around line 64-82: The main entrypoint currently passes context.Background()
into run, which leaves registry and filesystem I/O unbounded and can hang the
CLI. Update main to create a signal-aware root context and wrap it with a
timeout from pkg/defaults before invoking run, keeping the change localized
around main and the run(context, o, os.Stdout) call. Ensure the timeout-bounded
context is used for all downstream ingest work so pull/verify operations inherit
cancellation and deadline handling.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: e438f24f-ff60-49e9-ae2b-a29701140989

📥 Commits

Reviewing files that changed from the base of the PR and between 6fc3444 and 723dab7.

📒 Files selected for processing (18)
  • .github/actions/export-evidence-ref/action.yml
  • .github/labeler.yml
  • .github/scripts/evidence-ingest.sh
  • .github/workflows/evidence-ingest.yaml
  • .github/workflows/uat-aws.yaml
  • .github/workflows/uat-gcp.yaml
  • docs/contributor/evidence-ingest.md
  • docs/index.yml
  • pkg/evidence/project/classify.go
  • pkg/evidence/project/classify_test.go
  • pkg/evidence/project/doc.go
  • pkg/evidence/project/idhash.go
  • pkg/evidence/project/idhash_test.go
  • pkg/evidence/project/meta.go
  • pkg/evidence/project/synthesize.go
  • pkg/evidence/project/synthesize_test.go
  • tools/evidence-project/main.go
  • tools/evidence-project/main_test.go

Comment thread .github/actions/export-evidence-ref/action.yml
Comment thread .github/workflows/evidence-ingest.yaml Outdated
Comment thread .github/workflows/evidence-ingest.yaml Outdated
Comment on lines +18 to +35
```
pointer / bundle ref
materialize (ORAS pull, digest-pinned)
verify ── issuer + identity pins, signature, registry allowlist
│ (credential-free; fails closed)
classify (allowlist → class + allowlisted)
synthesize results/<group>/<dashboard>/<tab>/<idHash>/<runId>/
│ meta.json + ctrf/<phase>.json
publish (separate, credentialed step) → GCS
```

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add language tags to these fenced blocks.

markdownlint will keep flagging both fences until they declare a language. text is enough for the pipeline diagram and the hash formula block.

Suggested fix
-```
+```text
 pointer / bundle ref
         │
         ▼
@@
-```
+```text

Also applies to: 99-101

🧰 Tools
🪛 markdownlint-cli2 (0.22.1)

[warning] 18-18: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/contributor/evidence-ingest.md` around lines 18 - 35, Add a language tag
to the fenced block(s) in the contributor evidence ingest doc so markdownlint
stops flagging them; update the diagram/code fence around the pipeline
illustration to use a text fence, and make the same change for the other
referenced fence in this section. Use the existing fenced-block content in the
document and keep the formatting otherwise unchanged.

Source: Linters/SAST tools

Comment thread pkg/evidence/project/classify.go Outdated
Comment thread pkg/evidence/project/idhash.go Outdated
Comment thread pkg/evidence/project/synthesize.go Outdated
Comment thread pkg/evidence/project/synthesize.go
Comment thread tools/evidence-project/main.go
Comment thread tools/evidence-project/main.go Outdated
Address review comments on the GP2 evidence-ingest producer:

- export-evidence-ref: fail closed on a partial bundle (oci without
  digest, or a non-sha256 digest) instead of silently skipping ingest;
  only skip when both fields are absent.
- evidence-ingest.yaml: full-history checkout so a multi-commit push diff
  resolves its base; replace each run prefix with a delete-aware rsync so
  a re-ingest drops stale ctrf reports without touching other sources.
- project.Classify: tighten the nil-allowlist first-party heuristic to the
  canonical uat-(aws|gcp) workflow identity, not any path under the repo.
- project.Allowlist.Validate: reject regex identities using constructs the
  overlap check cannot sample (char class, '.', '?', repetition) so
  overlaps() is sound rather than heuristic.
- SignerIDHash: widen the dedup key from 48 to 128 bits so an adversary
  cannot grind a colliding signer identity; pin a golden value in tests.
- project.Synthesize: stage into a temp dir and swap with a single rename
  (atomic idempotent replace); confine every bundle-local read to the
  bundle dir so a packed symlink cannot escape.
- evidence-project CLI: gate digest-pinned refs through the trusted
  registry check even under --allow-unpinned-tag; run under a
  signal-aware, deadline-bounded context.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
docs/contributor/evidence-ingest.md (1)

155-157: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Align the planned GCS role with delete-aware publishing.

The publish workflow now uses rsync --delete-unmatched-destination-objects, so a future objectCreator-only identity would not be able to replace stale per-run objects. Document a prefix-scoped role that also grants object delete, or change the publish strategy before making that IAM switch.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/contributor/evidence-ingest.md` around lines 155 - 157, The publish
workflow description is inconsistent with the new delete-aware rsync behavior
used by the evidence publish path. Update the documentation around the publish
job and the planned identity switch to reflect that the future GCS role must
allow object deletion for the target prefix, or revise the publish approach
before describing an objectCreator-only account. Refer to the publish job
details and the rsync --delete-unmatched-destination-objects behavior when
updating the plan.
♻️ Duplicate comments (2)
pkg/evidence/project/synthesize.go (1)

220-227: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Avoid deleting the prior run before the replacement rename succeeds.

The write phase is staged, but the final swap still does RemoveAll(runDir) before Rename; if that rename fails, the last good run is gone. Rename the existing directory aside first and restore it on failure, then clean the backup only after the new run is in place.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/evidence/project/synthesize.go` around lines 220 - 227, The final swap in
synthesize should not delete the existing run before the replacement is safely
in place. Update the runDir/staging handoff in synthesize so the current
directory is first moved aside with os.Rename, then rename staging into runDir,
and if the second rename fails restore the backup before cleaning it up. Keep
the changes localized around the swap logic that currently uses os.RemoveAll and
os.Rename.
pkg/evidence/project/classify.go (1)

262-265: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Reject or enumerate all alternation branches before overlap checks.

syntax.OpAlternate is still accepted, but overlaps() only samples one representative branch. Same-issuer entries like ^(foo|bar)$ and ^(baz|bar)$ still share bar while validation can miss the overlap, leaving class resolution dependent on entry order. Either reject alternation or enumerate all finite alternatives for pairwise matching.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/evidence/project/classify.go` around lines 262 - 265, The overlap
validation in classify.go still treats syntax.OpAlternate as a single
representative branch, so class resolution can miss shared matches like
same-issuer alternations. Update the logic around overlaps() and the case
handling for syntax.OpAlternate to either reject alternation patterns outright
or expand all finite alternation branches and check them pairwise against
existing entries before accepting a class.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@docs/contributor/evidence-ingest.md`:
- Around line 155-157: The publish workflow description is inconsistent with the
new delete-aware rsync behavior used by the evidence publish path. Update the
documentation around the publish job and the planned identity switch to reflect
that the future GCS role must allow object deletion for the target prefix, or
revise the publish approach before describing an objectCreator-only account.
Refer to the publish job details and the rsync
--delete-unmatched-destination-objects behavior when updating the plan.

---

Duplicate comments:
In `@pkg/evidence/project/classify.go`:
- Around line 262-265: The overlap validation in classify.go still treats
syntax.OpAlternate as a single representative branch, so class resolution can
miss shared matches like same-issuer alternations. Update the logic around
overlaps() and the case handling for syntax.OpAlternate to either reject
alternation patterns outright or expand all finite alternation branches and
check them pairwise against existing entries before accepting a class.

In `@pkg/evidence/project/synthesize.go`:
- Around line 220-227: The final swap in synthesize should not delete the
existing run before the replacement is safely in place. Update the
runDir/staging handoff in synthesize so the current directory is first moved
aside with os.Rename, then rename staging into runDir, and if the second rename
fails restore the backup before cleaning it up. Keep the changes localized
around the swap logic that currently uses os.RemoveAll and os.Rename.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: fd01be36-1578-4193-94b2-908a63f1a292

📥 Commits

Reviewing files that changed from the base of the PR and between 723dab7 and b59bf84.

📒 Files selected for processing (12)
  • .github/actions/export-evidence-ref/action.yml
  • .github/workflows/evidence-ingest.yaml
  • docs/contributor/evidence-ingest.md
  • pkg/defaults/timeouts.go
  • pkg/evidence/project/classify.go
  • pkg/evidence/project/classify_test.go
  • pkg/evidence/project/idhash.go
  • pkg/evidence/project/idhash_test.go
  • pkg/evidence/project/synthesize.go
  • pkg/evidence/project/synthesize_test.go
  • tools/evidence-project/main.go
  • tools/evidence-project/main_test.go

mchmarny
mchmarny previously approved these changes Jun 26, 2026

@mchmarny mchmarny left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GP2 ingest — solid, security-conscious work. Walked the full verify-before-count path and it's sound and fail-closed at every gate:

  • Trust gate (tools/evidence-project/main.go): trusted-registry allowlist fails closed on an empty list, matches on a path-segment boundary (ghcr.io/nvidia won't match ghcr.io/nvidia-evil), and is enforced before any pull. Digest-pinned refs are always gated regardless of --allow-unpinned-tag.
  • Verify before synthesize: issuer + identity pins are required (non-empty), and synthesis refuses ExitInvalid / nil signer / nil predicate. Ingesting ExitValidPhaseFailures is correct — it records attested phase results.
  • Anti-sybil allowlist (classify.go): Classify is nil-safe, Validate rejects over-broad regex constructs and overlapping entries, full-string identity match. The interim first-party heuristic is tightly anchored and only active when the allowlist file is absent.
  • Path safety (synthesize.go): filepath.IsLocal on the recipe-derived name and runId, plus a Rel+IsLocal recheck on ctrf extraction — a verified-but-unallowlisted community bundle can't traverse out of its prefix.
  • Fork safety: verify job holds no GCS creds; the credentialed publish job is separate and repo-gated; the GCS rsync --delete is scoped to a single run prefix.

One non-blocking robustness nit inline (empty-ingest → publish artifact download). Doesn't block merge.

Note: several CI legs (Test, E2E, analyze, Mirror E2E) were still pending at review time — worth a glance before merge, though the change is goroutine-free and the Go lint/CLI-E2E legs already passed.

uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: evidence-tree
path: out

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit (non-blocking): empty-ingest edge. In push mode evidence-ingest.sh exits 0 with no out/results when the push touched only the allowlist, deleted a pointer, or had no changed pointers. The verify job's upload then hits if-no-files-found: ignore → no evidence-tree artifact. But publish (needs: verify) still runs and download-artifact fails hard when the named artifact is absent — a spurious red run on an otherwise-fine push.

The if [ ! -d out/results ] guard in the rsync step never gets reached because download fails first. Options: gate publish on a verify-job output flag (e.g. emit produced=true only when out/results exists), or set the download step tolerant and keep the existing dir guard. The UAT bundle_ref path always produces output, so this only bites the recipes/evidence/** push trigger.

@mchmarny mchmarny enabled auto-merge (squash) June 26, 2026 13:03
@mchmarny mchmarny disabled auto-merge June 26, 2026 13:03
Push-mode ingest legitimately produces no tree (allowlist-only change,
deleted pointer, or no changed pointers). The verify job then skips the
artifact upload, but publish (needs: verify) still ran and its
download-artifact step failed hard on the missing artifact — a spurious
red run on an otherwise-fine push.

Add a verify-job 'produced' output, set true only when out/results holds
files, upload the artifact only then, and gate publish on it. The UAT
bundle_ref path always produces output and is unaffected.

Signed-off-by: Nathan Hensley <[email protected]>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/uat-aws.yaml:
- Around line 303-305: The summary currently rebuilds a mutable run tag instead
of using the validated digest-pinned reference, which can point reviewers at the
wrong artifact. Update the summary generation in uat-aws workflow to use the
exported reference from steps.evidence_ref.outputs.ref everywhere the bundle ref
is printed or linked, and remove any reconstruction of the :run-${{
github.run_id }} tag so the summary always matches the verified evidence bundle.

In @.github/workflows/uat-gcp.yaml:
- Around line 272-276: The summary is still using the mutable `:run-${{
github.run_id }}` OCI tag instead of the digest-pinned ref already exported by
`steps.evidence_ref.outputs.ref`. Update the summary generation in the workflow
to reference `steps.evidence_ref.outputs.ref` wherever the evidence bundle link
is rendered, so the human-facing pointer matches the verified artifact and stays
immutable.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 06d268b6-6996-4e57-9b22-33320602e5f8

📥 Commits

Reviewing files that changed from the base of the PR and between 22adee4 and 4e05fb8.

📒 Files selected for processing (2)
  • .github/workflows/uat-aws.yaml
  • .github/workflows/uat-gcp.yaml

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/uat-aws.yaml:
- Around line 303-305: The summary currently rebuilds a mutable run tag instead
of using the validated digest-pinned reference, which can point reviewers at the
wrong artifact. Update the summary generation in uat-aws workflow to use the
exported reference from steps.evidence_ref.outputs.ref everywhere the bundle ref
is printed or linked, and remove any reconstruction of the :run-${{
github.run_id }} tag so the summary always matches the verified evidence bundle.

In @.github/workflows/uat-gcp.yaml:
- Around line 272-276: The summary is still using the mutable `:run-${{
github.run_id }}` OCI tag instead of the digest-pinned ref already exported by
`steps.evidence_ref.outputs.ref`. Update the summary generation in the workflow
to reference `steps.evidence_ref.outputs.ref` wherever the evidence bundle link
is rendered, so the human-facing pointer matches the verified artifact and stays
immutable.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 06d268b6-6996-4e57-9b22-33320602e5f8

📥 Commits

Reviewing files that changed from the base of the PR and between 22adee4 and 4e05fb8.

📒 Files selected for processing (2)
  • .github/workflows/uat-aws.yaml
  • .github/workflows/uat-gcp.yaml
🛑 Comments failed to post (2)
.github/workflows/uat-aws.yaml (1)

303-305: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Use the exported digest ref in the summary.

Line 355 reconstructs a mutable :run-${{ github.run_id }} tag even though this job already derives a validated, digest-pinned ref via steps.evidence_ref.outputs.ref. That can drift from the actual bundle if the naming scheme changes or the tag is later repointed, so the Actions summary may send reviewers to the wrong artifact while ingest still uses the verified digest.

Suggested change
            if [[ "${{ steps.conformance.outcome }}" == "success" ]]; then
              echo ""
-              echo "**Evidence (OCI):** \`ghcr.io/nvidia/aicr-evidence/h100-eks-ubuntu-training-kubeflow:run-${{ github.run_id }}\`"
+              echo "**Evidence (OCI):** \`${{ steps.evidence_ref.outputs.ref }}\`"
            fi

Also applies to: 355-355

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/uat-aws.yaml around lines 303 - 305, The summary currently
rebuilds a mutable run tag instead of using the validated digest-pinned
reference, which can point reviewers at the wrong artifact. Update the summary
generation in uat-aws workflow to use the exported reference from
steps.evidence_ref.outputs.ref everywhere the bundle ref is printed or linked,
and remove any reconstruction of the :run-${{ github.run_id }} tag so the
summary always matches the verified evidence bundle.
.github/workflows/uat-gcp.yaml (1)

272-276: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Use the exported digest ref in the summary.

Line 326 hardcodes the :run-${{ github.run_id }} tag instead of reusing steps.evidence_ref.outputs.ref. Since the export action already normalizes this to a digest-pinned OCI ref, keeping the summary on the tag makes the human-facing link mutable and prone to drift from the artifact the ingest workflow actually verified and published.

Suggested change
            if [[ "${{ steps.conformance.outcome }}" == "success" ]]; then
              echo ""
-              echo "**Evidence (OCI):** \`ghcr.io/nvidia/aicr-evidence/h100-gke-cos-training-kubeflow:run-${{ github.run_id }}\`"
+              echo "**Evidence (OCI):** \`${{ steps.evidence_ref.outputs.ref }}\`"
            fi

Also applies to: 326-326

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/uat-gcp.yaml around lines 272 - 276, The summary is still
using the mutable `:run-${{ github.run_id }}` OCI tag instead of the
digest-pinned ref already exported by `steps.evidence_ref.outputs.ref`. Update
the summary generation in the workflow to reference
`steps.evidence_ref.outputs.ref` wherever the evidence bundle link is rendered,
so the human-facing pointer matches the verified artifact and stays immutable.

mchmarny
mchmarny previously approved these changes Jun 26, 2026

@mchmarny mchmarny left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verify-before-count is correctly fail-closed and pin-enforced: the ingest CLI requires non-empty issuer + identity pins before any pull, enforces the trusted-registry allowlist on a path-segment boundary before the network is touched, and the sigstore policy actually enforces the pins so an unsigned/non-matching bundle yields no verified signer and synthesizeVerified refuses it. Fork-safety is sound — no pull_request_target, no PR-HEAD checkout, both ingest and upstream UAT jobs gated on github.repository, credential-free verify split from the WIF publish job, and the GCS write scoped to a results/ prefix with delete-confined rsync. Synthesis is pure/offline, deterministic, atomic (stage+rename), and confines bundle-local reads against symlink escape. Test coverage hits the security-critical paths.

Three minor inline notes (two low, one nit) — none blocking. LGTM.

run_dir="$(dirname "${meta}")"
rel="${run_dir#out/results/}"
echo "syncing run ${rel}"
gcloud storage rsync -r \

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the gcloud storage rsync call inside the per-meta.json loop isn't timeout-wrapped, so a wedged object-store call parks the publish step until the job timeout-minutes backstop fires. Low risk — UAT produces one run dir per call — but timeout 300 gcloud storage rsync ... makes a hung call fail fast and loud rather than burning the whole job budget.

defer mat.Cleanup()

// Step 2 — verify on the materialized directory with non-empty pins.
vr, err := verifier.Verify(ctx, verifier.VerifyOptions{

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This Verify runs on mat.BundleDir (a directory), so verifier re-detects the input form, r.Pointer is nil, and CrossCheckPointerSigner does not run on this pass. That's not a hole today — in push mode evidence-ingest.sh derives the identity pin (^<claimed-identity>$) from the pointer's claimed signer, so a lying pointer still fails the identity pin. But a future refactor could wrongly assume the pointer→signer cross-check fires here. Worth a one-line comment noting the cross-check is intentionally covered by the derived pin rather than this pass.

# Discover added/modified pointer files in this push. A deleted pointer
# is absent at HEAD and is skipped (the tree entry it produced is left
# in the bucket; pruning retired sources is a separate concern).
changed=$(git diff --name-only --diff-filter=AM "${BEFORE_SHA}" "${HEAD_SHA}" -- 'recipes/evidence/*.yaml' || true)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: || true swallows a genuine git diff failure (e.g. a bad SHA range) into an empty changed, indistinguishable from "no pointers changed" → exit 0. Soft false-pass. Low risk here since BEFORE_SHA/HEAD_SHA come from trusted github.event.before/github.sha with fetch-depth: 0, but capturing rc and failing on non-zero is the fail-loud form.

@njhensley njhensley merged commit 14e347a into NVIDIA:main Jun 26, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci area/docs size/XL theme/supply-chain SLSA, SBOM, Sigstore, and provenance verification

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GP2 — Ingest CI (pointers → verify-before-count → source-keyed GCS)

2 participants