test(evidence): signed recipe-evidence pointers for h100 GKE + EKS inference by yuanchen8911 · Pull Request #1164 · NVIDIA/aicr

yuanchen8911 · 2026-06-03T01:14:37Z

Summary

Adds two signed recipe-evidence v1 pointers generated from live conformance validation on H100 inference clusters, to exercise the warning-only recipe-evidence drift gate (#1065, "Recipe Evidence: Verify").

Recipe	Cluster	OCI digest	Rekor
`h100-gke-cos-inference-dynamo`	GKE / COS (aicr-demo5)	`sha256:6a59465f…`	1706764289
`h100-eks-ubuntu-inference-dynamo`	EKS / Ubuntu (aicr3)	`sha256:da9d8838…`	1706788485

Both bundles were built with aicr validate --phase conformance --emit-attestation against the live clusters (11/11 conformance checks passed on each), then signed + pushed to ghcr.io/nvidia/aicr-evidence via aicr evidence publish (Sigstore keyless OIDC).

Motivation / Context

Validates the recipe-evidence verify pipeline end-to-end on real evidence: the gate should fetch each signed OCI artifact, verify its Sigstore signature, and compare the signed recipe digest against the current recipe.

Refs #1160

Type of Change

CI / evidence (test of the recipe-evidence gate)

Testing

Conformance phase passed 11/11 on both clusters (GKE H100/COS, EKS H100/Ubuntu).
Bundles signed (Sigstore keyless) and pushed to GHCR; pointers pin per-recipe digests.
This PR's purpose is to trigger "Recipe Evidence: Verify" — observe its result.

coderabbitai · 2026-06-03T01:18:20Z

📝 Walkthrough

Walkthrough

Adds two new evidence attestation recipe YAMLs under recipes/evidence: h100-eks-ubuntu-inference-dynamo.yaml and h100-gke-cos-inference-dynamo.yaml. Each file declares schemaVersion and recipe name and contains a single attestations entry recording an OCI bundle reference with sha256 digest, a predicateType URL, signer identity and issuer, a rekorLogIndex, and an attestedAt timestamp.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: adding signed recipe-evidence pointers for two specific H100 inference clusters (GKE and EKS), which matches the changeset of adding two YAML evidence files.
Description check	✅ Passed	The description is directly related to the changeset, explaining what signed recipe-evidence pointers were added, from which clusters, and why they were added to test the recipe-evidence drift gate.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@recipes/evidence/h100-eks-ubuntu-inference-dynamo.yaml`:
- Line 5: Replace the tag-based OCI reference under the oci field with the
provided digest-pinned reference: change "ghcr.io/nvidia/aicr-evidence:v1" to
"ghcr.io/nvidia/aicr-evidence@sha256:da9d8838e1ce4032256b04d75e27dc7248a3fd230bd7fd0f112bd306bc367317"
so the recipe uses an immutable, content-addressable OCI reference that will
pass the aicr evidence verify check (avoid relying on --allow-unpinned-tag).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 61b4da68-c576-412d-a13c-2516cdcd6788

📥 Commits

Reviewing files that changed from the base of the PR and between 4d5ac90 and 9062a04.

📒 Files selected for processing (2)

recipes/evidence/h100-eks-ubuntu-inference-dynamo.yaml
recipes/evidence/h100-gke-cos-inference-dynamo.yaml

coderabbitai · 2026-06-03T01:18:23Z

+recipe: h100-eks-ubuntu-inference-dynamo
+attestations:
+    - bundle:
+        oci: ghcr.io/nvidia/aicr-evidence:v1


⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

OCI reference must be digest-pinned, not tag-based.

The OCI reference uses tag notation (:v1) instead of digest-pinned notation (@sha256:...). Per the PR objectives, this bundle should reference ghcr.io/nvidia/aicr-evidence@sha256:da9d8838e1ce4032256b04d75e27dc7248a3fd230bd7fd0f112bd306bc367317. The aicr evidence verify CLI refuses unpinned tags by default, so verification will fail unless --allow-unpinned-tag is passed. Digest-pinning ensures immutability and content-addressable verification. As per the relevant code snippet from pkg/cli/evidence_verify.go (lines 118-123), the --allow-unpinned-tag flag is described as "Accept tag-only OCI references (default: refuse)."

🔒 Proposed fix to use digest-pinned OCI reference

- oci: ghcr.io/nvidia/aicr-evidence:v1 + oci: ghcr.io/nvidia/aicr-evidence@sha256:da9d8838e1ce4032256b04d75e27dc7248a3fd230bd7fd0f112bd306bc367317

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

oci: ghcr.io/nvidia/aicr-evidence:v1

oci: ghcr.io/nvidia/aicr-evidence@sha256:da9d8838e1ce4032256b04d75e27dc7248a3fd230bd7fd0f112bd306bc367317

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@recipes/evidence/h100-eks-ubuntu-inference-dynamo.yaml` at line 5, Replace the tag-based OCI reference under the oci field with the provided digest-pinned reference: change "ghcr.io/nvidia/aicr-evidence:v1" to "ghcr.io/nvidia/aicr-evidence@sha256:da9d8838e1ce4032256b04d75e27dc7248a3fd230bd7fd0f112bd306bc367317" so the recipe uses an immutable, content-addressable OCI reference that will pass the aicr evidence verify check (avoid relying on --allow-unpinned-tag).

…S inference Commit signed recipe-evidence v1 pointers generated from live conformance validation on two H100 inference clusters, to exercise the warning-only recipe-evidence drift gate (NVIDIA#1065). - h100-gke-cos-inference-dynamo → ghcr.io/nvidia/aicr-evidence@sha256:6a59465f… (Rekor 1706764289) - h100-eks-ubuntu-inference-dynamo → ghcr.io/nvidia/aicr-evidence@sha256:da9d8838… (Rekor 1706788485) Both bundles built via 'aicr validate --phase conformance --emit-attestation' on aicr-demo5 (GKE/COS) and aicr3 (EKS/Ubuntu), signed + pushed via 'aicr evidence publish' (Sigstore keyless). Refs NVIDIA#1160

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@recipes/evidence/h100-gke-cos-inference-dynamo.yaml`:
- Line 5: Replace the tag-based OCI reference used in bundle.oci (the line
containing "oci: ghcr.io/nvidia/aicr-evidence:v1") with a digest-pinned
reference (ghcr...@sha256:<digest>) so the bundle is verifiably pinned; obtain
the canonical image digest (e.g., via docker/skopeo inspect or your registry UI)
and update the value to the `@sha256`:<digest> form to satisfy the default
verification flow that refuses unpinned tags.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: f061ad77-8614-4a2c-bc42-4f320f025fbc

📥 Commits

Reviewing files that changed from the base of the PR and between 9062a04 and de5edd6.

📒 Files selected for processing (2)

recipes/evidence/h100-eks-ubuntu-inference-dynamo.yaml
recipes/evidence/h100-gke-cos-inference-dynamo.yaml

coderabbitai · 2026-06-03T01:54:59Z

+recipe: h100-gke-cos-inference-dynamo
+attestations:
+  - bundle:
+      oci: ghcr.io/nvidia/aicr-evidence:v1


⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

Use a digest-pinned OCI reference on Line 5.

bundle.oci is tag-based (:v1), which is refused by default verification flow unless --allow-unpinned-tag is set.

🔒 Proposed fix

- oci: ghcr.io/nvidia/aicr-evidence:v1 + oci: ghcr.io/nvidia/aicr-evidence@sha256:6a59465f829489b5bf30eb56904817a573864f28afcce5022f2425f022ddb74d

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

oci: ghcr.io/nvidia/aicr-evidence:v1

oci: ghcr.io/nvidia/aicr-evidence@sha256:6a59465f829489b5bf30eb56904817a573864f28afcce5022f2425f022ddb74d

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@recipes/evidence/h100-gke-cos-inference-dynamo.yaml` at line 5, Replace the tag-based OCI reference used in bundle.oci (the line containing "oci: ghcr.io/nvidia/aicr-evidence:v1") with a digest-pinned reference (ghcr...@sha256:<digest>) so the bundle is verifiably pinned; obtain the canonical image digest (e.g., via docker/skopeo inspect or your registry UI) and update the value to the `@sha256`:<digest> form to satisfy the default verification flow that refuses unpinned tags.

yuanchen8911 · 2026-06-03T02:11:53Z

Closing — this was a test PR to exercise the recipe-evidence gate (#1065), not intended for merge. It did its job and surfaced three findings:

The gate passes vacuously on evidence-only PRs ("Affected leaf overlays: 0" → aicr evidence verify never runs, since the gate only verifies recipes the PR changes).
aicr evidence publish writes pointer.yaml at 4-space indent, which fails the repo's yamllint → fix up as fix(evidence): emit recipe-evidence pointer.yaml at 2-space indent #1165.
The pointer's bundle.oci is tag-based (:v1), which aicr evidence verify refuses by default, and the shared :v1 tag collides across recipes (CodeRabbit flagged this).

The evidence here was also generated from recipes carrying a local allowedSourceRanges override that isn't in the repo, so it isn't canonical. Real committable evidence should be regenerated from the canonical recipe with digest-pinned refs. Closing without merge.

MarshalPointer used yaml.v3's default 4-space sequence indentation, so the committed recipes/evidence/<recipe>.yaml pointer failed the repo's yamllint (.yamllint spaces: 2), breaking the documented publish -> commit -> PR flow (make lint failed on every generated pointer). Switch to a yaml.v3 encoder with SetIndent(2). Affects both 'aicr validate --emit-attestation' and 'aicr evidence publish' (both call WritePointer -> MarshalPointer). Adds TestMarshalPointer_TwoSpaceIndent regression guard. Refs NVIDIA#1164

yuanchen8911 requested a review from a team as a code owner June 3, 2026 01:14

github-actions Bot added area/recipes size/S labels Jun 3, 2026

coderabbitai Bot reviewed Jun 3, 2026

View reviewed changes

yuanchen8911 force-pushed the test/recipe-evidence-h100-gke-eks branch from 9062a04 to de5edd6 Compare June 3, 2026 01:53

coderabbitai Bot reviewed Jun 3, 2026

View reviewed changes

yuanchen8911 mentioned this pull request Jun 3, 2026

fix(evidence): emit recipe-evidence pointer.yaml at 2-space indent #1165

Merged

1 task

yuanchen8911 closed this Jun 3, 2026

yuanchen8911 mentioned this pull request Jun 3, 2026

ci: flag pointer-only changes in recipe-evidence gate #1166

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(evidence): signed recipe-evidence pointers for h100 GKE + EKS inference#1164

test(evidence): signed recipe-evidence pointers for h100 GKE + EKS inference#1164
yuanchen8911 wants to merge 1 commit into
NVIDIA:mainfrom
yuanchen8911:test/recipe-evidence-h100-gke-eks

yuanchen8911 commented Jun 3, 2026

Uh oh!

coderabbitai Bot commented Jun 3, 2026 •

edited

Loading

Walkthrough

Estimated code review effort

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jun 3, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jun 3, 2026

Uh oh!

yuanchen8911 commented Jun 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	oci: ghcr.io/nvidia/aicr-evidence:v1
	oci: ghcr.io/nvidia/aicr-evidence@sha256:da9d8838e1ce4032256b04d75e27dc7248a3fd230bd7fd0f112bd306bc367317

Uh oh!

Conversation

yuanchen8911 commented Jun 3, 2026

Summary

Motivation / Context

Type of Change

Testing

Uh oh!

coderabbitai Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Estimated code review effort

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

yuanchen8911 commented Jun 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Jun 3, 2026 •

edited

Loading