Summary

Adds a warning-only PR gate that surfaces stale or missing recipe-evidence bundles. For each leaf overlay touched by the diff, the gate computes the current canonical digest, compares it to the signed predicate.recipe.digest (via aicr evidence verify), and posts a sticky comment.

Motivation / Context

PR #1055 landed aicr evidence digest, giving us a cheap local way to detect drift between a signed evidence pointer and the current recipe. This PR wires it into CI so contributors are nudged — not blocked — to refresh evidence when they touch a recipe.

The gate is intentionally warning-only because regenerating evidence requires a cluster matching the recipe's criteria, and most contributors don't have that hardware on hand. Surfacing the drift in the PR comment makes it visible for reviewers and gives the maintainer-with-hardware a chance to refresh before merge.

Fixes: #751
Related: ADR-007, follow-up to #1055

Type of Change

• New feature (non-breaking change that adds functionality)
• Build/CI/tooling

Component(s) Affected

• Other: GitHub Actions workflows + helper shell script (.github/)

Implementation Notes

Three files:

• .github/scripts/recipe-evidence-check.sh — shared gate logic. Discovers leaf overlays touched by the BASE_SHA...HEAD_SHA diff, runs aicr evidence digest + aicr evidence verify against each, builds a Markdown report. Runnable locally.
• .github/workflows/recipe-evidence.yaml — pull_request trigger, read-only token, builds aicr, invokes the script, uploads the report as an artifact.
• .github/workflows/recipe-evidence-comment.yaml — workflow_run trigger, downloads the artifact, posts/updates the sticky PR comment under the standard write-scoped token. PR association is re-derived from workflow_run.head_sha (not the artifact, which is fork-attacker-controllable).

Why the two-workflow split: standard GitHub pattern (matches the existing fern-docs-preview-build + fern-docs-preview-comment). Keeps the PR-comment write token out of the untrusted PR-branch job.

Leaf discovery rules:

1. Overlay file itself changed.
2. Ancestor overlay in the spec.base chain changed.
3. A component valuesFile referenced by this overlay (or any ancestor) changed.

Plus a broad-impact path: if recipes/registry.yaml or recipes/overlays/base.yaml changed, every leaf is potentially affected and the report flags the breadth.

Local invocation (same code paths CI uses):

make build
AICR=$(pwd)/bin/aicr \
  BASE_SHA=$(git merge-base origin/main HEAD) \
  HEAD_SHA=$(git rev-parse HEAD) \
  REPORT_OUT=/tmp/report.md \
  REPO_URL=https://github.com/NVIDIA/aicr \
  .github/scripts/recipe-evidence-check.sh
cat /tmp/report.md

Known limitations (tracked as follow-ups in the script header):

• Pointer file name is computed as basename overlay .yaml, but aicr validate --emit-attestation uses the criteria-derived slug from RecipeNameFor. Non-canonical overlay names will be reported as missing-pointer until they're aligned.
• Discovery walks spec.base ancestors but not spec.mixins; mixin/check/component edits outside literal valuesFile refs aren't promoted. Parity with the existing kwok-recipes Tier 2 classifier.
• aicr evidence verify fetches OCI artifacts from PR-controlled URLs — an allow-list of trusted registries is a follow-up.

Example reports

These cover the three most representative response states. All render under the bot identity as a sticky comment (marker <!-- recipe-evidence-gate -->).

Clean — no recipes touched

Recipe evidence check

No leaf overlays affected by this PR.

This gate is warning-only and never blocks merge.

Single recipe, evidence on file, digest drifted (the marquee case)

Recipe evidence check

Affected leaf overlays: 1

Recipe	Pointer	Verify	Digest match
h100-eks-ubuntu-training	✅ present	✅ passed	⚠️ stale (a3f4c1d2e890… vs current b7e2d0a6f15c…)

How to refresh evidence

Run on a cluster matching the recipe's criteria:

aicr snapshot -o snapshot.yaml
aicr validate \
  -r recipes/overlays/<slug>.yaml \
  -s snapshot.yaml \
  --emit-attestation ./out \
  --push ghcr.io/<your-fork>/aicr-evidence
cp ./out/pointer.yaml recipes/evidence/<slug>.yaml

This gate is warning-only and never blocks merge. See ADR-007 for the trust model.

Broad-impact change (registry.yaml or base.yaml touched)

Recipe evidence check

Broad impact: recipes/registry.yaml or recipes/overlays/base.yaml changed; every leaf recipe is potentially affected. The list below covers all of them — each one would ideally have refreshed evidence before merge.

Affected leaf overlays: 42

Recipe	Pointer	Verify	Digest match
gb200-eks-inference	⚠️ missing	—	—
gb200-eks-training	✅ present	✅ passed	⚠️ stale (a3f4c1d2e890… vs current b7e2d0a6f15c…)
… (38 rows elided in this preview) …
h100-oke-ubuntu-training	⚠️ missing	—	—

Beyond MAX_ROWS (default 80), a truncation row replaces the rest: … +N more (truncated; raise MAX_ROWS or split the PR).

Testing

Ran the script locally against two SHA ranges on this branch:

Pointer-present + verify branch wasn't exercised live — recipes/evidence/ doesn't exist in the tree yet. Path traced by inspection in recipe-evidence-check.sh (lines 205-246).

make qualify: not run (YAML/shell-only change, no Go). yamllint and actionlint run in CI.

Risk Assessment

• Low — Additive CI workflows + helper script. Gate is warning-only: pull-requests: write is on the comment workflow only, posted body is verbatim script output, no failing exit code blocks the merge queue.

Rollout notes: Gate activates automatically on PRs touching recipes/**. No pre-existing pointers means most early PRs will show ⚠️ missing rows — that's expected and serves as a backlog of what to generate.

Checklist

• Tests pass locally (script verified against two ranges)
• Linter passes — N/A locally (no Go); CI runs yamllint + actionlint
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality — script self-tested via local runs
• I updated docs if user-facing behavior changed — gate trailer points to existing ADR-007; contributor-facing nudge lives in the report itself
• Changes follow existing patterns in the codebase (two-workflow split mirrors fern-docs-preview-*)
• Commits are cryptographically signed (git commit -S)