Skip to content

feat(evidence-gate): protected/other partition + component-scoped cascade#1448

Merged
mchmarny merged 2 commits into
mainfrom
feat/evidence-gate-report-tuning
Jun 24, 2026
Merged

feat(evidence-gate): protected/other partition + component-scoped cascade#1448
mchmarny merged 2 commits into
mainfrom
feat/evidence-gate-report-tuning

Conversation

@mchmarny

Copy link
Copy Markdown
Member

Summary

Tunes the warning-only recipe-evidence gate (.github/scripts/recipe-evidence-check.sh) so broad-impact PRs stop rendering an alarming wall of "missing"/"drifted" rows. Three changes: a protected-vs-other partition, a component-scoped registry cascade, and classified verify-failure causes in the comment.

Motivation / Context

Part of the evidence-gate pilot epic. PR #1418 (an aws-efa-only registry bump) promoted ~all leaf recipes to "affected" and #1427 surfaced an unhelpful bare "invalid". This is the gate-report half of that tuning; it consumes the failureCause JSON contract added in #1445.

Fixes: #1432, #1435, #1437
Related: #1431

Type of Change

  • New feature (non-breaking change that adds functionality)
  • Documentation update

Component(s) Affected

  • Docs/examples (docs/, examples/)
  • Other: CI gate script (.github/scripts/recipe-evidence-check.sh) + ADR-007

Implementation Notes

Testing

Local smoke tests (built aicr, temp commits, recipe-evidence-check.sh):

Scenario Result
aws-efa-only registry edit 23 affected = 1 protected (gb200-eks-ubuntu-training) + 22 EKS collapsed; GKE h100-gke-cos-training not affected
base.yaml edit (broad) 63 affected → 2 protected + 61 collapsed; warnings 63 → 4
single overlay edit Rules 1–4 still scope correctly (3 affected)

bash -n + shellcheck -S warning clean. (No Go changed; the consumed failureCause/pending JSON shipped in #1445.)

Risk Assessment

  • Low — Warning-only gate; never blocks merge. Changes only the CI script + a doc. Worst case is over/under-listing in a PR comment, with the digest comparison as ground truth.

Rollout notes: No config or workflow changes; the workflow invokes the same script with the same env contract.

Checklist

  • Linter passes (shellcheck, bash -n)
  • I did not skip/disable tests to make CI green
  • I updated docs if user-facing behavior changed (ADR-007)
  • Changes follow existing patterns in the codebase
  • Commits are cryptographically signed (git commit -S)

@mchmarny mchmarny requested a review from a team as a code owner June 24, 2026 16:03
@mchmarny mchmarny added the theme/ci-dx CI pipelines, developer experience, and build tooling label Jun 24, 2026
@mchmarny mchmarny requested a review from a team as a code owner June 24, 2026 16:03
@mchmarny mchmarny added theme/supply-chain SLSA, SBOM, Sigstore, and provenance verification theme/ci-dx CI pipelines, developer experience, and build tooling labels Jun 24, 2026
@mchmarny mchmarny self-assigned this Jun 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

Copy link
Copy Markdown
Contributor

Recipe evidence check

No leaf overlays affected by this PR.

This gate is warning-only and never blocks merge.

@coderabbitai

coderabbitai Bot commented Jun 24, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The recipe-evidence CI gate now scopes registry-change promotion to recipes whose resolved components intersect changed registry entries, separates affected recipes into protected and other affected groups, and renders the report with conditional protected output and a collapsed section for recipes without evidence. It also classifies aicr evidence verify --format json results with structured failure fields and updates the design document to match the implemented behavior.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related issues

Possibly related PRs

  • NVIDIA/aicr#1166: Both PRs modify .github/scripts/recipe-evidence-check.sh around evidence-pointer handling and affected-recipe/report generation logic.
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title matches the main changes: protected/other partition and registry-scoped cascade.
Description check ✅ Passed The description describes the same gate-report and ADR-007 updates as the changeset.
Linked Issues check ✅ Passed The script now treats committed evidence pointers as protected, separates other affected recipes, and documents the behavior in ADR-007.
Out of Scope Changes check ✅ Passed No unrelated code changes are evident beyond the gate script and its supporting ADR update.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/evidence-gate-report-tuning

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/scripts/recipe-evidence-check.sh:
- Around line 266-279: Update the protected/others partitioning in
recipe-evidence-check.sh so deleted evidence pointers still count as protected:
when building the protected set in the affected-recipes loop, check whether
recipes/evidence/${slug}.yaml exists in either the BASE commit or the current
HEAD working tree, not just HEAD. Use the existing protected/others jq
accumulation logic and the slug loop, but adjust the file-presence test so
pointer deletions remain in protected and continue to be verified.
- Around line 397-443: The verifier handling in recipe-evidence-check.sh is
treating exit 1 as a generic verify error, but the contract says valid bundles
with recorded phase failures should be classified as Exit 1 and invalid bundles
as Exit 2. Update the JSON parsing in the verification block to read the `.exit`
field from the verifier output, then branch on that value in the `case`/status
mapping so Exit 1 produces the phase-failure informational path and Exit 2
remains the invalid-bundle path; keep the existing `signed_digest`, `pending`,
and `failureCause` handling in the same flow and reference the `verify_json`,
`verify_exit`, and `verify_cell` logic when making the change.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 1e168d10-38e8-4e57-9ae5-61ad26884e33

📥 Commits

Reviewing files that changed from the base of the PR and between e878161 and e3dad1f.

📒 Files selected for processing (2)
  • .github/scripts/recipe-evidence-check.sh
  • docs/design/007-recipe-evidence.md

Comment thread .github/scripts/recipe-evidence-check.sh
Comment thread .github/scripts/recipe-evidence-check.sh Outdated
@github-actions

Copy link
Copy Markdown
Contributor

Coverage Report ✅

Metric Value
Coverage 77.3%
Threshold 75%
Status Pass
Coverage Badge
![Coverage](https://img.shields.io/badge/coverage-77.3%25-green)

No Go source files changed in this PR.

@njhensley njhensley left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review — multi-persona, meta-confirmed

Method: four persona passes (shell-correctness, security, CI/DX, domain-correctness), each finding independently confirmed/refuted by a senior meta-reviewer against the resolved code. Tier legend: 🔴 Blocker · 🟠 Major · 🟡 Minor · 🔵 Nitpick.

Overall — no blockers; two Majors to resolve or defer in writing

Clean, well-commented tuning of a warning-only, never-blocking gate, and the noise-reduction direction is right. The privilege split (unprivileged pull_request build + workflow_run comment with no PR checkout) is implemented correctly — the most important control here, and it's sound. Nothing in this PR can break CI or merge a bad artifact.

Two Major items keep it from a clean approve (both inline):

  1. The component-scoped cascade walks a different inheritance graph than the one the digest is computed over → systematic false-negatives on registry-only edits to shared/base/wildcard components (the exact case the feature targets).
  2. The classified-cause/pending rendering reads verifier JSON fields that don't exist yet → graceful dead code that the ADR nonetheless marks "(implemented)".

🟡 Minor (not anchorable inline — file not in this diff)

Stale "Known limitations" comment in .github/workflows/recipe-evidence.yaml L32–35 still says discovery "walks spec.base ancestors but not spec.mixins" and "promote-all only matches registry.yaml or base.yaml" — both made false by this PR. The script header was updated; the duplicated workflow copy was not.

✅ Confirmed non-issues (examined and cleared)

recipe_components() has its own base-chain cycle guard (seen[]); removing the [[ ! -f "$pointer" ]] branch is safe (non-pointer recipes route to the others partition; a vanished pointer fails-closed to :x: verify error); mktemp files are cleaned on every continue; comm -12 | grep -q is correct under pipefail; the new protected=/other_affected= stdout keys break no consumer; the truncation row was correctly updated to 3 columns; the lower MAX_ROWS header comment was updated; trigger-model split is correct.

Summary

Tier Count
🔴 Blocker 0
🟠 Major 2
🟡 Minor 6
🔵 Nitpick 2

Recommendation: not a blocker for a warning-only gate, but please either fix the cascade graph mismatch (it defeats the feature's core purpose for registry edits) or scope it down + document the gap, and reconcile the dead-code vs. "implemented" ADR before merge.

Comment thread .github/scripts/recipe-evidence-check.sh
Comment thread .github/scripts/recipe-evidence-check.sh
Comment thread .github/scripts/recipe-evidence-check.sh Outdated
Comment thread .github/scripts/recipe-evidence-check.sh
Comment thread .github/scripts/recipe-evidence-check.sh Outdated
Comment thread .github/scripts/recipe-evidence-check.sh Outdated
Comment thread .github/scripts/recipe-evidence-check.sh Outdated
Comment thread .github/scripts/recipe-evidence-check.sh Outdated
Comment thread docs/design/007-recipe-evidence.md Outdated
@mchmarny mchmarny force-pushed the feat/evidence-gate-report-tuning branch 2 times, most recently from 4f75ff9 to 1894bfc Compare June 24, 2026 17:03

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/scripts/recipe-evidence-check.sh:
- Around line 157-159: The failure handling in recipe_components currently
treats any aicr recipe timeout, resolver error, or jq parse failure as an empty
component list, which can cause Rule 5 to skip a recipe instead of
conservatively including it. Update recipe_components to fail open by surfacing
resolution/parsing failures in a way that Rule 5 can detect, and adjust the Rule
5 logic that consumes recipe_components so unresolved recipes are included
rather than silently skipped; use the existing recipe_components function and
the Rule 5 check in this script as the main touchpoints.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: aa5615c5-0742-42f0-9a1e-6463e0fd4692

📥 Commits

Reviewing files that changed from the base of the PR and between 4f75ff9 and 1894bfc.

📒 Files selected for processing (2)
  • .github/scripts/recipe-evidence-check.sh
  • docs/design/007-recipe-evidence.md

Comment thread .github/scripts/recipe-evidence-check.sh Outdated
@mchmarny mchmarny force-pushed the feat/evidence-gate-report-tuning branch from 1894bfc to ff0b4b6 Compare June 24, 2026 17:37

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/scripts/recipe-evidence-check.sh:
- Line 280: The component-resolution warning in the recipe evidence check is
writing the PR-controlled overlay value directly into a GitHub workflow command.
Update the warning path in recipe-evidence-check.sh to pass the overlay through
the existing sanitizer before emitting the ::warning:: message, using the same
helper used elsewhere for workflow-command output, so the warning text cannot be
interpreted as a command.
- Around line 121-122: The component lookup in the recipe evidence script uses
env(CN), which can coerce scalar-like names and miss matches; update the yq
selections in the lookup logic to use strenv(CN) instead. Apply this change
consistently in the component-name matching expressions for both the base and
head registry queries in the script.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 51eefa7a-0045-4e20-9c7f-a8aa7daf881e

📥 Commits

Reviewing files that changed from the base of the PR and between 1894bfc and ff0b4b6.

📒 Files selected for processing (2)
  • .github/scripts/recipe-evidence-check.sh
  • docs/design/007-recipe-evidence.md

Comment thread .github/scripts/recipe-evidence-check.sh Outdated
Comment thread .github/scripts/recipe-evidence-check.sh Outdated
njhensley
njhensley previously approved these changes Jun 24, 2026

@njhensley njhensley left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approve with notes

Re-reviewed the force-push (e3dad1f1894bfc) and verified each fix against the code. 8 of 10 findings fixed, both Majors addressed substantively — approving. The two Majors in particular:

  • Cascade graph mismatch — fully fixed: recipe_components() now resolves via aicr recipe --format json (the same criteria-resolution engine the digest uses); jq path confirmed correct against RecipeResult (pkg/recipe/metadata.go:415). The base/wildcard false-negative is closed.
  • Classified causes / pending — improved (now branches on the real VerifyResult.Exit rather than OS-exit heuristics, no regression), but the pending/failureCause fields still aren't emitted by the verifier, so that rendering stays inert until the #1437 Go half. See inline.

Also confirmed fixed: cosmetic-edit banner guard, double-counted warnings (verify_ok), Markdown escaping (md_escape), yq env(CN), OTHER_MAX cap, log_sanitize on ::warning:: echoes, ADR "matches" wording. Bonus: the de-protection handling (BASE-pointer present, deleted at HEAD → :warning: evidence pointer removed) is a good catch beyond the review.

Two non-blocking items remain (fine as follow-ups)

  1. 🟠 ADR overstates classified causes as "(implemented)" while the verifier doesn't emit failureCause/pending yet — land the Go fields or soften the wording (inline).
  2. 🟡 Stale comment in .github/workflows/recipe-evidence.yaml:32-33 still says discovery "walks spec.base ancestors but not spec.mixins" and "promote-all only matches registry.yaml" — both now false (resolution is via aicr; registry is component-scoped). The script's own header note was correctly removed; this duplicated copy was missed.

None of these block a warning-only gate. Thanks for the thorough turnaround.

Comment thread .github/scripts/recipe-evidence-check.sh
Comment thread .github/scripts/recipe-evidence-check.sh
Comment thread docs/design/007-recipe-evidence.md
@mchmarny mchmarny force-pushed the feat/evidence-gate-report-tuning branch from ff0b4b6 to b909a03 Compare June 24, 2026 17:47

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/scripts/recipe-evidence-check.sh:
- Around line 72-77: PR-controlled text is still reaching workflow commands
unsanitized in the annotation paths, specifically where `overlay` and `pointer`
are embedded in `::warning::` output. Update the warning/error construction in
`recipe-evidence-check.sh` so every PR-derived value goes through `log_sanitize`
before being echoed, and extend `log_sanitize` to collapse carriage returns as
well as newlines while neutralizing `::`. Use the existing `log_sanitize`,
`digest_err`, and `verify_err` handling as the pattern for the remaining
annotation messages.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 33a7a52e-a29a-43d2-87ea-9adfb5543951

📥 Commits

Reviewing files that changed from the base of the PR and between ff0b4b6 and b909a03.

📒 Files selected for processing (2)
  • .github/scripts/recipe-evidence-check.sh
  • docs/design/007-recipe-evidence.md

Comment thread .github/scripts/recipe-evidence-check.sh
…ade, classified causes

Tune the warning-only recipe-evidence gate so broad-impact PRs stop
rendering an alarming wall of "missing"/"drifted" rows.

Protected vs. other affected (#1432):
- A recipe is "protected" implicitly — iff it has a committed pointer in
  recipes/evidence/. The gate verifies the protected set (status table)
  and collapses affected-but-unevidenced recipes into a single <details>
  count instead of per-recipe "missing" warnings. No new files to
  maintain; the set grows as evidence is added.

Component-scoped registry cascade (#1435):
- A recipes/registry.yaml change no longer promotes every leaf. The gate
  diffs the registry at the component-entry level and marks a recipe
  affected only if its resolved component set (walked across base chain +
  spec.mixins) intersects the changed entries — so an aws-efa-only edit
  flags only recipes that reference aws-efa. base.yaml stays broad. The
  recomputed digest is still ground truth (match => not drift).

Classified verify failures (#1437, script half):
- Parse the structured failureCause {class, httpStatus, hint} and pending
  flag from `aicr evidence verify --format json` and render an actionable
  cause in the verify column (e.g. "invalid — registry-forbidden (HTTP
  403): make the fork's aicr-evidence package public"); echo the full
  error into the job log.

Docs: ADR-007 gains a "Gate report structure" subsection describing the
implicit-protected definition, component-scoped cascade, and classified
causes.

Smoke-tested locally: aws-efa-only edit flags 23 EKS recipes (1 protected
+ 22 collapsed) and leaves GKE/OCI recipes untouched; base.yaml edit
partitions 63 -> 2 protected + 61 collapsed (warnings 63 -> 4).
@mchmarny mchmarny force-pushed the feat/evidence-gate-report-tuning branch from b909a03 to fb91a2d Compare June 24, 2026 18:02

@njhensley njhensley left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Approve

Re-reviewed 1894bfcfb91a2d. Script-only delta, and every change is hardening with no regressions (bash -n clean):

  • log_sanitize now strips \r as well as \n — fully closes the ::warning:: injection nitpick.
  • strenv(CN) instead of env(CN) — fixes a scalar-typing edge case beyond the original ask (inline).
  • recipe_components() now fails open on resolver timeout/error (include + warn) instead of treating a hiccup as "references no changed component" — correct direction for a protection gate, and it neatly resolves the perf/robustness note from my last pass (inline).
  • log_sanitize applied consistently to overlay/name/parent/pointer in the remaining ::warning:: echoes.

Approving. The two non-blocking follow-ups from the prior review remain open and are fine to land separately: the ADR's "Classified verify failures" still sits under "(implemented)" while the verifier doesn't emit failureCause/pending yet, and the stale comment in .github/workflows/recipe-evidence.yaml:32-33. Neither blocks a warning-only gate. Nice iteration.

include=true
fi
else
# Resolver failed (aicr recipe timeout/error). Fail open: include the

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ This is the right call — failing open (include the recipe + emit a ::warning::) when aicr recipe times out or errors means a transient resolver hiccup can't silently hide real drift on a protected recipe. Fail-open is the correct direction for a protection gate. Verified the control flow: if rc=$(recipe_components …) captures the non-zero return cleanly under set -e, and an empty-but-successful resolve still falls through to "no intersection → not included".

# yq expression: a PR-controlled component name must be data, not code.
# strenv (not env) keeps it a string so a scalar-looking name (`true`,
# `null`, `123`) isn't YAML-typed and made to miss its registry entry.
b=$(CN="$cn" yq eval '.components[] | select(.name == strenv(CN))' "$reg_base" 2>/dev/null || true)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

strenv(CN) over env(CN) is a nice catch beyond the original note — keeps a scalar-looking component name (true, null, 123) typed as a string so it can't be YAML-coerced and miss its registry entry.

@mchmarny mchmarny enabled auto-merge (squash) June 24, 2026 18:09
@mchmarny mchmarny merged commit d25244b into main Jun 24, 2026
34 checks passed
@mchmarny mchmarny deleted the feat/evidence-gate-report-tuning branch June 24, 2026 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci area/docs size/L theme/ci-dx CI pipelines, developer experience, and build tooling theme/supply-chain SLSA, SBOM, Sigstore, and provenance verification

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evidence gate: curated 'protected recipes' set (implicit by committed pointer)

2 participants