feat(evidence-gate): protected/other partition + component-scoped cascade#1448
Conversation
Recipe evidence checkNo leaf overlays affected by this PR. This gate is warning-only and never blocks merge. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe recipe-evidence CI gate now scopes registry-change promotion to recipes whose resolved components intersect changed registry entries, separates affected recipes into protected and other affected groups, and renders the report with conditional protected output and a collapsed section for recipes without evidence. It also classifies Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/scripts/recipe-evidence-check.sh:
- Around line 266-279: Update the protected/others partitioning in
recipe-evidence-check.sh so deleted evidence pointers still count as protected:
when building the protected set in the affected-recipes loop, check whether
recipes/evidence/${slug}.yaml exists in either the BASE commit or the current
HEAD working tree, not just HEAD. Use the existing protected/others jq
accumulation logic and the slug loop, but adjust the file-presence test so
pointer deletions remain in protected and continue to be verified.
- Around line 397-443: The verifier handling in recipe-evidence-check.sh is
treating exit 1 as a generic verify error, but the contract says valid bundles
with recorded phase failures should be classified as Exit 1 and invalid bundles
as Exit 2. Update the JSON parsing in the verification block to read the `.exit`
field from the verifier output, then branch on that value in the `case`/status
mapping so Exit 1 produces the phase-failure informational path and Exit 2
remains the invalid-bundle path; keep the existing `signed_digest`, `pending`,
and `failureCause` handling in the same flow and reference the `verify_json`,
`verify_exit`, and `verify_cell` logic when making the change.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 1e168d10-38e8-4e57-9ae5-61ad26884e33
📒 Files selected for processing (2)
.github/scripts/recipe-evidence-check.shdocs/design/007-recipe-evidence.md
Coverage Report ✅
Coverage BadgeNo Go source files changed in this PR. |
njhensley
left a comment
There was a problem hiding this comment.
📋 Review — multi-persona, meta-confirmed
Method: four persona passes (shell-correctness, security, CI/DX, domain-correctness), each finding independently confirmed/refuted by a senior meta-reviewer against the resolved code. Tier legend: 🔴 Blocker · 🟠 Major · 🟡 Minor · 🔵 Nitpick.
Overall — no blockers; two Majors to resolve or defer in writing
Clean, well-commented tuning of a warning-only, never-blocking gate, and the noise-reduction direction is right. The privilege split (unprivileged pull_request build + workflow_run comment with no PR checkout) is implemented correctly — the most important control here, and it's sound. Nothing in this PR can break CI or merge a bad artifact.
Two Major items keep it from a clean approve (both inline):
- The component-scoped cascade walks a different inheritance graph than the one the digest is computed over → systematic false-negatives on registry-only edits to shared/base/wildcard components (the exact case the feature targets).
- The classified-cause/pending rendering reads verifier JSON fields that don't exist yet → graceful dead code that the ADR nonetheless marks "(implemented)".
🟡 Minor (not anchorable inline — file not in this diff)
Stale "Known limitations" comment in .github/workflows/recipe-evidence.yaml L32–35 still says discovery "walks spec.base ancestors but not spec.mixins" and "promote-all only matches registry.yaml or base.yaml" — both made false by this PR. The script header was updated; the duplicated workflow copy was not.
✅ Confirmed non-issues (examined and cleared)
recipe_components() has its own base-chain cycle guard (seen[]); removing the [[ ! -f "$pointer" ]] branch is safe (non-pointer recipes route to the others partition; a vanished pointer fails-closed to :x: verify error); mktemp files are cleaned on every continue; comm -12 | grep -q is correct under pipefail; the new protected=/other_affected= stdout keys break no consumer; the truncation row was correctly updated to 3 columns; the lower MAX_ROWS header comment was updated; trigger-model split is correct.
Summary
| Tier | Count |
|---|---|
| 🔴 Blocker | 0 |
| 🟠 Major | 2 |
| 🟡 Minor | 6 |
| 🔵 Nitpick | 2 |
Recommendation: not a blocker for a warning-only gate, but please either fix the cascade graph mismatch (it defeats the feature's core purpose for registry edits) or scope it down + document the gap, and reconcile the dead-code vs. "implemented" ADR before merge.
4f75ff9 to
1894bfc
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/scripts/recipe-evidence-check.sh:
- Around line 157-159: The failure handling in recipe_components currently
treats any aicr recipe timeout, resolver error, or jq parse failure as an empty
component list, which can cause Rule 5 to skip a recipe instead of
conservatively including it. Update recipe_components to fail open by surfacing
resolution/parsing failures in a way that Rule 5 can detect, and adjust the Rule
5 logic that consumes recipe_components so unresolved recipes are included
rather than silently skipped; use the existing recipe_components function and
the Rule 5 check in this script as the main touchpoints.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: aa5615c5-0742-42f0-9a1e-6463e0fd4692
📒 Files selected for processing (2)
.github/scripts/recipe-evidence-check.shdocs/design/007-recipe-evidence.md
1894bfc to
ff0b4b6
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/scripts/recipe-evidence-check.sh:
- Line 280: The component-resolution warning in the recipe evidence check is
writing the PR-controlled overlay value directly into a GitHub workflow command.
Update the warning path in recipe-evidence-check.sh to pass the overlay through
the existing sanitizer before emitting the ::warning:: message, using the same
helper used elsewhere for workflow-command output, so the warning text cannot be
interpreted as a command.
- Around line 121-122: The component lookup in the recipe evidence script uses
env(CN), which can coerce scalar-like names and miss matches; update the yq
selections in the lookup logic to use strenv(CN) instead. Apply this change
consistently in the component-name matching expressions for both the base and
head registry queries in the script.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 51eefa7a-0045-4e20-9c7f-a8aa7daf881e
📒 Files selected for processing (2)
.github/scripts/recipe-evidence-check.shdocs/design/007-recipe-evidence.md
njhensley
left a comment
There was a problem hiding this comment.
✅ Approve with notes
Re-reviewed the force-push (e3dad1f → 1894bfc) and verified each fix against the code. 8 of 10 findings fixed, both Majors addressed substantively — approving. The two Majors in particular:
- Cascade graph mismatch — fully fixed:
recipe_components()now resolves viaaicr recipe --format json(the same criteria-resolution engine the digest uses); jq path confirmed correct againstRecipeResult(pkg/recipe/metadata.go:415). The base/wildcard false-negative is closed. - Classified causes / pending — improved (now branches on the real
VerifyResult.Exitrather than OS-exit heuristics, no regression), but thepending/failureCausefields still aren't emitted by the verifier, so that rendering stays inert until the #1437 Go half. See inline.
Also confirmed fixed: cosmetic-edit banner guard, double-counted warnings (verify_ok), Markdown escaping (md_escape), yq env(CN), OTHER_MAX cap, log_sanitize on ::warning:: echoes, ADR "matches" wording. Bonus: the de-protection handling (BASE-pointer present, deleted at HEAD → :warning: evidence pointer removed) is a good catch beyond the review.
Two non-blocking items remain (fine as follow-ups)
- 🟠 ADR overstates classified causes as "(implemented)" while the verifier doesn't emit
failureCause/pendingyet — land the Go fields or soften the wording (inline). - 🟡 Stale comment in
.github/workflows/recipe-evidence.yaml:32-33still says discovery "walksspec.baseancestors but notspec.mixins" and "promote-all only matches registry.yaml" — both now false (resolution is viaaicr; registry is component-scoped). The script's own header note was correctly removed; this duplicated copy was missed.
None of these block a warning-only gate. Thanks for the thorough turnaround.
ff0b4b6 to
b909a03
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/scripts/recipe-evidence-check.sh:
- Around line 72-77: PR-controlled text is still reaching workflow commands
unsanitized in the annotation paths, specifically where `overlay` and `pointer`
are embedded in `::warning::` output. Update the warning/error construction in
`recipe-evidence-check.sh` so every PR-derived value goes through `log_sanitize`
before being echoed, and extend `log_sanitize` to collapse carriage returns as
well as newlines while neutralizing `::`. Use the existing `log_sanitize`,
`digest_err`, and `verify_err` handling as the pattern for the remaining
annotation messages.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 33a7a52e-a29a-43d2-87ea-9adfb5543951
📒 Files selected for processing (2)
.github/scripts/recipe-evidence-check.shdocs/design/007-recipe-evidence.md
…ade, classified causes Tune the warning-only recipe-evidence gate so broad-impact PRs stop rendering an alarming wall of "missing"/"drifted" rows. Protected vs. other affected (#1432): - A recipe is "protected" implicitly — iff it has a committed pointer in recipes/evidence/. The gate verifies the protected set (status table) and collapses affected-but-unevidenced recipes into a single <details> count instead of per-recipe "missing" warnings. No new files to maintain; the set grows as evidence is added. Component-scoped registry cascade (#1435): - A recipes/registry.yaml change no longer promotes every leaf. The gate diffs the registry at the component-entry level and marks a recipe affected only if its resolved component set (walked across base chain + spec.mixins) intersects the changed entries — so an aws-efa-only edit flags only recipes that reference aws-efa. base.yaml stays broad. The recomputed digest is still ground truth (match => not drift). Classified verify failures (#1437, script half): - Parse the structured failureCause {class, httpStatus, hint} and pending flag from `aicr evidence verify --format json` and render an actionable cause in the verify column (e.g. "invalid — registry-forbidden (HTTP 403): make the fork's aicr-evidence package public"); echo the full error into the job log. Docs: ADR-007 gains a "Gate report structure" subsection describing the implicit-protected definition, component-scoped cascade, and classified causes. Smoke-tested locally: aws-efa-only edit flags 23 EKS recipes (1 protected + 22 collapsed) and leaves GKE/OCI recipes untouched; base.yaml edit partitions 63 -> 2 protected + 61 collapsed (warnings 63 -> 4).
b909a03 to
fb91a2d
Compare
njhensley
left a comment
There was a problem hiding this comment.
✅ Approve
Re-reviewed 1894bfc → fb91a2d. Script-only delta, and every change is hardening with no regressions (bash -n clean):
log_sanitizenow strips\ras well as\n— fully closes the::warning::injection nitpick.strenv(CN)instead ofenv(CN)— fixes a scalar-typing edge case beyond the original ask (inline).recipe_components()now fails open on resolver timeout/error (include + warn) instead of treating a hiccup as "references no changed component" — correct direction for a protection gate, and it neatly resolves the perf/robustness note from my last pass (inline).log_sanitizeapplied consistently tooverlay/name/parent/pointerin the remaining::warning::echoes.
Approving. The two non-blocking follow-ups from the prior review remain open and are fine to land separately: the ADR's "Classified verify failures" still sits under "(implemented)" while the verifier doesn't emit failureCause/pending yet, and the stale comment in .github/workflows/recipe-evidence.yaml:32-33. Neither blocks a warning-only gate. Nice iteration.
| include=true | ||
| fi | ||
| else | ||
| # Resolver failed (aicr recipe timeout/error). Fail open: include the |
There was a problem hiding this comment.
✅ This is the right call — failing open (include the recipe + emit a ::warning::) when aicr recipe times out or errors means a transient resolver hiccup can't silently hide real drift on a protected recipe. Fail-open is the correct direction for a protection gate. Verified the control flow: if rc=$(recipe_components …) captures the non-zero return cleanly under set -e, and an empty-but-successful resolve still falls through to "no intersection → not included".
| # yq expression: a PR-controlled component name must be data, not code. | ||
| # strenv (not env) keeps it a string so a scalar-looking name (`true`, | ||
| # `null`, `123`) isn't YAML-typed and made to miss its registry entry. | ||
| b=$(CN="$cn" yq eval '.components[] | select(.name == strenv(CN))' "$reg_base" 2>/dev/null || true) |
There was a problem hiding this comment.
✅ strenv(CN) over env(CN) is a nice catch beyond the original note — keeps a scalar-looking component name (true, null, 123) typed as a string so it can't be YAML-coerced and miss its registry entry.
Summary
Tunes the warning-only recipe-evidence gate (
.github/scripts/recipe-evidence-check.sh) so broad-impact PRs stop rendering an alarming wall of "missing"/"drifted" rows. Three changes: a protected-vs-other partition, a component-scoped registry cascade, and classified verify-failure causes in the comment.Motivation / Context
Part of the evidence-gate pilot epic. PR #1418 (an
aws-efa-only registry bump) promoted ~all leaf recipes to "affected" and #1427 surfaced an unhelpful bare "invalid". This is the gate-report half of that tuning; it consumes thefailureCauseJSON contract added in #1445.Fixes: #1432, #1435, #1437
Related: #1431
Type of Change
Component(s) Affected
docs/,examples/).github/scripts/recipe-evidence-check.sh) + ADR-007Implementation Notes
recipes/evidence/. The gate verifies the protected set (status table); affected-but-unevidenced recipes collapse into a single<details>count — no longer per-recipe "missing" warnings. No new files to maintain.registry.yamlchanges are diffed at the component-entry level; a recipe is affected only if its resolved component set (walked across base chain +spec.mixins) intersects the changed entries.base.yamlstays broad. The recomputed digest remains ground truth (match ⇒ not drift).failureCause {class, httpStatus, hint}+pendingfromaicr evidence verify --format json; render an actionable cause in the verify column and echo the full error to the job log.Testing
Local smoke tests (built
aicr, temp commits,recipe-evidence-check.sh):aws-efa-only registry editgb200-eks-ubuntu-training) + 22 EKS collapsed; GKEh100-gke-cos-trainingnot affectedbase.yamledit (broad)bash -n+shellcheck -S warningclean. (No Go changed; the consumedfailureCause/pendingJSON shipped in #1445.)Risk Assessment
Rollout notes: No config or workflow changes; the workflow invokes the same script with the same env contract.
Checklist
shellcheck,bash -n)git commit -S)