Skip to content

Evidence gate: surface underlying verify failure cause (e.g. registry 403) #1437

Description

@njhensley

Part of #1431.

Summary

When aicr evidence verify fails inside the gate, the PR comment table and the job logs only convey a coarse verdict (effectively "invalid") with no actionable cause. In #1427 the contributor's fork GHCR package was private, so the gate's verify pull got an HTTP 403 (registry not accessible) — but that root cause was not discoverable from either the table output or the workflow logs. Surface the underlying error.

Motivation / Context

  • aicr evidence verify distinguishes "bundle invalid" (exit 2) from "recorded validator failures" (exit 1) from "valid" (exit 0), but the gate collapses the failure into a single column value without the reason (registry auth/403, not-found/404, signature mismatch, schema/integrity error).
  • A contributor can't self-serve the fix (make the package public) if the gate never tells them the pull was a 403.

Proposed approach

  • Ensure aicr evidence verify --format json emits a machine-readable error/cause field for the failure path (registry status code, error code, message); add it if missing.
  • In .github/scripts/recipe-evidence-check.sh, capture stderr + the JSON cause from the verify call and:
    • render a short classified cause in the table's verify column or as a footnote (e.g. "invalid — registry 403 (package not public?)"), and
    • echo the full error into the job log (currently swallowed).
  • Classify the common cases so the message is actionable: 403/401 → "registry not accessible (make the fork's aicr-evidence package public)"; 404 → "bundle not found at ref"; signature/schema → as reported.

Acceptance criteria

  • A 403 on the evidence pull shows a clear, actionable cause in the PR comment and in the job log — not just "invalid."
  • evidence verify --format json carries a structured error/cause for non-zero exits.
  • Other failure classes (404, signature mismatch, schema/integrity) render distinct, readable causes.

References

  • .github/scripts/recipe-evidence-check.sh (verify invocation + report rendering)
  • pkg/cli/evidence_verify.go (exit codes, --format json); pkg/evidence/verifier/
  • Related: the fork-based signing workflow / two-phase publish issues (fork GHCR visibility is the fix the message should point at).

Metadata

Metadata

Assignees

Labels

External DSXarea/citheme/ci-dxCI pipelines, developer experience, and build toolingtheme/supply-chainSLSA, SBOM, Sigstore, and provenance verification

Fields

No fields configured for Enhancement.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions