Skip to content

refactor(uat): stable digest-addressed OCI ref for evidence push#1479

Merged
mchmarny merged 4 commits into
NVIDIA:mainfrom
njhensley:refactor/uat-oci-address
Jun 26, 2026
Merged

refactor(uat): stable digest-addressed OCI ref for evidence push#1479
mchmarny merged 4 commits into
NVIDIA:mainfrom
njhensley:refactor/uat-oci-address

Conversation

@njhensley

Copy link
Copy Markdown
Member

Summary

Switch the UAT evidence push target to a stable, digest-addressed repo-per-recipe layout: the repo path is the recipe identity and the run id moves to a :run-<id> tag.

Motivation / Context

The UAT pipeline baked the CI run_id into the OCI repository name and duplicated recipe-coordinate dims across both the repo and the derived tag, fragmenting evidence across a new GHCR package per run and making it hard to discover, list, or consume.

before: ghcr.io/nvidia/aicr-uat-eks-h100-training-<run_id>:h100-eks-ubuntu-training-kubeflow-<fp12>
after:  ghcr.io/nvidia/aicr-evidence/h100-eks-ubuntu-training-kubeflow:run-<run_id>

This adopts the issue's preferred repo-per-recipe layout, eliminating all four redundancies:

  1. run_id no longer in the repo name → one stable package per recipe; oras repo tags lists run history; visibility set once.
  2. Coordinate dims no longer duplicated across repo and tag.
  3. Non-verifiable 12-hex fingerprint prefix dropped — an explicit tag suppresses aicr's <recipe-slug>-<fingerprint> derivation; consumers pin immutably via @sha256:….
  4. The full recipe slug is the single canonical coordinate (the repo path).

The run id is not lost: the bundle predicate carries rekorLogIndex + attestedAt, and the tag preserves CI provenance.

Fixes: #1478
Related: N/A

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • Build/CI/tooling

Component(s) Affected

  • Other: UAT pipeline (tests/uat/{aws,gcp}, .github/workflows/uat-{aws,gcp}.yaml)

Implementation Notes

  • tests/uat/{aws,gcp}/tests/h100-training-config.yamlpush: is now the bare per-recipe repo (matching the canonical overlay names h100-eks-ubuntu-training-kubeflow / h100-gke-cos-training-kubeflow).
  • tests/uat/{aws,gcp}/runinject_push_target applies a :run-${RUN_ID} tag idempotently, stripping any prior :run-… tag first so a re-run (or a changed RUN_ID) in the same workspace does not double-append.
  • .github/workflows/uat-{aws,gcp}.yaml — the **Evidence (OCI):** summary echo and surrounding comments reference the new ref.
  • An explicit tag is required to suppress aicr's <recipe-slug>-<fingerprint> tag derivation (which only runs when the ref omits a tag); the nested repo path + tag parses cleanly through oci.ParseOutputTarget (docker reference.ParseNormalizedNamed).

Testing

yamllint tests/uat/{aws,gcp}/tests/h100-training-config.yaml .github/workflows/uat-{aws,gcp}.yaml  # clean
bash -n tests/uat/{aws,gcp}/run                                                                    # syntax OK

YAML/shell-only change (no Go sources), so the Go test/lint/coverage gates do not apply. inject_push_target idempotency verified across fresh, repeat, changed-RUN_ID, and local-<ts> forms.

Risk Assessment

  • Low — Isolated change to the UAT pipeline, easy to revert.

Rollout notes: First run on the new layout creates the ghcr.io/nvidia/aicr-evidence/<recipe> package; set its visibility once. No consumer pins the old aicr-uat-* packages.

Checklist

  • Tests pass locally (make test with -race)
  • Linter passes (make lint)
  • I did not skip/disable tests to make CI green
  • I added/updated tests for new functionality
  • I updated docs if user-facing behavior changed
  • Changes follow existing patterns in the codebase
  • Commits are cryptographically signed (git commit -S)

Switch the UAT evidence push target to the repo-per-recipe layout: the
repo path is the stable recipe identity and the run id moves to a
`:run-<id>` tag, so consumers pin immutably by digest.

  before: ghcr.io/nvidia/aicr-uat-eks-h100-training-<run_id>
          :h100-eks-ubuntu-training-kubeflow-<fp12>
  after:  ghcr.io/nvidia/aicr-evidence/h100-eks-ubuntu-training-kubeflow
          :run-<run_id>

Eliminates the four redundancies in NVIDIA#1478: run_id no longer fragments
GHCR into a package per run (`oras repo tags` now lists run history,
visibility set once); coordinate dims are no longer duplicated across
repo and tag; the non-verifiable 12-hex fingerprint prefix is dropped
(an explicit tag suppresses the aicr <slug>-<fingerprint> derivation);
and the full recipe slug is the single canonical coordinate.

- tests/uat/{aws,gcp}/tests/h100-training-config.yaml: bare per-recipe
repo
- tests/uat/{aws,gcp}/run: inject_push_target applies :run-${RUN_ID}
  idempotently (strips any prior :run-… tag)
- .github/workflows/uat-{aws,gcp}.yaml: summary echo + comments

Closes NVIDIA#1478
@njhensley njhensley requested review from a team as code owners June 25, 2026 23:18
@njhensley njhensley added area/ci theme/supply-chain SLSA, SBOM, Sigstore, and provenance verification labels Jun 25, 2026
@coderabbitai

coderabbitai Bot commented Jun 25, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: b80c1bab-852a-40a5-a7c2-2adbe7e9f9e0

📥 Commits

Reviewing files that changed from the base of the PR and between 6fc3444 and 56b6439.

📒 Files selected for processing (6)
  • .github/workflows/uat-aws.yaml
  • .github/workflows/uat-gcp.yaml
  • tests/uat/aws/run
  • tests/uat/aws/tests/h100-training-config.yaml
  • tests/uat/gcp/run
  • tests/uat/gcp/tests/h100-training-config.yaml

📝 Walkthrough

Walkthrough

UAT AWS and GCP evidence references were updated to use stable ghcr.io/nvidia/aicr-evidence/... repositories with :run-${RUN_ID} tags. The H100 training configs change the attestation push repositories and comments. The AWS and GCP run scripts now normalize any existing :run-* suffix before setting the push target. The AWS and GCP workflow comments and Test Summary outputs now print the new evidence OCI references.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested labels

theme/ci-dx

Suggested reviewers

  • mchmarny
🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise, specific, and matches the main UAT evidence OCI ref refactor.
Description check ✅ Passed The description is clearly related to the UAT evidence push target changes and their motivation.
Linked Issues check ✅ Passed The changes align with #1478 by moving run identity to a tag and using stable per-recipe repositories in UAT.
Out of Scope Changes check ✅ Passed All modified files are scoped to the UAT evidence ref, run scripts, and workflow text updates.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@mchmarny mchmarny left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean refactor — moves the CI run_id out of the OCI repo name into a :run-<id> tag on a stable per-recipe repo, killing the per-run package sprawl and the duplicated coordinate dims.

Verified the load-bearing piece: an explicit tag suppresses aicr's <slug>-<fingerprint> derivation (effectiveEvidenceRef, pkg/evidence/attestation/oci.go:159), and the inject_push_target strip-and-reapply (%%:run-*) is idempotent across re-runs and changed RUN_ID (the :run- glob won't clobber a registry port or other tags). No other references to the old aicr-uat-* packages remain. All CI green. LGTM.

One operational reminder (already in your rollout notes): the first run creates a fresh ghcr.io/nvidia/aicr-evidence/<recipe> package — its visibility needs setting once for downstream pulls.

@mchmarny mchmarny enabled auto-merge (squash) June 26, 2026 12:57
@mchmarny mchmarny merged commit e19cb68 into NVIDIA:main Jun 26, 2026
30 checks passed
atif1996 added a commit that referenced this pull request Jun 29, 2026
- Restore `case "oci": return "oke"` in parseProvider alongside the new
  ocid1. prefix check — both formats must map to oke
- Fix inject_push_target in aws/run, gcp/run, oci/run to use the
  :run-${RUN_ID} tag scheme (#1478/#1479), not the repo-suffix approach
- Update apiVersion aicr.nvidia.com/v1alpha1 → aicr.run/v1alpha2 in
  l40s-training-config.yaml and assert-recipe.yaml
- Add `ol` to OSDimension comment in pkg/fingerprint/types.go
- Add `ol` and `talos` to Supported values comment in pkg/client/v1/types.go

Signed-off-by: Atif Mahmood <[email protected]>
Signed-off-by: Atif Mahmood <[email protected]>
atif1996 added a commit that referenced this pull request Jun 29, 2026
- Restore `case "oci": return "oke"` in parseProvider alongside the new
  ocid1. prefix check — both formats must map to oke
- Fix inject_push_target in aws/run, gcp/run, oci/run to use the
  :run-${RUN_ID} tag scheme (#1478/#1479), not the repo-suffix approach
- Update apiVersion aicr.nvidia.com/v1alpha1 → aicr.run/v1alpha2 in
  l40s-training-config.yaml and assert-recipe.yaml
- Add `ol` to OSDimension comment in pkg/fingerprint/types.go
- Add `ol` and `talos` to Supported values comment in pkg/client/v1/types.go

Signed-off-by: Atif Mahmood <[email protected]>
Signed-off-by: Atif Mahmood <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci area/tests size/M theme/supply-chain SLSA, SBOM, Sigstore, and provenance verification

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Switch UAT evidence OCI ref to stable digest-addressed format

2 participants