Summary

aicr validate fails with ErrImagePull / ImagePullBackOff when run from any tagged-release binary because the validator catalog rewrites validator image tags to :sha-<commit> of the binary's own build, but the release workflow publishes those validator images tagged with the git tag (e.g. :v0.13.0-rc3), not :sha-<commit>. The two naming schemes never meet, so every release install of aicr hits a 404 on the first validate against a cluster.

This is structurally the same gap that AICR_VALIDATOR_IMAGE_TAG was added for (per pkg/validator/catalog/catalog.go:197-202, intended for feature-branch dev builds where no :sha-<commit> image exists). It turns out tagged-release builds are in the same bucket — but those are the binaries end users actually install.

Repro

1. Install a released aicr binary (e.g. 0.13.0-rc3; homebrew tap, gh release download, or build from the v0.13.0-rc3 git tag).

2. Run any aicr validate that schedules a job in-cluster:

   aicr validate --config aicr-config.yaml --phase conformance --output report.json

3. The job's pod stalls in ImagePullBackOff:

   Failed to pull image \"ghcr.io/nvidia/aicr-validators/conformance:sha-481507af51a9cc4a9b9501f35d680734bfb5a925\":
     ghcr.io/nvidia/aicr-validators/conformance:sha-481507af51a9cc4a9b9501f35d680734bfb5a925: not found
   

4. GHCR confirms the per-SHA image was never pushed for that commit, but the release workflow did push v0.13.0-rc3:

   gh api 'orgs/NVIDIA/packages/container/aicr-validators%2Fconformance/versions?per_page=100' \
     --jq '[.[] | .metadata.container.tags[]] | unique | map(select(test(\"^v[0-9]\")))'
   # → [\"v0.13.0-rc1\",\"v0.13.0-rc2\",\"v0.13.0-rc3\", ...]

Root cause

The validator catalog (pkg/validator/catalog/catalog.go) has one tag-rewrite path: when the binary's version string carries a non-empty commit, rewrite :latest to :sha-<commit> (see replaceLatestWithSHA, comment at catalog.go:52 and catalog.go:85-88). That rule was written against the on-push.yaml publishing scheme, which tags by commit SHA on every main push. The release workflow (goreleaser / release-tag CI) doesn't push sha-<commit> aliases — it tags validators with the same v<version> string goreleaser stamps into the binary. So:

AICR_VALIDATOR_IMAGE_TAG (per catalog.go:197-209 / replaceTag) is the existing escape hatch for the last two rows. The second row — every shipping release install — should not need an escape hatch.

Severity

User-visible: validate is unusable from the install path most users follow (tagged release). The error surfaces as a 5-minute ImagePullBackOff loop with no actionable hint in aicr output, so time-to-debug is high. No data loss path.

Proposed fix (binary side, preferred)

Teach the catalog to detect a release-style version string and prefer :<version> over :sha-<commit>. Sketch in pkg/validator/catalog/catalog.go:

// resolveDefaultTag picks the right tag for the bundled-in default validator
// images. Precedence (highest to lowest):
//   1. AICR_VALIDATOR_IMAGE_TAG env var (explicit override; existing).
//   2. Release-style version (matches /^v\d+\.\d+\.\d+([.-].+)?$/) — use :<version>.
//      This is what goreleaser stamps into the binary AND what the release
//      workflow publishes the validator image as. The two move in lockstep.
//   3. Non-empty hex commit SHA — use :sha-<commit> (matches on-push.yaml).
//   4. Fall back to :latest (dev builds; user can override with the env var).

The detection only needs version.Version() (already plumbed through to the catalog) plus a regexp; goreleaser emits exactly the same string into the binary version metadata and into the image tag, so the alignment is structural.

Optional follow-on: when the resolved tag is unreachable (image pull would 404), fall back to :edge and log a single slog.Warn — but this needs a registry probe which is a separable concern and arguably out of scope.

Test plan

• Unit: table-driven tests in pkg/validator/catalog/catalog_test.go exercising (a) v0.13.0-rc3 → :v0.13.0-rc3, (b) ca62d0c4e97f... (40-char hex) → :sha-ca62d0c4..., (c) (devel) → :latest, (d) env-var override beats all of the above.
• Integration: run the existing aicr validate --no-cluster chainsaw tests under a version.Version() = v0.13.0-rc3 fixture and assert the rendered job manifests reference :v0.13.0-rc3.
• Manual: install v0.13.0-rc3 binary on a fresh cluster, run aicr validate --config aicr-config.yaml --phase conformance — should pull ghcr.io/nvidia/aicr-validators/conformance:v0.13.0-rc3 and succeed without AICR_VALIDATOR_IMAGE_TAG.

Alternatives considered

• CI side: release workflow also pushes sha-<commit> aliases. Cheap, doesn't require a binary change, but duplicates tags on every release and doesn't help local make build or feature-branch builds (those rows still need the env var). Doesn't fix the broken default for the install path most users take.
• Document AICR_VALIDATOR_IMAGE_TAG=v<version> in the install path. Pushes Kubernetes-image-tag trivia onto users for a problem the binary already knows the answer to (its own version is right there).

Affected paths

• pkg/validator/catalog/catalog.go — tag resolution (replaceLatestWithSHA, replaceTag, surrounding wiring).
• pkg/validator/job/deployer.go:248 — comment already nods at this exact failure mode ("override or it will resolve to the same unpublished :sha-"); the fix removes the need for the override on releases.
• internal/version/ (or wherever version.Version() is sourced) — no change, but its current shape determines the regexp.

Workaround until fixed

AICR_VALIDATOR_IMAGE_TAG=v0.13.0-rc3 aicr validate --config aicr-config.yaml \
    --phase conformance \
    --output report.json

Or use the floating edge tag (current main HEAD) if the user is comfortable running validator images ahead of the binary's release version:

AICR_VALIDATOR_IMAGE_TAG=edge aicr validate ...

Related

• [Bug]: helmfile bundle fails on fresh cluster — CRD/CR render order between nodewright-operator and nodewright-customizations #914 (helmfile fresh-cluster CRD/CR ordering) and [Bug]: helmfile/helm bundles fail on GKE — system-*-critical priorityClass pods rejected by ResourceQuota admission #915 (GKE ResourceQuota / priorityClass) — separate problems, same demo path. All three should be resolved before demos/cuj1-gke-config.md runs cleanly end-to-end with no manual steps on a tagged-release binary.

Are you willing to contribute?

Maybe, with guidance — happy to PR the binary-side fix once a maintainer signs off on the version-string regexp shape (and whether to deal with the + build-metadata suffix goreleaser sometimes appends).