Skip to content

Digest-pin explicit image references in recipes #749

Description

@mchmarny

Parent: #739
Builds on the chart-version pinning issue (filed alongside)

Summary

After chart versions are pinned, take a one-time pass to digest-pin every container image referenced explicitly in this repo. Sustain it via Renovate's digest auto-bump and a CI gate.

Scope: explicit refs only

This issue covers ~25 image references that AICR controls directly:

  • recipes/components/*/values.yaml — overrides where we pin a specific image (e.g., aws-ebs-csi-driver, aws-efa)
  • recipes/components/*/manifests/*.yaml — embedded K8s manifests (e.g., gke-nccl-tcpxo, nodewright-customizations, kubeflow-trainer's pytorch image, network-operator's busybox)
  • AICR's own runtime images and validator images (already digest-pinned by goreleaser/Sigstore — verify, don't change)

The current BOM lists every one of these — see dist/bom/bom.md after make bom.

Out of scope

Digest-pinning images deployed by upstream Helm charts (gpu-operator's ~14 sub-images, kube-prometheus-stack's ~8, cert-manager's ~4, etc.) is not part of this issue. Most charts don't expose a digest field for sub-images, and maintaining a per-sub-image override matrix per chart is intractable. The right answer there is admission-time digest verification (Kyverno/Cosign), tracked under #744.

Why digests

A :tag reference is mutable. Upstream can republish :v1.2.3 with new bytes and your "pinned" recipe quietly deploys something different. Digest pinning gives byte-for-byte reproducibility for the surface we control, and produces a CycloneDX BOM that's signature-attestable end-to-end.

Proposed approach

  1. Resolve digests — for each explicit ref, crane digest <image>:<tag> (or oras manifest fetch). Update the file with the canonical <image>:<tag>@sha256:<digest> form so both the human-readable tag and the immutable digest are present.
  2. Renovate config — enable pinDigests: true for OCI image references so new builds under the same tag auto-open digest-bump PRs.
  3. CI enforcement — extend the BOM tool with a -require-digests flag that fails if any explicit image ref in recipes/ lacks @sha256:. Wire into make qualify.
  4. Dev ergonomics — add a make pin-digests helper that resolves all current tags to digests in one pass, for use during chart-version bumps.

Success criteria

  • Every image: reference under recipes/components/*/values.yaml and recipes/components/*/manifests/*.yaml includes an @sha256: digest.
  • make bom BOM_REQUIRE_DIGESTS=1 (or equivalent) is part of make qualify.
  • Renovate auto-opens digest-bump PRs and they merge cleanly.
  • The CycloneDX BOM populates the digest qualifier and the @version slot becomes spec-compliant for these refs (related to PR feat(bom): add CycloneDX 1.6 image BOM generator #747 review thread on purl spec compliance).

Sustainability notes

  • Renovate's pinDigests: true is the durable mechanism — without it, digests fall behind upstream patches.
  • Bumping a chart version in Tier 1 will invalidate digests of any sub-images we explicitly override; the BOM CI gate catches that.
  • Expect 1–2 digest-bump PRs per week per component once Renovate is active. Bundling via Renovate groupName keeps the noise manageable.

Effort estimate

  • Resolving digests + updating files: ~1 day for the ~25 explicit refs.
  • BOM -require-digests flag + qualify wiring: ~half a day.
  • Renovate config + verification: ~half a day.
  • E2E validation across the changed components: parallelizable, ~1 day wall-clock.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions