You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Parent: #739
Builds on the chart-version pinning issue (filed alongside)
Summary
After chart versions are pinned, take a one-time pass to digest-pin every container image referenced explicitly in this repo. Sustain it via Renovate's digest auto-bump and a CI gate.
Scope: explicit refs only
This issue covers ~25 image references that AICR controls directly:
recipes/components/*/values.yaml — overrides where we pin a specific image (e.g., aws-ebs-csi-driver, aws-efa)
AICR's own runtime images and validator images (already digest-pinned by goreleaser/Sigstore — verify, don't change)
The current BOM lists every one of these — see dist/bom/bom.md after make bom.
Out of scope
Digest-pinning images deployed by upstream Helm charts (gpu-operator's ~14 sub-images, kube-prometheus-stack's ~8, cert-manager's ~4, etc.) is not part of this issue. Most charts don't expose a digest field for sub-images, and maintaining a per-sub-image override matrix per chart is intractable. The right answer there is admission-time digest verification (Kyverno/Cosign), tracked under #744.
Why digests
A :tag reference is mutable. Upstream can republish :v1.2.3 with new bytes and your "pinned" recipe quietly deploys something different. Digest pinning gives byte-for-byte reproducibility for the surface we control, and produces a CycloneDX BOM that's signature-attestable end-to-end.
Proposed approach
Resolve digests — for each explicit ref, crane digest <image>:<tag> (or oras manifest fetch). Update the file with the canonical <image>:<tag>@sha256:<digest> form so both the human-readable tag and the immutable digest are present.
Renovate config — enable pinDigests: true for OCI image references so new builds under the same tag auto-open digest-bump PRs.
CI enforcement — extend the BOM tool with a -require-digests flag that fails if any explicit image ref in recipes/ lacks @sha256:. Wire into make qualify.
Dev ergonomics — add a make pin-digests helper that resolves all current tags to digests in one pass, for use during chart-version bumps.
Success criteria
Every image: reference under recipes/components/*/values.yaml and recipes/components/*/manifests/*.yaml includes an @sha256: digest.
make bom BOM_REQUIRE_DIGESTS=1 (or equivalent) is part of make qualify.
Renovate auto-opens digest-bump PRs and they merge cleanly.
The CycloneDX BOM populates the digest qualifier and the @version slot becomes spec-compliant for these refs (related to PR feat(bom): add CycloneDX 1.6 image BOM generator #747 review thread on purl spec compliance).
Sustainability notes
Renovate's pinDigests: true is the durable mechanism — without it, digests fall behind upstream patches.
Bumping a chart version in Tier 1 will invalidate digests of any sub-images we explicitly override; the BOM CI gate catches that.
Expect 1–2 digest-bump PRs per week per component once Renovate is active. Bundling via Renovate groupName keeps the noise manageable.
Effort estimate
Resolving digests + updating files: ~1 day for the ~25 explicit refs.
BOM -require-digests flag + qualify wiring: ~half a day.
Renovate config + verification: ~half a day.
E2E validation across the changed components: parallelizable, ~1 day wall-clock.
Parent: #739
Builds on the chart-version pinning issue (filed alongside)
Summary
After chart versions are pinned, take a one-time pass to digest-pin every container image referenced explicitly in this repo. Sustain it via Renovate's digest auto-bump and a CI gate.
Scope: explicit refs only
This issue covers ~25 image references that AICR controls directly:
recipes/components/*/values.yaml— overrides where we pin a specific image (e.g.,aws-ebs-csi-driver,aws-efa)recipes/components/*/manifests/*.yaml— embedded K8s manifests (e.g.,gke-nccl-tcpxo,nodewright-customizations,kubeflow-trainer's pytorch image, network-operator's busybox)The current BOM lists every one of these — see
dist/bom/bom.mdaftermake bom.Out of scope
Digest-pinning images deployed by upstream Helm charts (gpu-operator's ~14 sub-images, kube-prometheus-stack's ~8, cert-manager's ~4, etc.) is not part of this issue. Most charts don't expose a digest field for sub-images, and maintaining a per-sub-image override matrix per chart is intractable. The right answer there is admission-time digest verification (Kyverno/Cosign), tracked under #744.
Why digests
A
:tagreference is mutable. Upstream can republish:v1.2.3with new bytes and your "pinned" recipe quietly deploys something different. Digest pinning gives byte-for-byte reproducibility for the surface we control, and produces a CycloneDX BOM that's signature-attestable end-to-end.Proposed approach
crane digest <image>:<tag>(ororas manifest fetch). Update the file with the canonical<image>:<tag>@sha256:<digest>form so both the human-readable tag and the immutable digest are present.pinDigests: truefor OCI image references so new builds under the same tag auto-open digest-bump PRs.-require-digestsflag that fails if any explicit image ref inrecipes/lacks@sha256:. Wire intomake qualify.make pin-digestshelper that resolves all current tags to digests in one pass, for use during chart-version bumps.Success criteria
image:reference underrecipes/components/*/values.yamlandrecipes/components/*/manifests/*.yamlincludes an@sha256:digest.make bom BOM_REQUIRE_DIGESTS=1(or equivalent) is part ofmake qualify.digestqualifier and the@versionslot becomes spec-compliant for these refs (related to PR feat(bom): add CycloneDX 1.6 image BOM generator #747 review thread on purl spec compliance).Sustainability notes
pinDigests: trueis the durable mechanism — without it, digests fall behind upstream patches.groupNamekeeps the noise manageable.Effort estimate
-require-digestsflag + qualify wiring: ~half a day.