Skip to content

fix(recipes): fully-qualify image refs in component manifests#761

Merged
mchmarny merged 3 commits into
mainfrom
feat/pin-manifest-image-refs
May 5, 2026
Merged

fix(recipes): fully-qualify image refs in component manifests#761
mchmarny merged 3 commits into
mainfrom
feat/pin-manifest-image-refs

Conversation

@mchmarny

@mchmarny mchmarny commented May 5, 2026

Copy link
Copy Markdown
Member

Summary

Closes #743. Two real fixes plus a regression test that prevents the class of bug going forward.

Motivation / Context

Inspection of recipes/components/*/manifests/*.yaml (driven by the BOM tool from #747) flagged a handful of image references as "untagged" — risk of silent :latest drift in production. Investigation showed only one was a real bug; the rest were a BOM-tool false positive against a CRD pattern.

Fixes: #743
Refs: #739

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality) — BOM tool extension, regression test

Component(s) Affected

  • Recipe data (recipes/components/gke-nccl-tcpxo/manifests/)
  • Core libraries (pkg/bom)
  • Test (recipes/manifest_images_test.go)

Implementation Notes

1. Real fix: gke-nccl-tcpxo init container. Was image: \"ubuntu\" with no tag — resolved to :latest from Docker Hub. Now pinned to ubuntu:24.04@sha256:c4a8... for byte-level reproducibility.

2. BOM tool: handle CRD-style triplets. NicClusterPolicy (network-operator) and Skyhook Package (nodewright-customizations) declare the deployed image as three sibling fields:

ofedDriver:
  repository: nvcr.io/nvidia/mellanox
  image: doca-driver
  version: doca3.2.0-25.10-1.2.8.0-2

The operator combines them at reconcile time. The previous BOM extractor only saw image: doca-driver and flagged it as untagged. walkForImages now collects sibling scalars up front and emits a fully-qualified ref (nvcr.io/nvidia/mellanox/doca-driver:doca3.2.0-25.10-1.2.8.0-2). The new logic:

  • If image is already fully qualified (contains / or has its own tag), it's used as-is.
  • Otherwise repository is prepended when present, and version is appended as a tag when image doesn't already carry one.

3. Regression test. TestComponentManifestImagesAreFullyQualified walks every components/*/manifests/*.yaml via the embedded recipes.FS and asserts every extracted image has a :tag or @digest. Currently verifies 13 image refs across the registry. This is the lint rule the issue asked for, but in test form so it runs as part of make qualify.

Image set unchanged in production. The versions for the previously "untagged" refs were already pinned in adjacent version: fields; this PR just makes the BOM reflect them correctly. The only deployed-byte change is the ubuntu init container, which now uses 24.04 LTS at a specific digest.

Testing

unset GITLAB_TOKEN && make qualify
# Codebase qualification completed

make bom after the change shows fully-qualified refs everywhere:

### gke-nccl-tcpxo
- ubuntu:24.04@sha256:c4a8d5503dfb...

### network-operator
- nvcr.io/nvidia/mellanox/doca-driver:doca3.2.0-25.10-1.2.8.0-2
- nvcr.io/nvidia/mellanox/k8s-rdma-shared-dev-plugin:network-operator-v26.1.0
- nvcr.io/nvidia/doca/doca_telemetry:1.22.5-doca3.1.0-host

### nodewright-customizations
- ghcr.io/nvidia/nodewright-packages/nvidia-setup:0.2.2
- ghcr.io/nvidia/nodewright-packages/nvidia-tuned:0.3.0
- ghcr.io/nvidia/skyhook-packages/nvidia-tuning-gke:0.1.1
- ghcr.io/nvidia/skyhook-packages/shellscript:1.1.1

pkg/bom coverage held at 87.7%.

Risk Assessment

  • Low — One real YAML pin (init container, doesn't affect main workload images), one BOM-tool extractor improvement that never lowers the set of detected images, one new test. Easy to revert if a regression surfaces.

Rollout notes: the new TestComponentManifestImagesAreFullyQualified test will gate future PRs that introduce bare image: foo references in component manifests.

Checklist

  • Tests pass locally (make test with -race)
  • Linter passes (make lint)
  • I did not skip/disable tests to make CI green
  • I added/updated tests for new functionality
  • I updated docs if user-facing behavior changed (n/a — internal extractor behavior)
  • Changes follow existing patterns in the codebase
  • Commits are cryptographically signed (git commit -S)

Closes #743. Two real fixes plus a regression test that prevents the
class of bug going forward:

1. gke-nccl-tcpxo init container pinned: was \`image: \"ubuntu\"\` (no
   tag, resolved to :latest from Docker Hub), now
   \`ubuntu:24.04@sha256:c4a8...\` so the deployed bytes are reproducible.

2. BOM tool now combines CRD-style triplets. The \`image:\`/\`repository:\`
   /\`version:\` siblings used by NicClusterPolicy (network-operator) and
   Skyhook Package (nodewright-customizations) are not concatenated in
   YAML — the operator combines them at reconcile time. The previous
   extractor saw only \`image: doca-driver\` and reported it as untagged.
   walkForImages now collects the three siblings up front and emits a
   fully-qualified ref, e.g.
   \`nvcr.io/nvidia/mellanox/doca-driver:doca3.2.0-25.10-1.2.8.0-2\`.

3. New TestComponentManifestImagesAreFullyQualified walks every
   components/*/manifests/*.yaml via the embedded recipes.FS and asserts
   every extracted image has a :tag or @digest. 13 image refs verified.
   This catches future bare \`image: foo\` regressions in CI.

The deployed image set is unchanged — versions for the network-operator
and nodewright-customizations refs were already pinned in adjacent
\`version:\` fields; this PR just makes the BOM reflect them correctly.
@mchmarny mchmarny requested review from a team as code owners May 5, 2026 18:32
@mchmarny mchmarny self-assigned this May 5, 2026
@github-actions github-actions Bot added the size/M label May 5, 2026
@github-actions

github-actions Bot commented May 5, 2026

Copy link
Copy Markdown
Contributor

Coverage Report ✅

Metric Value
Coverage 75.1%
Threshold 70%
Status Pass
Coverage Badge
![Coverage](https://img.shields.io/badge/coverage-75.1%25-green)

Merging this branch will increase overall coverage

Impacted Packages Coverage Δ 🤖
github.com/NVIDIA/aicr/pkg/bom 90.75% (+0.85%) 👍

Coverage by file

Changed files (no unit tests)

Changed File Coverage Δ Total Covered Missed 🤖
github.com/NVIDIA/aicr/pkg/bom/extract.go 98.11% (+0.41%) 106 (+19) 104 (+19) 2 👍

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

@coderabbitai

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

Addresses CodeRabbit's correctness finding on combineCRDTriplet. The
previous logic bailed out of the repository prepend whenever \`image\`
contained any slash, which silently dropped the registry for inputs
shaped like:

  repository: nvcr.io
  image:      nvidia/mellanox/doca-driver
  version:    v1.0

That produced \`nvidia/mellanox/doca-driver:v1.0\`, which ParseImageRef
then defaulted to docker.io — wrong registry, no warning.

Fix: only treat \`image\` as already-qualified when its first path
segment is an actual registry host (contains '.', ':', or equals
'localhost'), reusing the existing isRegistryHost helper. This pattern
doesn't appear in our current manifests today (NicClusterPolicy splits
at the registry boundary, so \`image\` is single-segment), but the
failure mode would silently corrupt the BOM for a plausible future
contribution. New regression test pins the corrected behavior.
@github-actions github-actions Bot added size/L and removed size/M labels May 5, 2026
@mchmarny mchmarny enabled auto-merge (squash) May 5, 2026 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Air-gap and private-registry mirroring guide

3 participants