Skip to content

[Feature]: aicr mirror list image and chart discovery with air-gap tool output formats #949

Description

@haarchri

Prerequisites

  • I searched existing issues

Feature Summary

A new aicr mirror list command that discovers all container images and Helm charts referenced by a recipe and outputs them in formats that existing air-gap tools (Hauler, Zarf) can consume directly. Plus --mirror-registry on aicr bundle to rewrite image references for private registries.

Problem/Use Case

AICR recipes reference ~50 container images across 9+ public registries (nvcr.io, ghcr.io, gcr.io, quay.io, registry.k8s.io, docker.io, public.ecr.aws, ...) and ~12 Helm charts. In air-gapped or private-registry environments there is no way to:

  1. Discover which images and charts a recipe actually needs
  2. Feed that list into existing air-gap tooling
  3. Generate a bundle that points at the private registry

As agreed in #743, AICR should not implement its own registry push/copy logic — tools like Hauler and Zarf already do this well and users in air-gapped environments are already using them. AICR should make these tools first-class by providing discovery and output in their native formats.

Proposed Solution

aicr mirror list

# Table (default)
aicr mirror list --recipe recipe.yaml
# Hauler manifest — pipe straight into hauler
aicr mirror list --recipe recipe.yaml --format hauler > hauler-manifest.yaml
# Zarf package — pipe straight into zarf
aicr mirror list --recipe recipe.yaml --format zarf > zarf.yaml
# General scripting
aicr mirror list --recipe recipe.yaml --format json
aicr mirror list --recipe recipe.yaml --format yaml

Hauler workflow:

aicr mirror list --recipe recipe.yaml --format hauler > hauler-manifest.yaml
hauler store sync -f hauler-manifest.yaml
hauler store save -o aicr-airgap.tar.zst
# transfer to air-gapped side
hauler store load aicr-airgap.tar.zst
hauler store serve registry

Generated hauler-manifest.yaml:

apiVersion: content.hauler.cattle.io/v1
kind: Images
metadata:
  name: aicr-images
spec:
  images:
    - name: nvcr.io/nvidia/gpu-operator:v25.3.0
    - name: nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
    - name: nvcr.io/nvidia/driver:570.124.06-ubuntu22.04
    - name: nvcr.io/nvidia/cuda:12.8.1-base-ubi8
    - name: nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.8.0
    - name: nvcr.io/nvidia/k8s/container-toolkit:v1.17.5-ubuntu20.04
    - name: nvcr.io/nvidia/cloud-native/dcgm:4.1.0-1-ubuntu22.04
    - name: nvcr.io/nvidia/k8s/dcgm-exporter:4.1.0-4.0.0-ubuntu22.04
    - name: ghcr.io/nvidia/network-operator:v25.1.0
    - name: nvcr.io/nvidia/cloud-native/multus:v4.1.4
    - name: ghcr.io/kai-scheduler/kai-scheduler:v0.2.71
    - name: public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.41.0
    - name: registry.k8s.io/sig-storage/csi-provisioner:v5.2.0
    - name: registry.k8s.io/sig-storage/csi-attacher:v4.8.0
    # ... ~53 images total
---
apiVersion: content.hauler.cattle.io/v1
kind: Charts
metadata:
  name: aicr-charts
spec:
  charts:
    - name: gpu-operator
      repoURL: oci://ghcr.io/nvidia
      version: v25.3.0
    - name: network-operator
      repoURL: oci://ghcr.io/nvidia
      version: v25.1.0
    - name: kai-scheduler
      repoURL: oci://ghcr.io/kai-scheduler
      version: v0.2.71
    - name: kueue
      repoURL: oci://registry.k8s.io/kueue
      version: v0.12.1
    - name: aws-ebs-csi-driver
      repoURL: https://kubernetes-sigs.github.io/aws-ebs-csi-driver
      version: 2.40.0
    - name: aws-efa-k8s-device-plugin
      repoURL: https://aws.github.io/eks-charts
      version: v0.5.7
    - name: kube-prometheus-stack
      repoURL: https://prometheus-community.github.io/helm-charts
      version: 70.4.1
    # ... ~12 charts total

Zarf workflow:

aicr mirror list --recipe recipe.yaml --format zarf > zarf.yaml
zarf package create .
# transfer and deploy

Generated zarf.yaml:

kind: ZarfPackageConfig
metadata:
  name: aicr-airgap
  description: AICR container images and Helm charts for air-gapped deployment
components:
  - name: aicr
    required: true
    images:
      - nvcr.io/nvidia/gpu-operator:v25.3.0
      - nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
      - nvcr.io/nvidia/driver:570.124.06-ubuntu22.04
      - nvcr.io/nvidia/cuda:12.8.1-base-ubi8
      - nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.8.0
      - nvcr.io/nvidia/k8s/container-toolkit:v1.17.5-ubuntu20.04
      - nvcr.io/nvidia/cloud-native/dcgm:4.1.0-1-ubuntu22.04
      - nvcr.io/nvidia/k8s/dcgm-exporter:4.1.0-4.0.0-ubuntu22.04
      - ghcr.io/nvidia/network-operator:v25.1.0
      - nvcr.io/nvidia/cloud-native/multus:v4.1.4
      - ghcr.io/kai-scheduler/kai-scheduler:v0.2.71
      - public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.41.0
      - registry.k8s.io/sig-storage/csi-provisioner:v5.2.0
      - registry.k8s.io/sig-storage/csi-attacher:v4.8.0
      # ... ~53 images total
    charts:
      - name: gpu-operator
        version: v25.3.0
        url: oci://ghcr.io/nvidia/gpu-operator
      - name: network-operator
        version: v25.1.0
        url: oci://ghcr.io/nvidia/network-operator
      - name: kai-scheduler
        version: v0.2.71
        url: oci://ghcr.io/kai-scheduler/kai-scheduler
      - name: kueue
        version: v0.12.1
        url: oci://registry.k8s.io/kueue/kueue
      - name: aws-ebs-csi-driver
        version: 2.40.0
        url: https://kubernetes-sigs.github.io/aws-ebs-csi-driver
        repoName: aws-ebs-csi-driver
      - name: aws-efa-k8s-device-plugin
        version: v0.5.7
        url: https://aws.github.io/eks-charts
        repoName: aws-efa-k8s-device-plugin
      - name: kube-prometheus-stack
        version: 70.4.1
        url: https://prometheus-community.github.io/helm-charts
        repoName: kube-prometheus-stack
      # ... ~12 charts total

Bundle integration

Once images are in the private registry (via Hauler/Zarf/whatever), the bundle needs to point at them. How exactly this looks in practice still needs some refinement, whether --mirror-registry alone is enough depends on the final registry layout (e.g. nested vs flat path structure, how each tool organizes mirrored images). We'll figure out the right interface as we iterate on the integration.

aicr bundle --recipe recipe.yaml \
  --mirror-registry registry.internal \
  --vendor-charts \
  -o ./bundle/

--mirror-registry rewrites image refs in Helm values to point at the private registry
--vendor-charts pulls chart .tgz into the bundle at build time (already exists)

Success Criteria

  • aicr mirror list --recipe recipe.yaml discovers all images and charts
  • --format hauler produces a valid Hauler manifest that hauler store sync accepts
  • --format zarf produces a valid Zarf package that zarf package create accepts
  • --format json and --format yaml work for general scripting
  • --mirror-registry on aicr bundle rewrites image references in generated values
  • Documentation with end-to-end examples for both Hauler and Zarf workflows

Alternatives Considered

  • AICR implements its own registry copy/push — rejected in Air-gap and private-registry mirroring guide #743. Duplicates what Hauler/Zarf already do well, adds maintenance burden, and users in air-gapped environments already have these tools.
  • Only output a flat image list not enough. Hauler and Zarf each have their own manifest formats, and outputting those directly removes a manual step for users.

Component

CLI (aicr)

Priority

Important (would improve my workflow)

Compatibility / Breaking Changes

New command, no breaking changes to existing CLI
--vendor-charts already exists and is unchanged
--mirror-registry is a new optional flag on aicr bundle

Operational Considerations

The Hauler/Zarf output formats are based on their current manifest schemas. If those schemas change upstream, the output formats need updating.

Are you willing to contribute?

Yes, I can open a PR

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions