Skip to content

bug(bundler): argocd-helm generates gpu-operator-pre Application with path instead of chart — blocks OCI sync #1018

Description

@atif1996

Summary

When bundling with --deployer argocd-helm and pushing to an OCI registry, the generated gpu-operator-pre ArgoCD Application uses spec.source.path instead of spec.source.chart. ArgoCD then tries to resolve the bundle's targetRevision as a tag on the OCI registry base rather than on the named chart, which always fails with not found.

The root aicr-stack (app-of-apps) Application has the same shape and also fails to load its target state.

The downstream effect on a real cluster is severe: gpu-operator-pre never syncs, the nvidia-kernel-module-params ConfigMap is never created, the GPU Operator ClusterPolicy stays notReady ("Failed to reconcile state-driver: ... ConfigMap "nvidia-kernel-module-params" not found"), the NVIDIA driver is never installed on GPU nodes, and downstream consumers (e.g. nvidia-dra-driver-gpu kubelet plugin) hang in Init forever waiting for /run/nvidia/driver.

Environment

  • aicr built from main (post-v0.13.0)
  • Deployer: argocd-helm
  • Bundle artifact pushed to a private OCI registry as a single Helm chart, e.g. oci://<registry>/<org>/<chart>:<tag>
  • ArgoCD: argo-cd Helm chart, default install
  • Cluster: EKS with GB200 (Ubuntu 24.04)

Reproduction

aicr bundle \
  --recipe recipe.yaml \
  --deployer argocd-helm \
  --dynamic gpuoperator:driver.version \
  --output oci://<registry>/<org>/<chart>:<tag>

# On the target cluster (ArgoCD already installed, OCI repo secret registered):
helm install <release> oci://<registry>/<org>/<chart>:<tag> \
  --namespace argocd \
  --set repoURL=oci://<registry>/<org> \
  --set gpuoperator.driver.version=<version>

After install, kubectl get applications -n argocd shows:

NAME                       SYNC STATUS   HEALTH STATUS
aicr-stack                 Unknown       Healthy
gpu-operator-pre           Unknown       Healthy
gpu-operator               OutOfSync     Healthy
nvidia-dra-driver-gpu      OutOfSync     Progressing
...

Generated Application (broken)

# Application: gpu-operator-pre
spec:
  source:
    repoURL: oci://<registry>/<org>
    path: "005-gpu-operator-pre"          # ← wrong: 'path' is for Git
    targetRevision: <tag>
    helm:
      values: |
        driver:
          version: <version>

ArgoCD error:

Failed to load target state: failed to generate manifest for source 1 of 1:
rpc error: code = Unknown desc = failed to resolve revision "<tag>":
cannot get digest for revision <tag>:
<registry>/<org>:<tag>: not found

The <registry>/<org>:<tag> lookup fails because ArgoCD is treating repoURL as the full OCI artifact and path as a sub-directory inside it, which is Git-source semantics. For OCI Helm sources, ArgoCD needs chart: <name>.

Expected Generated Application

# Application: gpu-operator-pre
spec:
  source:
    repoURL: oci://<registry>/<org>
    chart: gpu-operator-pre                # ← correct for OCI Helm
    targetRevision: <tag>
    helm:
      values: |
        driver:
          version: <version>

By comparison, the sibling gpu-operator Application (which ArgoCD does resolve correctly) already uses chart: gpu-operator against an OCI Helm source.

Suggested Fix

In the argocd-helm deployer's Application template, switch path: NNN-<name> to chart: <name> for child charts that resolve to OCI Helm sources packaged inside the bundle artifact.

Impact

  • Critical for any consumer trying to deploy AICR bundles via argocd-helm from an OCI registry
  • Silent at bundle build time; only surfaces after helm install on the target cluster
  • Cascades into the GPU stack never coming up (driver missing → DRA kubelet plugin blocked)

Metadata

Metadata

Assignees

Type

Fields

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions