Summary
When bundling with --deployer argocd-helm and pushing to an OCI registry, the generated gpu-operator-pre ArgoCD Application uses spec.source.path instead of spec.source.chart. ArgoCD then tries to resolve the bundle's targetRevision as a tag on the OCI registry base rather than on the named chart, which always fails with not found.
The root aicr-stack (app-of-apps) Application has the same shape and also fails to load its target state.
The downstream effect on a real cluster is severe: gpu-operator-pre never syncs, the nvidia-kernel-module-params ConfigMap is never created, the GPU Operator ClusterPolicy stays notReady ("Failed to reconcile state-driver: ... ConfigMap "nvidia-kernel-module-params" not found"), the NVIDIA driver is never installed on GPU nodes, and downstream consumers (e.g. nvidia-dra-driver-gpu kubelet plugin) hang in Init forever waiting for /run/nvidia/driver.
Environment
aicr built from main (post-v0.13.0)
- Deployer:
argocd-helm
- Bundle artifact pushed to a private OCI registry as a single Helm chart, e.g.
oci://<registry>/<org>/<chart>:<tag>
- ArgoCD: argo-cd Helm chart, default install
- Cluster: EKS with GB200 (Ubuntu 24.04)
Reproduction
aicr bundle \
--recipe recipe.yaml \
--deployer argocd-helm \
--dynamic gpuoperator:driver.version \
--output oci://<registry>/<org>/<chart>:<tag>
# On the target cluster (ArgoCD already installed, OCI repo secret registered):
helm install <release> oci://<registry>/<org>/<chart>:<tag> \
--namespace argocd \
--set repoURL=oci://<registry>/<org> \
--set gpuoperator.driver.version=<version>
After install, kubectl get applications -n argocd shows:
NAME SYNC STATUS HEALTH STATUS
aicr-stack Unknown Healthy
gpu-operator-pre Unknown Healthy
gpu-operator OutOfSync Healthy
nvidia-dra-driver-gpu OutOfSync Progressing
...
Generated Application (broken)
# Application: gpu-operator-pre
spec:
source:
repoURL: oci://<registry>/<org>
path: "005-gpu-operator-pre" # ← wrong: 'path' is for Git
targetRevision: <tag>
helm:
values: |
driver:
version: <version>
ArgoCD error:
Failed to load target state: failed to generate manifest for source 1 of 1:
rpc error: code = Unknown desc = failed to resolve revision "<tag>":
cannot get digest for revision <tag>:
<registry>/<org>:<tag>: not found
The <registry>/<org>:<tag> lookup fails because ArgoCD is treating repoURL as the full OCI artifact and path as a sub-directory inside it, which is Git-source semantics. For OCI Helm sources, ArgoCD needs chart: <name>.
Expected Generated Application
# Application: gpu-operator-pre
spec:
source:
repoURL: oci://<registry>/<org>
chart: gpu-operator-pre # ← correct for OCI Helm
targetRevision: <tag>
helm:
values: |
driver:
version: <version>
By comparison, the sibling gpu-operator Application (which ArgoCD does resolve correctly) already uses chart: gpu-operator against an OCI Helm source.
Suggested Fix
In the argocd-helm deployer's Application template, switch path: NNN-<name> to chart: <name> for child charts that resolve to OCI Helm sources packaged inside the bundle artifact.
Impact
- Critical for any consumer trying to deploy AICR bundles via
argocd-helm from an OCI registry
- Silent at bundle build time; only surfaces after
helm install on the target cluster
- Cascades into the GPU stack never coming up (driver missing → DRA kubelet plugin blocked)
Summary
When bundling with
--deployer argocd-helmand pushing to an OCI registry, the generatedgpu-operator-preArgoCDApplicationusesspec.source.pathinstead ofspec.source.chart. ArgoCD then tries to resolve the bundle'stargetRevisionas a tag on the OCI registry base rather than on the named chart, which always fails withnot found.The root
aicr-stack(app-of-apps) Application has the same shape and also fails to load its target state.The downstream effect on a real cluster is severe:
gpu-operator-prenever syncs, thenvidia-kernel-module-paramsConfigMap is never created, the GPU OperatorClusterPolicystaysnotReady("Failed to reconcile state-driver: ... ConfigMap "nvidia-kernel-module-params" not found"), the NVIDIA driver is never installed on GPU nodes, and downstream consumers (e.g.nvidia-dra-driver-gpukubelet plugin) hang inInitforever waiting for/run/nvidia/driver.Environment
aicrbuilt frommain(post-v0.13.0)argocd-helmoci://<registry>/<org>/<chart>:<tag>Reproduction
After install,
kubectl get applications -n argocdshows:Generated Application (broken)
ArgoCD error:
The
<registry>/<org>:<tag>lookup fails because ArgoCD is treatingrepoURLas the full OCI artifact andpathas a sub-directory inside it, which is Git-source semantics. For OCI Helm sources, ArgoCD needschart: <name>.Expected Generated Application
By comparison, the sibling
gpu-operatorApplication (which ArgoCD does resolve correctly) already useschart: gpu-operatoragainst an OCI Helm source.Suggested Fix
In the
argocd-helmdeployer's Application template, switchpath: NNN-<name>tochart: <name>for child charts that resolve to OCI Helm sources packaged inside the bundle artifact.Impact
argocd-helmfrom an OCI registryhelm installon the target cluster