Summary

On a fresh cluster, helmfile apply (and helmfile diff) against an AICR helmfile bundle aborts before installing anything because helm-diff renders nodewright-customizations (which uses kind: Skyhook from skyhook.nvidia.com/v1alpha1) against live cluster discovery, and the Skyhook CRD is not yet registered — it is installed by nodewright-operator, which helmfile would install first by needs:, but the diff pre-pass runs before any release lands.

Same root cause for both diff and apply. helmfile sync would avoid it (no diff plugin), and helmfile apply --skip-diff-on-install works around it (current demo uses the latter).

Repro

1. Provision a fresh GKE H100 cluster (e.g. via gh workflow run uat-gcp.yaml -f skip_delete=true -f skip_tests=true).

2. Generate a helmfile-deployer bundle:

   spec:
     bundle:
       deployment:
         deployer: helmfile

   aicr bundle --config aicr-config.yaml

3. Install the helm-diff plugin: helm plugin install https://github.com/databus23/helm-diff.

4. From ./bundle/, run helmfile apply (or helmfile diff).

Observed

in ./helmfile.yaml: command \"/opt/homebrew/bin/helm\" exited with non-zero status:

ARGS:
  0: helm
  1: diff
  2: upgrade
  3: --allow-unreleased
  4: nodewright-customizations
  5: 006-nodewright-customizations
  ...
  8: --namespace
  9: skyhook

STDERR:
  Error: Failed to render chart: exit status 1: Error: unable to build kubernetes objects from release manifest: resource mapping not found for name: \"tuning\" namespace: \"skyhook\" from \"\": no matches for kind \"Skyhook\" in version \"skyhook.nvidia.com/v1alpha1\"
  ensure CRDs are installed first
  Error: plugin \"diff\" exited with error

helmfile's needs: correctly orders nodewright-operator → nodewright-customizations, but helm-diff performs its render pass against the live cluster's REST mapper before any release is installed, so the dependent chart can't resolve the kind.

Expected

helmfile apply against a freshly bundled output should succeed on a fresh cluster with no flag gymnastics. A user following demos/cuj1-gke-config.md shouldn't need to know about --skip-diff-on-install, plugin internals, or CRD/CR ordering quirks.

Workarounds (in increasing invasiveness)

• A. CLI flag, no bundle change: helmfile apply --skip-diff-on-install. helmfile bypasses helm diff for releases that don't yet exist, so the first install renders against the in-progress cluster (CRDs from nodewright-operator land before nodewright-customizations is rendered). Subsequent re-applies on the populated cluster diff normally. Currently used in demos/cuj1-gke-config.md.
• B. Two-phase install: helmfile -l name=nodewright-operator sync first, then helmfile apply. Explicit, no flag, but adds a step.
• C. Bundle-side fix (preferred): stop emitting a single helmfile that mixes CRD-owner and CRD-consumer charts. Options:
  • C1. Generate a multi-helmfiles: document with a helmfile.yaml that pulls a crds.yaml sub-helmfile first (helmfile renders sub-helmfiles sequentially). nodewright-operator (and any other CRD-owning charts — cert-manager, kube-prometheus-stack, nvsentinel, etc.) moves into the CRD layer.
  • C2. Continue emitting one helmfile.yaml but tag releases with crd: true on CRD owners and have the template wrap them in a helmfiles: block, so --skip-diff-on-install is the documented norm only as a re-apply fallback.
  • C3. Pre-install CRDs out-of-band via a generated crds/ directory + a small apply-crds.sh the bundle README invokes before helmfile apply. Mirrors what some upstream operator charts ship as a separate install path.

C1 is the most idiomatic helmfile pattern and survives future additions of CRD-providing charts.

Affected files

• pkg/bundler/deployer/helmfile/ — releases.go (release graph), templates/helmfile.yaml.tmpl (top-level shape), templates/README.md.tmpl (currently documents helmfile diff / helmfile apply with no plugin or CRD-ordering caveat).
• recipes/registry.yaml / recipes/overlays/* — any component that ships a CRD that other components reference would feed into the same bug. nodewright-operator is the one tripping today; the broader set includes any chart whose helm install registers a CRD that another chart's templates use directly.
• demos/cuj1-gke-config.md — already updated to --skip-diff-on-install; revert the workaround when this issue lands.

Operational considerations

• Bundle reproducibility shouldn't change — same chart versions, same values, just different helmfile.yaml layout.
• The helm-diff plugin is now effectively mandatory for any helmfile bundle (helmfile apply calls it under the hood). Worth surfacing in the generated README regardless of how this is resolved.

Severity

User-visible: demo-blocking on a fresh cluster, and the workaround is non-obvious. Low data-loss risk (we abort before mutating cluster state), high time-to-debug for a first-time user.

Are you willing to contribute?

Maybe, with guidance — happy to PR the README/template + --skip-diff-on-install hint immediately; the C1 bundle-side restructure benefits from a maintainer signoff on the preferred layout.