Summary

Replace #922's 2-way crds.yaml / releases.yaml split with an N-way DAG-stratified
layout: one sub-helmfile per topological level (level-0.yaml … level-N.yaml).
Cross-chart CRD ordering is now handled by dependencyRefs alone; the
installsCRDs flag is removed. Self-reference cases (CRD + CR in the same
release) get a per-release disableValidation: true driven by a new
hasSelfRefCRDs registry field.

Motivation / Context

Issue #914 is the helm-diff "CRD not yet registered" race that fires on a fresh
cluster when a chart's templates reference CRDs installed by a sibling (or by
the same chart). #922 fixed the cross-chart case for a hand-marked subset of
charts via a 2-way sub-helmfile split. In live testing on kind we discovered
several gaps:

• nvsentinel was incorrectly marked installsCRDs: true (no in-bundle consumer
  of HealthCheck / NodeHealth CRs), causing it to land in crds.yaml and race
  against prometheus-operator-crds' diff in the same tier.
• network-operator's NodeFeatureRule CR depends on nfd's CRD, which wasn't
  marked.
• gpu-operator, kai-scheduler, and kubeflow-trainer all self-reference CRDs in
  their own crds/ directory from their own templates — a failure mode the
  2-way split can't solve regardless of which tier they land in.
• kube-prometheus-stack's CRDs ship in a subchart templates/, so its own
  templates render against an unregistered REST mapper even on fresh installs.

The fix is to stop hand-marking charts and let the DAG do the ordering. A
chart that produces CRDs and a chart that consumes them are already related
via dependencyRefs; the bundler now reads that graph directly.

Fixes: #914
Related: #922

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• Refactoring (no functional changes — installsCRDs removal, levels infra)

Component(s) Affected

• Recipe engine / data (pkg/recipe)
• Bundlers (pkg/bundler, pkg/component/*)
• Docs/examples (docs/, examples/)

Implementation Notes

recipe.TopologicalLevels (new in pkg/recipe/metadata.go): Kahn variant
that drains the entire current zero-in-degree generation as one batch before
processing its dependents. Returns [][]string where every component in level
i is mutually independent (no edges among them) and depends only on
components in levels <i. 14 unit-test cases cover cycle detection
(self-loop, two- and three-node), missing-dep surfacing as cycle, diamond,
wide root, the realistic AICR-shape, alphabetical tie-break within a level,
and a cross-check against TopologicalSort that flattened levels respect
every dependencyRefs edge.

Helmfile bundler refactor (pkg/bundler/deployer/helmfile/):

• splitFoldersByCRD → splitFoldersByLevel. N-way partition by parent
  component's DAG level. Folders with unknown parents default to level 0
  (defensive).
• writeHelmfileLayout emits one sub-helmfile per non-empty level
  (level-0.yaml, level-1.yaml, ...) and a top-level helmfile.yaml
  carrying helmfiles: [level-0.yaml, ...]. Single-level bundles collapse
  to a single helmfile.yaml (no sub-helmfile overhead).
• Cross-level needs: edges dissolve because helmfile processes the
  helmfiles: list sequentially. Within a level there's still a needs:
  chain for deterministic install order.

Registry schema:

• New hasSelfRefCRDs: bool field. Drives per-release disableValidation: true
  in helmfile. Marked components: gpu-operator (ClusterPolicy template +
  CRD), kai-scheduler (default-queue / default-shard), kubeflow-trainer
  (default ClusterTrainingRuntimes).
• installsCRDs field removed. The DAG-level partition makes it redundant
  for placement, and helmfile v1 doesn't accept args: per release (so we
  can't piggyback SSA on it). Removed from ComponentConfig, every registry
  entry, and TestRegistry_CRDOwnersStillMarked.

kube-prometheus-stack split:

• New prometheus-operator-crds component (chart 28.0.1, matches operator
  v0.90.1 pinned by kube-prometheus-stack 84.4.0).
• kube-prometheus-stack ships with crds.enabled: false and declares
  dependencyRefs: [prometheus-operator-crds], so the operator chart lands
  in a later level than its CRDs.

Server-side apply NOT emitted: helmfile v1 removed per-release args:,
and helmDefaults.args passes flags to helm-diff too (which doesn't accept
--server-side). The supported escape hatch is HELM_DIFF_IGNORE_UNKNOWN_FLAGS=true
as an env var — explicitly rejected because the bundle should be self-contained.
If the annotation-too-large failure surfaces in practice, follow-up will mark
specific releases with another registry flag.

Known follow-up (NOT in this PR):

• TopologicalSort (Kahn with per-dequeue tie-break) and TopologicalLevels
  (Kahn with generation-drain) produce different flat orderings. Folder
  numbering (001-N, driven by TopologicalSort) doesn't strictly match
  install order (driven by levels). Cosmetic — bundle deploys correctly
  either way. Unifying is a separate PR because it touches every chainsaw
  • UAT recipe golden.

Testing

unset GITLAB_TOKEN && make qualify
# 21/21 chainsaw e2e tests pass
# golangci-lint, yamllint, license headers, MDX, chart-pin checks all clean
# Grype vulnerability scan clean
# Coverage: 75.1% (no per-package regression >0.5%)

Live tested against a fresh kind cluster:

• helmfile apply on an eks/h100/training/kubeflow bundle → fixes nvsentinel
  PodMonitor race, network-operator NodeFeatureRule race
• gpu-operator ClusterPolicy self-reference resolved via disableValidation: true on the gpu-operator release
• kai-scheduler default-queue / default-shard self-reference resolved likewise
• kubeflow-trainer default ClusterTrainingRuntime self-reference resolved
  likewise

Per-package coverage on the affected paths:

• pkg/recipe: 90.2% (added 14 cases for TopologicalLevels)
• pkg/bundler/deployer/helmfile: stable; goldens regenerated for the new
  level-N.yaml layout

Risk Assessment

• Medium — Touches recipe, bundler, registry schema, and every
  generated bundle layout. The DAG-based partition is a strictly more general
  algorithm than the previous flag-based one; existing recipes produce
  correct (and structurally simpler) bundles. The schema change is removing
  a field, not adding one — recipes that referenced installsCRDs: true
  in YAML overlays would be silently ignored on the new schema, so a quick
  grep across consumers before merge is worthwhile.

Rollout notes:

• No CLI flag, no API surface change.
• Operators regenerating bundles after merge will see the new
  level-N.yaml layout instead of crds.yaml + releases.yaml. Both layouts
  are valid helmfile.yaml inputs; existing helmfile apply / helmfile diff invocations against the bundle directory work without changes.
• Any downstream tooling that grepped for crds.yaml / releases.yaml in a
  generated bundle needs to be updated to level-*.yaml. (None known
  in-tree; chainsaw bundle-helmfile test updated.)

Checklist

• Tests pass locally (make test with -race)
• Linter passes (make lint)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality
• I updated docs if user-facing behavior changed
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)