Summary

Fix undeploy script PVC deletion ordering that causes namespace termination hangs, make PVC deletion opt-in, harden deploy script error handling, and add comprehensive deployment script reference documentation.

Motivation / Context

During redeployment of a dynamo inference bundle on EKS, the undeploy script hung indefinitely. Root cause: when multiple components share a namespace (e.g., monitoring with prometheus-adapter, k8s-ephemeral-storage-metrics, kube-prometheus-stack), PVC deletion ran after each component uninstall. The StatefulSet owner (kube-prometheus-stack) was uninstalled last, so earlier PVC deletions hung on kubernetes.io/pvc-protection finalizers.

Additionally, the deploy script silently masked all pre-install manifest errors, accepted unknown CLI flags without warning, and the deployment script behaviors were undocumented — making it difficult for users to succeed without reading the script internals.

Fixes: N/A
Related: N/A

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Refactoring (no functional changes)
• Build/CI/tooling

Component(s) Affected

• CLI (cmd/aicr, pkg/cli)
• API server (cmd/aicrd, pkg/api, pkg/server)
• Recipe engine / data (pkg/recipe)
• Bundlers (pkg/bundler, pkg/component/*)
• Collectors / snapshotter (pkg/collector, pkg/snapshotter)
• Validator (pkg/validator)
• Core libraries (pkg/errors, pkg/k8s)
• Docs/examples (docs/, examples/)
• Other: ____________

Implementation Notes

Undeploy (undeploy.sh.tmpl + helm.go)

• PVC deletion and namespace cleanup moved from per-component loop to a post-loop phase iterating over unique namespaces. This ensures all components in a shared namespace are fully uninstalled before any PVC/namespace cleanup.
• Added --delete-pvcs flag (default: off). PVCs are preserved by default to retain historical data (Prometheus metrics, etc.). EBS AZ-binding issues are situational and better handled reactively.
• PVC deletion skips protected namespaces (kube-system, kube-public, kube-node-lease, default).
• Namespace cleanup scoped to Helm/Kustomize-managed components only — manifest-only components no longer trigger namespace deletion.
• Added Namespaces field to undeployTemplateData and uniqueNamespaces() helper.

Deploy (deploy.sh.tmpl)

• Pre-install manifest apply: classifies error-like output lines (error:, forbidden, denied, timed out, etc.) and only suppresses known CRD-race patterns (no matches for kind, ensure CRDs are installed first). Informational lines (created, configured) no longer cause false failures.
• Unknown CLI flags now error instead of being silently ignored.

Documentation

• Added "Deploy Script Behavior" and "Undeploy Script Behavior" reference sections to docs/user/cli-reference.md and site/content/docs/user/cli-reference.md, covering:
  • Pre-install manifest CRD ordering and error filtering
  • --best-effort and async component behavior
  • PVC preservation semantics and --delete-pvcs flag
  • Shared namespace ordering rationale
  • Stuck release and orphaned webhook handling
• Added doc pointers in generated deploy.sh and undeploy.sh script headers linking to the CLI Reference, so users who run the scripts directly can find detailed behavior docs.
• Added node labels/taints reference section to demos/cuj2-demo.md.
• Updated bundle command examples and demos/workloads/inference/vllm-agg.yaml to use dedicated=gpu-workload / dedicated=system-workload naming convention.

Testing

go test -race ./pkg/bundler/deployer/helm/...

• All bundler tests pass including new tests for --delete-pvcs flag behavior, protected namespace exclusion, and uniqueNamespaces().
• Verified generated undeploy.sh no longer has PVC deletion in per-component loop.
• Live tested on EKS cluster: undeploy completed without hangs on shared monitoring namespace.

Risk Assessment

• Low — Isolated change, well-tested, easy to revert

Rollout notes: The --delete-pvcs flag is a new opt-in behavior. Existing users who relied on automatic PVC deletion during undeploy will need to add --delete-pvcs to their undeploy commands. This is intentional — preserving data by default is the safer choice.

Checklist

• Tests pass locally (make test with -race)
• Linter passes (make lint) — golangci-lint version mismatch (pre-existing)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality
• I updated docs if user-facing behavior changed
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)