Prerequisites

• I searched existing issues and found no duplicates
• I can reproduce this issue consistently
• This is not a security vulnerability (use Security Advisories instead)

Bug Description

Summary

undeploy.sh hangs indefinitely when custom resources have finalizers that require their controller to process, but the controller has already been uninstalled.

Root Cause

The undeploy script uninstalls operators before cleaning up their custom resources. When CRD deletion triggers Kubernetes' built-in customresourcecleanup.apiextensions.k8s.io finalizer, it waits for all CR instances to be deleted. If
those CRs have their own finalizers (e.g., vllm-agg DynamoGraphDeployment), the finalizer can't be processed because the controller (dynamo-platform) was already uninstalled. This creates a deadlock:

CRD delete blocked → waiting for CR instances to be deleted
└→ CR delete blocked → waiting for controller to process finalizer
└→ controller → already uninstalled

Ctrl+C does not help — re-running hits the same deadlock. The only workaround is manually patching finalizers:

kubectl patch dynamographdeployment vllm-agg -n dynamo-workload \
  -p '{"metadata":{"finalizers":[]}}' --type=merge

Proposed Fix

Add a pre-flight check to undeploy.sh that scans for custom resources with finalizers in non-bundle namespaces (e.g., user workload namespaces). If found, warn the user and exit with instructions to clean up before re-running undeploy:

WARNING: Found DynamoGraphDeployment "vllm-agg" in namespace "dynamo-workload" with active finalizers.
These must be deleted before undeploy can proceed — the controller that processes
their finalizers will be removed during teardown, causing an unrecoverable hang.

kubectl delete dynamographdeployments -A --all

Then re-run: ./undeploy.sh

This avoids the complexity of automatic finalizer clearing while preventing the silent hang.

Impact

Blocking (cannot proceed)

Component

CLI (aicr)

Regression?

Yes, this worked before (please specify version below)

Steps to Reproduce

CUJ2 testing on EKS (2x p5.48xlarge H100). The dynamo-workload namespace with vllm-agg DGD was created by the user workload step. Running undeploy.sh hung after deleting dynamographdeployments.nvidia.com CRD.

Expected Behavior

tell/warn the user, or fix the issue for them.

Actual Behavior

hangs

Environment

• AICR v0.10.15
• EKS v1.35.0-eks-3a10415
• Ubuntu 24.04, kernel 6.14.0-1018-aws

Command / Request Used

No response

Logs / Error Output

Additional Context

No response