[Bug]: ate-setup delete demo returns before the demo namespace finishes terminating #3525

What happened?

ate-setup delete demo <name> (Substrate.Delete in cmd/ate-setup/internal/demos/substrate.go) calls e.Kube.DeleteBytes(ctx, manifest) on the demo's WorkerPoolManifest and returns while the demo Namespace is still Terminating.

Running ate-setup deploy demo <name> right after ate-setup delete demo <name> (for example, redeploying counter with --with-external-volume) fails because the Kubernetes NamespaceLifecycle admission controller rejects new objects in a terminating namespace:

Error: while applying workerpool/counter -n ate-demo-counter: workerpools.ate.dev "counter" is forbidden: unable to create new content in namespace ate-demo-counter because it is being terminated

Expected Behavior

ate-setup delete demo <name> and ate-setup deploy demo <name> can be run back to back, or the async behavior is an explicit CLI contract.

Steps to Reproduce

1. Deploy ate-system and the counter demo on a cluster:

   go run ./cmd/ate-setup deploy demo counter

2. Delete and immediately redeploy the demo:

   go run ./cmd/ate-setup delete demo counter
   go run ./cmd/ate-setup deploy demo counter

3. deploy demo counter fails applying workerpool/counter while namespace/ate-demo-counter is still terminating.

Sandbox Runtime

Both / Runtime Agnostic

Agent Substrate Version / Commit SHA

8a2c015

Kubernetes Version & Environment

kind v0.33.0 / Kubernetes v1.37.0

Host OS & Architecture

Linux (x86_64)

Relevant Logs and Diagnostic Output

[step]: demo-counter_deploy
[step]: ensure_crds
[step]: images: built from source with ko
  (ko resolve -f - took 7.003s)
  (apply of 2 objects took 0.017s)
Error: while applying workerpool/counter -n ate-demo-counter: workerpools.ate.dev "counter" is forbidden: unable to create new content in namespace ate-demo-counter because it is being terminated

Additional Context

Client.Delete (cmd/ate-setup/internal/kube/apply.go) documents itself as the kubectl delete --ignore-not-found -f equivalent. kubectl delete defaults to --wait=true, whereas Client.DeleteOne calls ri.Delete via client-go and returns as soon as deletion is requested.

Meanwhile, Substrate.Deploy blocks synchronously on workload rollouts and golden snapshot creation, and DeleteAteSystem (cmd/ate-setup/internal/steps/delete.go) calls e.Kube.WaitDeleted on ate-system and podcertificate-controller-system.

For discussion: should ate-setup delete demo <name> wait on demo Namespace deletion by default (matching kubectl delete and DeleteAteSystem), or keep deletion asynchronous and leave kubectl wait --for=delete namespace/<ns> to callers that immediately redeploy?

Confirmation

• I have searched existing issues and verified that this is not a duplicate.
• I have verified that this issue occurs on the latest commit on main.