Prerequisites

• I searched existing issues and found no duplicates
• I can reproduce this issue consistently
• This is not a security vulnerability (use Security Advisories instead)

Bug Description

• go run ./cmd/aicr recipe --service eks --accelerator h100 --intent inference --os ubuntu --platform dynamo -o /tmp/aicr3-dynamo-recipe.yaml

  1. unclear that this is not against the live-cluster.
  2. does not print a list of components, only numbers 18 components
  3. tried -o table giving me a distorted mix of jsonPath[3] value and YAML that are not indented or escaped.

• go run ./cmd/aicr recipe --intent inference --platform dynamo -o /tmp/aicr3-dynamo-recipe.yaml shows any values without clarity whether is good, bad or actually some kind of autodetection will happen.

• go run ./cmd/aicr validate without --no-cluster:

  • [cli] agent log streaming skipped: pod did not become ready: ... – which pod? Give ns/name.
  • [cli] command failed: error=[INTERNAL] failed to capture snapshot: [INTERNAL] job failed (auto-injected node selector nvidia.com/gpu.present=true — if no GPU nodes are schedulable, pass --node-selector or --require-gpu explicitly): [TIMEOUT] job completion timeout: context deadline exceeded exitCode=8 – tell me which flags exactly.
  • aicr validate --recipe recipe.yaml --phase deployment --output dry-run.json --node-selector nvidia.com/gpu.present=true \ --toleration dedicated=worker-workload:NoSchedule \ --toleration dedicated=worker-workload:NoExecute – [cli] validator completed: name=expected-resources status=failed
    1. this line is printed in green. Should be red.
    2. doesn't say anything about which resources. Now I am lost as user.
    3. I found out later that the output has the message, but it is messy and noisy and seeing some chainsaw: no such file or directory error:

        "stdout": [
          "time=2026-06-15T11:33:11.629Z level=INFO msg=\"starting check\" name=expected-resources",
          "  Namespace agentgateway-system: Active",
          "  Namespace kube-system: Active",
          "  Namespace cert-manager: Active",
          "  Namespace dynamo-system: Active",
          "  Namespace gpu-operator: Active",
          "  Namespace monitoring: Active",
          "  Namespace kai-scheduler: Active",
          "  Namespace node-feature-discovery: Active",
          "  Namespace skyhook: Active",
          "  Namespace nvidia-dra-driver: Active",
          "  Namespace nvsentinel: Active",
          "  Nodewright tuning: complete",
          "  ClusterPolicy cluster-policy: ready",
          "  DaemonSet nvidia-dra-driver/nvidia-dra-driver-gpu-kubelet-plugin: healthy",
          "time=2026-06-15T11:33:11.870Z level=INFO msg=\"running health check assertions\" components=18",
          "Failed resources:",
          "  agentgateway: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  agentgateway-crds: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  aws-ebs-csi-driver: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  aws-efa: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  cert-manager: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  dynamo-platform: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  gpu-operator: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  grove: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  k8s-ephemeral-storage-metrics: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  kai-scheduler: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  kube-prometheus-stack: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  nfd: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  nodewright-customizations: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  nodewright-operator: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  nvidia-dra-driver-gpu: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  nvsentinel: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  prometheus-adapter: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "  prometheus-operator-crds: chainsaw health check failed",
          "error: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",
          "time=2026-06-15T11:33:11.873Z level=ERROR msg=FAIL message=\"[NOT_FOUND] expected resource check failed: 18 issue(s):\\n  agentgateway: chainsaw health check failed\\nerror: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory\\n  agentgateway-crds: chainsaw health check failed\\nerror: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory\\n  aws-ebs-csi-driver: chainsaw health check failed\\nerror: [INTERNAL] failed to e... [truncated 2442 chars]"
       

• I ignore the error and go ahead with bundling aicr bundle --recipe recipe.yaml --accelerated-node-selector nodeGroup=gpu-worker --accelerated-node-toleration dedicated=worker-workload:NoSchedule --accelerated-node-toleration dedicated=worker-workload:NoExecute --system-node-selector nodeGroup=system-worker --system-node-toleration dedicated=system-workload:NoSchedule --system-node-toleration dedicated=system-workload:NoExecute --output bundle --storage-class ebs-csi-default-sc and calling deploy.sh:

• somewhere in the middle of lots of noisy output I see this. Should I do something?

  
  kubectl get pod -n kube-system -l "app.kubernetes.io/name=aws-ebs-csi-driver,app.kubernetes.io/instance=aws-ebs-csi-driver"
  

• The output need indention and color. This does not look trustworthy at all. Random log output.

• W0615 14:02:10.714233 86679 warnings.go:70] tls: failed to find any PEM data in certificate input – maybe part of grove install.

• Irritating: ```Error: UPGRADE FAILED: cannot patch "skyhook-operator-controller-manager" with kind Deployment: Deployment.apps "skyhook-operator-controller-manager" is invalid: spec.selector: Invalid value: {"matchLabels":{"app.kubernetes.io/instance":"nodewright-operator","app.kubernetes.io/name":"nodewright","control-plane":"controller-manager"}}: field is immutable
  RETRY: nodewright-operator install failed (attempt 1/5), retrying in 5s...
  Pulled: ghcr.io/nvidia/nodewright/charts/nodewright:v0.17.0
  Digest: sha256:4f32bf09026b9b82c134fc8a2beaa304609b6aad6351e51c053316bcea47aaa2
  Error: UPGRADE FAILED: cannot patch "skyhook-operator-controller-manager" with kind Deployment: Deployment.apps "skyhook-operator-controller-manager" is invalid: spec.selector: Invalid value: {"matchLabels":{"app.kubernetes.io/instance":"nodewright-operator","app.kubernetes.io/name":"nodewright","control-plane":"controller-manager"}}: field is immutable
  RETRY: nodewright-operator install failed (attempt 2/5), retrying in 20s...

• ERROR: nodewright-operator install failed after 6 attempts

• after manually removing the skyhook deployment and rerunning deploy.sh, it succeeded.

• aicr validate \ T/kubectx-shell-2593990620.yaml main --recipe recipe.yaml \ --toleration dedicated=worker-workload:NoSchedule \ --toleration dedicated=worker-workload:NoExecute \ --phase all \ --output report.json leads to [cli] phase completed: phase=deployment status=failed validators=4 passed=3 failed=1 duration=22.755715291s, looking into dry-run.json reveals: failed\nerror: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory\n kai-scheduler: chainsaw health check failed\nerror: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory\n kube-prometheus-stack: chainsaw health check failed\nerror: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory\n nfd: chainsaw health check failed\nerror: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory\n nodewright-customizations: chainsaw health check failed\nerror: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory\n nodewright-operator: chainsaw health check failed\nerror: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory\n nvidia-dra-driver-gpu: chainsaw health check failed\nerror: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory\n nvsentinel: chainsaw health check failed\nerror: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory\n prometheus-adapter: chainsaw health check failed\nerror: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory\n prometheus-operator-crds: chainsaw health check failed\nerror: [INTERNAL] failed to execute chainsaw: fork/exec /usr/local/bin/chainsaw: no such file or directory",

Impact

High (major functionality broken)

Component

CLI (aicr)

Regression?

Unknown / first time using this feature

Steps to Reproduce

Follow the steps in https://github.com/ai-dynamo/dynamo/pull/10600/changes on an EKS cluster with 8x H100.

Expected Behavior

Deployment goes through with a good UX.

Actual Behavior

ERROR: nodewright-operator install failed after 6 attempts

Environment

• AICR version (CLI aicr version, API image tag, or commit SHA):
• Install method (release binary / build from source / container image):
• Platform (eks/gke/aks/oke/kind/lke/bcm/other):
• Kubernetes version:
• OS (ubuntu/cos/other) + version:
• Kernel version:
• GPU type (h100/h200/gb200/b200/a100/l40/rtx-pro-6000/other):
• Workload intent (training/inference):

Command / Request Used

No response

Logs / Error Output

Additional Context

No response