The orphaned-CRD cleanup pipeline introduced in NVIDIA#477 and the two helper
functions delete_release_cluster_resources and force_clear_namespace_finalizers
all share the shape:

    kubectl get ... 2>/dev/null \
      | jq -r '...' 2>/dev/null \
      | while read -r name; do ... done

With set -euo pipefail, a single transient kubectl failure (API 502, brief
auth refresh, etc.) causes the pipeline to exit non-zero and terminates the
script. In a typical run with ~15 components, this means any 100 ms control-plane
hiccup can kill undeploy.sh after Helm uninstalls succeeded but before
namespace cleanup, leaving the cluster in a partially-cleaned state that
the user must finish by hand.

Add a trailing || true to each of the three pipelines so transient API
errors during cleanup are tolerated. The adjacent ORPHANED_CRD_GROUPS loop
and stuck_crds extraction already use this pattern.

The orphaned-CRD cleanup pipeline introduced in NVIDIA#477 and the two helper
functions delete_release_cluster_resources and force_clear_namespace_finalizers
all share the shape:

    kubectl get ... 2>/dev/null \
      | jq -r '...' 2>/dev/null \
      | while read -r name; do ... done

With set -euo pipefail, a single transient kubectl failure (API 502, brief
auth refresh, etc.) causes the pipeline to exit non-zero and terminates the
script. In a typical run with ~15 components, this means any 100 ms control-plane
hiccup can kill undeploy.sh after Helm uninstalls succeeded but before
namespace cleanup, leaving the cluster in a partially-cleaned state that
the user must finish by hand.

Append a warn-on-failure handler to each of the three pipelines so transient
API errors during cleanup are tolerated AND visible: the script keeps going
into the post-flight verification phase, and operators see a clear stderr
warning naming the failing context (kind / release / namespace / component)
instead of nothing. The adjacent ORPHANED_CRD_GROUPS loop and stuck_crds
extraction already use the "tolerate-and-continue" pattern; this PR brings
the three remaining sites into line and adds context-rich warnings per
CodeRabbit review.

The orphaned-CRD cleanup pipeline introduced in NVIDIA#477 and the two helper
functions delete_release_cluster_resources and force_clear_namespace_finalizers
all share the shape:

    kubectl get ... 2>/dev/null \
      | jq -r '...' 2>/dev/null \
      | while read -r name; do ... done

With set -euo pipefail, a single transient kubectl failure (API 502, brief
auth refresh, etc.) causes the pipeline to exit non-zero and terminates the
script. In a typical run with ~15 components, this means any 100 ms control-plane
hiccup can kill undeploy.sh after Helm uninstalls succeeded but before
namespace cleanup, leaving the cluster in a partially-cleaned state that
the user must finish by hand.

Append a warn-on-failure handler to each of the three pipelines so transient
API errors during cleanup are tolerated AND visible: the script keeps going
into the post-flight verification phase, and operators see a clear stderr
warning naming the failing context (kind / release / namespace / component)
instead of nothing. The adjacent ORPHANED_CRD_GROUPS loop and stuck_crds
extraction already use the "tolerate-and-continue" pattern; this PR brings
the three remaining sites into line and adds context-rich warnings per
CodeRabbit review.

Combines three related hardening changes to the generated undeploy.sh
(previously split across NVIDIA#599, NVIDIA#601, NVIDIA#600). They share a file and a
theme -- make undeploy recover from partial or transient failure --
so bundling them avoids three sequential PR rebases over the same
template.

1. Transient-failure resilience (was NVIDIA#599).
   Three `kubectl | jq | while` pipelines in the post-uninstall cleanup
   path ran under `set -euo pipefail` with no fallback. A momentary
   control-plane 502 or auth refresh killed the whole script after
   Helm uninstalls succeeded, leaving orphan CRDs, webhooks, and
   `Active` namespaces. Wrap each pipeline's trailing `done` in
   `|| echo "Warning: ..." >&2` so the failure is visible but the
   script continues. Matches the `|| true` pattern already used in the
   adjacent ORPHANED_CRD_GROUPS loop.

2. Pre-flight CRD discovery via Helm annotation (was NVIDIA#601).
   check_release_for_stuck_crds sourced its CRD list only from
   `helm get manifest`, which returns CRDs under the chart's templates/
   section but omits CRDs installed from the chart's crds/ directory
   (the Helm-recommended layout used by most operator charts). CRs
   backed by crds/-installed CRDs (e.g., NIMService) slipped past the
   finalizer pre-flight. Union both sources (manifest + annotated) and
   dedupe. Uses `awk 'NF'` instead of `grep -v '^$'` to drop empty
   lines without the grep-exits-1-on-no-match behavior, which would
   abort the script under pipefail when a release has no CRDs.

3. Post-flight Helm-annotation CRD check (was NVIDIA#600).
   Helm-managed CRDs that survive the per-release deletion loop went
   unnoticed until the next deploy.sh run rejected them. Add a
   verification pass at the end of undeploy.sh that re-enumerates CRDs
   filtered by meta.helm.sh/release-name + release-namespace
   annotations and surfaces any leftovers as a post-flight warning.
   CRDs already being deleted (non-null .metadata.deletionTimestamp)
   are excluded: the earlier `kubectl delete crd ... --wait=false`
   returns immediately, so a slow finalizer can leave the CRD listed
   though cleanup is normal. Only truly stuck (not-yet-terminating)
   CRDs are reported.

Unit test TestUndeployScript_TransientFailureWarnsAndContinues
generates a bundle, stubs kubectl to exit non-zero, sources each of
the three helpers, and asserts the helper returns 0 and emits the
expected Warning: on stderr.

Fixes: N/A
Related: NVIDIA#477 (introduced the per-component CRD cleanup pipeline),
NVIDIA#599, NVIDIA#600, NVIDIA#601 (superseded by this PR).

Signed-off-by: Yuan Chen <[email protected]>

Combines three related hardening changes to the generated undeploy.sh
(previously split across NVIDIA#599, NVIDIA#601, NVIDIA#600). They share a file and a
theme -- make undeploy recover from partial or transient failure --
so bundling them avoids three sequential PR rebases over the same
template.

1. Transient-failure resilience (was NVIDIA#599).
   Three `kubectl | jq | while` pipelines in the post-uninstall cleanup
   path ran under `set -euo pipefail` with no fallback. A momentary
   control-plane 502 or auth refresh killed the whole script after
   Helm uninstalls succeeded, leaving orphan CRDs, webhooks, and
   `Active` namespaces. Wrap each pipeline's trailing `done` in
   `|| echo "Warning: ..." >&2` so the failure is visible but the
   script continues. Matches the `|| true` pattern already used in the
   adjacent ORPHANED_CRD_GROUPS loop.

2. Pre-flight CRD discovery via Helm annotation (was NVIDIA#601).
   check_release_for_stuck_crds sourced its CRD list only from
   `helm get manifest`, which returns CRDs under the chart's templates/
   section but omits CRDs installed from the chart's crds/ directory
   (the Helm-recommended layout used by most operator charts). CRs
   backed by crds/-installed CRDs (e.g., NIMService) slipped past the
   finalizer pre-flight. Union both sources (manifest + annotated) and
   dedupe. Uses `awk 'NF'` instead of `grep -v '^$'` to drop empty
   lines without the grep-exits-1-on-no-match behavior, which would
   abort the script under pipefail when a release has no CRDs.

3. Post-flight Helm-annotation CRD check (was NVIDIA#600).
   Helm-managed CRDs that survive the per-release deletion loop went
   unnoticed until the next deploy.sh run rejected them. Add a
   verification pass at the end of undeploy.sh that re-enumerates CRDs
   filtered by meta.helm.sh/release-name + release-namespace
   annotations and surfaces any leftovers as a post-flight warning.
   CRDs already being deleted (non-null .metadata.deletionTimestamp)
   are excluded: the earlier `kubectl delete crd ... --wait=false`
   returns immediately, so a slow finalizer can leave the CRD listed
   though cleanup is normal. Only truly stuck (not-yet-terminating)
   CRDs are reported.

Unit test TestUndeployScript_TransientFailureWarnsAndContinues
generates a bundle, stubs kubectl to exit non-zero, sources each of
the three helpers, and asserts the helper returns 0 and emits the
expected Warning: on stderr.

Fixes: N/A
Related: NVIDIA#477 (introduced the per-component CRD cleanup pipeline),
NVIDIA#599, NVIDIA#600, NVIDIA#601 (superseded by this PR).

Signed-off-by: Yuan Chen <[email protected]>

Combines three related hardening changes to the generated undeploy.sh
(previously split across NVIDIA#599, NVIDIA#601, NVIDIA#600). They share a file and a
theme -- make undeploy recover from partial or transient failure --
so bundling them avoids three sequential PR rebases over the same
template.

1. Transient-failure resilience (was NVIDIA#599).
   Three `kubectl | jq | while` pipelines in the post-uninstall cleanup
   path ran under `set -euo pipefail` with no fallback. A momentary
   control-plane 502 or auth refresh killed the whole script after
   Helm uninstalls succeeded, leaving orphan CRDs, webhooks, and
   `Active` namespaces. Wrap each pipeline's trailing `done` in
   `|| echo "Warning: ..." >&2` so the failure is visible but the
   script continues. Matches the `|| true` pattern already used in the
   adjacent ORPHANED_CRD_GROUPS loop.

2. Pre-flight CRD discovery via Helm annotation (was NVIDIA#601).
   check_release_for_stuck_crds sourced its CRD list only from
   `helm get manifest`, which returns CRDs under the chart's templates/
   section but omits CRDs installed from the chart's crds/ directory
   (the Helm-recommended layout used by most operator charts). CRs
   backed by crds/-installed CRDs (e.g., NIMService) slipped past the
   finalizer pre-flight. Union both sources (manifest + annotated) and
   dedupe. Uses `awk 'NF'` instead of `grep -v '^$'` to drop empty
   lines without the grep-exits-1-on-no-match behavior, which would
   abort the script under pipefail when a release has no CRDs.

3. Post-flight Helm-annotation CRD check (was NVIDIA#600).
   Helm-managed CRDs that survive the per-release deletion loop went
   unnoticed until the next deploy.sh run rejected them. Add a
   verification pass at the end of undeploy.sh that re-enumerates CRDs
   filtered by meta.helm.sh/release-name + release-namespace
   annotations and surfaces any leftovers as a post-flight warning.
   CRDs already being deleted (non-null .metadata.deletionTimestamp)
   are excluded: the earlier `kubectl delete crd ... --wait=false`
   returns immediately, so a slow finalizer can leave the CRD listed
   though cleanup is normal. Only truly stuck (not-yet-terminating)
   CRDs are reported.

Unit test TestUndeployScript_TransientFailureWarnsAndContinues
generates a bundle, stubs kubectl to exit non-zero, sources each of
the three helpers, and asserts the helper returns 0 and emits the
expected Warning: on stderr.

Fixes: N/A
Related: NVIDIA#477 (introduced the per-component CRD cleanup pipeline),
NVIDIA#599, NVIDIA#600, NVIDIA#601 (superseded by this PR).

Signed-off-by: Yuan Chen <[email protected]>

Combines three related hardening changes to the generated undeploy.sh
(previously split across NVIDIA#599, NVIDIA#601, NVIDIA#600). They share a file and a
theme -- make undeploy recover from partial or transient failure --
so bundling them avoids three sequential PR rebases over the same
template.

1. Transient-failure resilience (was NVIDIA#599).
   Three `kubectl | jq | while` pipelines in the post-uninstall cleanup
   path ran under `set -euo pipefail` with no fallback. A momentary
   control-plane 502 or auth refresh killed the whole script after
   Helm uninstalls succeeded, leaving orphan CRDs, webhooks, and
   `Active` namespaces. Wrap each pipeline's trailing `done` in
   `|| echo "Warning: ..." >&2` so the failure is visible but the
   script continues. Matches the `|| true` pattern already used in the
   adjacent ORPHANED_CRD_GROUPS loop.

2. Pre-flight CRD discovery via Helm annotation (was NVIDIA#601).
   check_release_for_stuck_crds sourced its CRD list only from
   `helm get manifest`, which returns CRDs under the chart's templates/
   section but omits CRDs installed from the chart's crds/ directory
   (the Helm-recommended layout used by most operator charts). CRs
   backed by crds/-installed CRDs (e.g., NIMService) slipped past the
   finalizer pre-flight. Union both sources (manifest + annotated) and
   dedupe. Uses `awk 'NF'` instead of `grep -v '^$'` to drop empty
   lines without the grep-exits-1-on-no-match behavior, which would
   abort the script under pipefail when a release has no CRDs.

3. Post-flight Helm-annotation CRD check (was NVIDIA#600).
   Helm-managed CRDs that survive the per-release deletion loop went
   unnoticed until the next deploy.sh run rejected them. Add a
   verification pass at the end of undeploy.sh that re-enumerates CRDs
   filtered by meta.helm.sh/release-name + release-namespace
   annotations and surfaces any leftovers as a post-flight warning.
   CRDs already being deleted (non-null .metadata.deletionTimestamp)
   are excluded: the earlier `kubectl delete crd ... --wait=false`
   returns immediately, so a slow finalizer can leave the CRD listed
   though cleanup is normal. Only truly stuck (not-yet-terminating)
   CRDs are reported.

Unit test TestUndeployScript_TransientFailureWarnsAndContinues
generates a bundle, stubs kubectl to exit non-zero, sources each of
the three helpers, and asserts the helper returns 0 and emits the
expected Warning: on stderr.

Fixes: N/A
Related: NVIDIA#477 (introduced the per-component CRD cleanup pipeline),
NVIDIA#599, NVIDIA#600, NVIDIA#601 (superseded by this PR).

Signed-off-by: Yuan Chen <[email protected]>

Combines three related hardening changes to the generated undeploy.sh
(previously split across NVIDIA#599, NVIDIA#601, NVIDIA#600). They share a file and a
theme -- make undeploy recover from partial or transient failure --
so bundling them avoids three sequential PR rebases over the same
template.

1. Transient-failure resilience (was NVIDIA#599).
   Three `kubectl | jq | while` pipelines in the post-uninstall cleanup
   path ran under `set -euo pipefail` with no fallback. A momentary
   control-plane 502 or auth refresh killed the whole script after
   Helm uninstalls succeeded, leaving orphan CRDs, webhooks, and
   `Active` namespaces. Wrap each pipeline's trailing `done` in
   `|| echo "Warning: ..." >&2` so the failure is visible but the
   script continues. Matches the `|| true` pattern already used in the
   adjacent ORPHANED_CRD_GROUPS loop.

2. Pre-flight CRD discovery via Helm annotation (was NVIDIA#601).
   check_release_for_stuck_crds sourced its CRD list only from
   `helm get manifest`, which returns CRDs under the chart's templates/
   section but omits CRDs installed from the chart's crds/ directory
   (the Helm-recommended layout used by most operator charts). CRs
   backed by crds/-installed CRDs (e.g., NIMService) slipped past the
   finalizer pre-flight. Union both sources (manifest + annotated) and
   dedupe. Uses `awk 'NF'` instead of `grep -v '^$'` to drop empty
   lines without the grep-exits-1-on-no-match behavior, which would
   abort the script under pipefail when a release has no CRDs.

3. Post-flight Helm-annotation CRD check (was NVIDIA#600).
   Helm-managed CRDs that survive the per-release deletion loop went
   unnoticed until the next deploy.sh run rejected them. Add a
   verification pass at the end of undeploy.sh that re-enumerates CRDs
   filtered by meta.helm.sh/release-name + release-namespace
   annotations and surfaces any leftovers as a post-flight warning.
   CRDs already being deleted (non-null .metadata.deletionTimestamp)
   are excluded: the earlier `kubectl delete crd ... --wait=false`
   returns immediately, so a slow finalizer can leave the CRD listed
   though cleanup is normal. Only truly stuck (not-yet-terminating)
   CRDs are reported.

Unit test TestUndeployScript_TransientFailureWarnsAndContinues
generates a bundle, stubs kubectl to exit non-zero, sources each of
the three helpers, and asserts the helper returns 0 and emits the
expected Warning: on stderr.

Fixes: N/A
Related: NVIDIA#477 (introduced the per-component CRD cleanup pipeline),

Signed-off-by: Yuan Chen <[email protected]>

Combines three related hardening changes to the generated undeploy.sh
(previously split across NVIDIA#599, NVIDIA#601, NVIDIA#600). They share a file and a
theme -- make undeploy recover from partial or transient failure --
so bundling them avoids three sequential PR rebases over the same
template.

1. Transient-failure resilience (was NVIDIA#599).
   Three `kubectl | jq | while` pipelines in the post-uninstall cleanup
   path ran under `set -euo pipefail` with no fallback. A momentary
   control-plane 502 or auth refresh killed the whole script after
   Helm uninstalls succeeded, leaving orphan CRDs, webhooks, and
   `Active` namespaces. Wrap each pipeline's trailing `done` in
   `|| echo "Warning: ..." >&2` so the failure is visible but the
   script continues. Matches the `|| true` pattern already used in the
   adjacent ORPHANED_CRD_GROUPS loop.

2. Pre-flight CRD discovery via Helm annotation (was NVIDIA#601).
   check_release_for_stuck_crds sourced its CRD list only from
   `helm get manifest`, which returns CRDs under the chart's templates/
   section but omits CRDs installed from the chart's crds/ directory
   (the Helm-recommended layout used by most operator charts). CRs
   backed by crds/-installed CRDs (e.g., NIMService) slipped past the
   finalizer pre-flight. Union both sources (manifest + annotated) and
   dedupe. Uses `awk 'NF'` instead of `grep -v '^$'` to drop empty
   lines without the grep-exits-1-on-no-match behavior, which would
   abort the script under pipefail when a release has no CRDs.

3. Post-flight Helm-annotation CRD check (was NVIDIA#600).
   Helm-managed CRDs that survive the per-release deletion loop went
   unnoticed until the next deploy.sh run rejected them. Add a
   verification pass at the end of undeploy.sh that re-enumerates CRDs
   filtered by meta.helm.sh/release-name + release-namespace
   annotations and surfaces any leftovers as a post-flight warning.
   CRDs already being deleted (non-null .metadata.deletionTimestamp)
   are excluded: the earlier `kubectl delete crd ... --wait=false`
   returns immediately, so a slow finalizer can leave the CRD listed
   though cleanup is normal. Only truly stuck (not-yet-terminating)
   CRDs are reported.

Unit test TestUndeployScript_TransientFailureWarnsAndContinues
generates a bundle, stubs kubectl to exit non-zero, sources each of
the three helpers, and asserts the helper returns 0 and emits the
expected Warning: on stderr.

Fixes: N/A
Related: NVIDIA#477 (introduced the per-component CRD cleanup pipeline),

Signed-off-by: Yuan Chen <[email protected]>

Combines three related hardening changes to the generated undeploy.sh
(previously split across NVIDIA#599, NVIDIA#601, NVIDIA#600). They share a file and a
theme -- make undeploy recover from partial or transient failure --
so bundling them avoids three sequential PR rebases over the same
template.

1. Transient-failure resilience (was NVIDIA#599).
   Three `kubectl | jq | while` pipelines in the post-uninstall cleanup
   path ran under `set -euo pipefail` with no fallback. A momentary
   control-plane 502 or auth refresh killed the whole script after
   Helm uninstalls succeeded, leaving orphan CRDs, webhooks, and
   `Active` namespaces. Wrap each pipeline's trailing `done` in
   `|| echo "Warning: ..." >&2` so the failure is visible but the
   script continues. Matches the `|| true` pattern already used in the
   adjacent ORPHANED_CRD_GROUPS loop.

2. Pre-flight CRD discovery via Helm annotation (was NVIDIA#601).
   check_release_for_stuck_crds sourced its CRD list only from
   `helm get manifest`, which returns CRDs under the chart's templates/
   section but omits CRDs installed from the chart's crds/ directory
   (the Helm-recommended layout used by most operator charts). CRs
   backed by crds/-installed CRDs (e.g., NIMService) slipped past the
   finalizer pre-flight. Union both sources (manifest + annotated) and
   dedupe. Uses `awk 'NF'` instead of `grep -v '^$'` to drop empty
   lines without the grep-exits-1-on-no-match behavior, which would
   abort the script under pipefail when a release has no CRDs.

3. Post-flight Helm-annotation CRD check (was NVIDIA#600).
   Helm-managed CRDs that survive the per-release deletion loop went
   unnoticed until the next deploy.sh run rejected them. Add a
   verification pass at the end of undeploy.sh that re-enumerates CRDs
   filtered by meta.helm.sh/release-name + release-namespace
   annotations and surfaces any leftovers as a post-flight warning.
   CRDs already being deleted (non-null .metadata.deletionTimestamp)
   are excluded: the earlier `kubectl delete crd ... --wait=false`
   returns immediately, so a slow finalizer can leave the CRD listed
   though cleanup is normal. Only truly stuck (not-yet-terminating)
   CRDs are reported.

Unit test TestUndeployScript_TransientFailureWarnsAndContinues
generates a bundle, stubs kubectl to exit non-zero, sources each of
the three helpers, and asserts the helper returns 0 and emits the
expected Warning: on stderr.

Fixes: N/A
Related: NVIDIA#477 (introduced the per-component CRD cleanup pipeline),

Signed-off-by: Yuan Chen <[email protected]>

Combines three related hardening changes to the generated undeploy.sh
(previously split across NVIDIA#599, NVIDIA#601, NVIDIA#600). They share a file and a
theme -- make undeploy recover from partial or transient failure --
so bundling them avoids three sequential PR rebases over the same
template.

1. Transient-failure resilience (was NVIDIA#599).
   Three `kubectl | jq | while` pipelines in the post-uninstall cleanup
   path ran under `set -euo pipefail` with no fallback. A momentary
   control-plane 502 or auth refresh killed the whole script after
   Helm uninstalls succeeded, leaving orphan CRDs, webhooks, and
   `Active` namespaces. Wrap each pipeline's trailing `done` in
   `|| echo "Warning: ..." >&2` so the failure is visible but the
   script continues. Matches the `|| true` pattern already used in the
   adjacent ORPHANED_CRD_GROUPS loop.

2. Pre-flight CRD discovery via Helm annotation (was NVIDIA#601).
   check_release_for_stuck_crds sourced its CRD list only from
   `helm get manifest`, which returns CRDs under the chart's templates/
   section but omits CRDs installed from the chart's crds/ directory
   (the Helm-recommended layout used by most operator charts). CRs
   backed by crds/-installed CRDs (e.g., NIMService) slipped past the
   finalizer pre-flight. Union both sources (manifest + annotated) and
   dedupe. Uses `awk 'NF'` instead of `grep -v '^$'` to drop empty
   lines without the grep-exits-1-on-no-match behavior, which would
   abort the script under pipefail when a release has no CRDs.

3. Post-flight Helm-annotation CRD check (was NVIDIA#600).
   Helm-managed CRDs that survive the per-release deletion loop went
   unnoticed until the next deploy.sh run rejected them. Add a
   verification pass at the end of undeploy.sh that re-enumerates CRDs
   filtered by meta.helm.sh/release-name + release-namespace
   annotations and surfaces any leftovers as a post-flight warning.
   CRDs already being deleted (non-null .metadata.deletionTimestamp)
   are excluded: the earlier `kubectl delete crd ... --wait=false`
   returns immediately, so a slow finalizer can leave the CRD listed
   though cleanup is normal. Only truly stuck (not-yet-terminating)
   CRDs are reported.

Unit test TestUndeployScript_TransientFailureWarnsAndContinues
generates a bundle, stubs kubectl to exit non-zero, sources each of
the three helpers, and asserts the helper returns 0 and emits the
expected Warning: on stderr.

Fixes: N/A
Related: NVIDIA#477 (introduced the per-component CRD cleanup pipeline),

Signed-off-by: Yuan Chen <[email protected]>

Combines three related hardening changes to the generated undeploy.sh
(previously split across NVIDIA#599, NVIDIA#601, NVIDIA#600). They share a file and a
theme -- make undeploy recover from partial or transient failure --
so bundling them avoids three sequential PR rebases over the same
template.

1. Transient-failure resilience (was NVIDIA#599).
   Three `kubectl | jq | while` pipelines in the post-uninstall cleanup
   path ran under `set -euo pipefail` with no fallback. A momentary
   control-plane 502 or auth refresh killed the whole script after
   Helm uninstalls succeeded, leaving orphan CRDs, webhooks, and
   `Active` namespaces. Wrap each pipeline's trailing `done` in
   `|| echo "Warning: ..." >&2` so the failure is visible but the
   script continues. Matches the `|| true` pattern already used in the
   adjacent ORPHANED_CRD_GROUPS loop.

2. Pre-flight CRD discovery via Helm annotation (was NVIDIA#601).
   check_release_for_stuck_crds sourced its CRD list only from
   `helm get manifest`, which returns CRDs under the chart's templates/
   section but omits CRDs installed from the chart's crds/ directory
   (the Helm-recommended layout used by most operator charts). CRs
   backed by crds/-installed CRDs (e.g., NIMService) slipped past the
   finalizer pre-flight. Union both sources (manifest + annotated) and
   dedupe. Uses `awk 'NF'` instead of `grep -v '^$'` to drop empty
   lines without the grep-exits-1-on-no-match behavior, which would
   abort the script under pipefail when a release has no CRDs.

3. Post-flight Helm-annotation CRD check (was NVIDIA#600).
   Helm-managed CRDs that survive the per-release deletion loop went
   unnoticed until the next deploy.sh run rejected them. Add a
   verification pass at the end of undeploy.sh that re-enumerates CRDs
   filtered by meta.helm.sh/release-name + release-namespace
   annotations and surfaces any leftovers as a post-flight warning.
   CRDs already being deleted (non-null .metadata.deletionTimestamp)
   are excluded: the earlier `kubectl delete crd ... --wait=false`
   returns immediately, so a slow finalizer can leave the CRD listed
   though cleanup is normal. Only truly stuck (not-yet-terminating)
   CRDs are reported.

Unit test TestUndeployScript_TransientFailureWarnsAndContinues
generates a bundle, stubs kubectl to exit non-zero, sources each of
the three helpers, and asserts the helper returns 0 and emits the
expected Warning: on stderr.

Fixes: N/A
Related: NVIDIA#477 (introduced the per-component CRD cleanup pipeline),

Signed-off-by: Yuan Chen <[email protected]>

Combines three related hardening changes to the generated undeploy.sh
(previously split across NVIDIA#599, NVIDIA#601, NVIDIA#600). They share a file and a
theme -- make undeploy recover from partial or transient failure --
so bundling them avoids three sequential PR rebases over the same
template.

1. Transient-failure resilience (was NVIDIA#599).
   Three `kubectl | jq | while` pipelines in the post-uninstall cleanup
   path ran under `set -euo pipefail` with no fallback. A momentary
   control-plane 502 or auth refresh killed the whole script after
   Helm uninstalls succeeded, leaving orphan CRDs, webhooks, and
   `Active` namespaces. Wrap each pipeline's trailing `done` in
   `|| echo "Warning: ..." >&2` so the failure is visible but the
   script continues. Matches the `|| true` pattern already used in the
   adjacent ORPHANED_CRD_GROUPS loop.

2. Pre-flight CRD discovery via Helm annotation (was NVIDIA#601).
   check_release_for_stuck_crds sourced its CRD list only from
   `helm get manifest`, which returns CRDs under the chart's templates/
   section but omits CRDs installed from the chart's crds/ directory
   (the Helm-recommended layout used by most operator charts). CRs
   backed by crds/-installed CRDs (e.g., NIMService) slipped past the
   finalizer pre-flight. Union both sources (manifest + annotated) and
   dedupe. Uses `awk 'NF'` instead of `grep -v '^$'` to drop empty
   lines without the grep-exits-1-on-no-match behavior, which would
   abort the script under pipefail when a release has no CRDs.

3. Post-flight Helm-annotation CRD check (was NVIDIA#600).
   Helm-managed CRDs that survive the per-release deletion loop went
   unnoticed until the next deploy.sh run rejected them. Add a
   verification pass at the end of undeploy.sh that re-enumerates CRDs
   filtered by meta.helm.sh/release-name + release-namespace
   annotations and surfaces any leftovers as a post-flight warning.
   CRDs already being deleted (non-null .metadata.deletionTimestamp)
   are excluded: the earlier `kubectl delete crd ... --wait=false`
   returns immediately, so a slow finalizer can leave the CRD listed
   though cleanup is normal. Only truly stuck (not-yet-terminating)
   CRDs are reported.

Unit test TestUndeployScript_TransientFailureWarnsAndContinues
generates a bundle, stubs kubectl to exit non-zero, sources each of
the three helpers, and asserts the helper returns 0 and emits the
expected Warning: on stderr.

Fixes: N/A
Related: NVIDIA#477 (introduced the per-component CRD cleanup pipeline),

Signed-off-by: Yuan Chen <[email protected]>

Combines three related hardening changes to the generated undeploy.sh
(previously split across #599, #601, #600). They share a file and a
theme -- make undeploy recover from partial or transient failure --
so bundling them avoids three sequential PR rebases over the same
template.

1. Transient-failure resilience (was #599).
   Three `kubectl | jq | while` pipelines in the post-uninstall cleanup
   path ran under `set -euo pipefail` with no fallback. A momentary
   control-plane 502 or auth refresh killed the whole script after
   Helm uninstalls succeeded, leaving orphan CRDs, webhooks, and
   `Active` namespaces. Wrap each pipeline's trailing `done` in
   `|| echo "Warning: ..." >&2` so the failure is visible but the
   script continues. Matches the `|| true` pattern already used in the
   adjacent ORPHANED_CRD_GROUPS loop.

2. Pre-flight CRD discovery via Helm annotation (was #601).
   check_release_for_stuck_crds sourced its CRD list only from
   `helm get manifest`, which returns CRDs under the chart's templates/
   section but omits CRDs installed from the chart's crds/ directory
   (the Helm-recommended layout used by most operator charts). CRs
   backed by crds/-installed CRDs (e.g., NIMService) slipped past the
   finalizer pre-flight. Union both sources (manifest + annotated) and
   dedupe. Uses `awk 'NF'` instead of `grep -v '^$'` to drop empty
   lines without the grep-exits-1-on-no-match behavior, which would
   abort the script under pipefail when a release has no CRDs.

3. Post-flight Helm-annotation CRD check (was #600).
   Helm-managed CRDs that survive the per-release deletion loop went
   unnoticed until the next deploy.sh run rejected them. Add a
   verification pass at the end of undeploy.sh that re-enumerates CRDs
   filtered by meta.helm.sh/release-name + release-namespace
   annotations and surfaces any leftovers as a post-flight warning.
   CRDs already being deleted (non-null .metadata.deletionTimestamp)
   are excluded: the earlier `kubectl delete crd ... --wait=false`
   returns immediately, so a slow finalizer can leave the CRD listed
   though cleanup is normal. Only truly stuck (not-yet-terminating)
   CRDs are reported.

Unit test TestUndeployScript_TransientFailureWarnsAndContinues
generates a bundle, stubs kubectl to exit non-zero, sources each of
the three helpers, and asserts the helper returns 0 and emits the
expected Warning: on stderr.

Fixes: N/A
Related: #477 (introduced the per-component CRD cleanup pipeline),

Signed-off-by: Yuan Chen <[email protected]>

Combines three related hardening changes to the generated undeploy.sh
(previously split across NVIDIA#599, NVIDIA#601, NVIDIA#600). They share a file and a
theme -- make undeploy recover from partial or transient failure --
so bundling them avoids three sequential PR rebases over the same
template.

1. Transient-failure resilience (was NVIDIA#599).
   Three `kubectl | jq | while` pipelines in the post-uninstall cleanup
   path ran under `set -euo pipefail` with no fallback. A momentary
   control-plane 502 or auth refresh killed the whole script after
   Helm uninstalls succeeded, leaving orphan CRDs, webhooks, and
   `Active` namespaces. Wrap each pipeline's trailing `done` in
   `|| echo "Warning: ..." >&2` so the failure is visible but the
   script continues. Matches the `|| true` pattern already used in the
   adjacent ORPHANED_CRD_GROUPS loop.

2. Pre-flight CRD discovery via Helm annotation (was NVIDIA#601).
   check_release_for_stuck_crds sourced its CRD list only from
   `helm get manifest`, which returns CRDs under the chart's templates/
   section but omits CRDs installed from the chart's crds/ directory
   (the Helm-recommended layout used by most operator charts). CRs
   backed by crds/-installed CRDs (e.g., NIMService) slipped past the
   finalizer pre-flight. Union both sources (manifest + annotated) and
   dedupe. Uses `awk 'NF'` instead of `grep -v '^$'` to drop empty
   lines without the grep-exits-1-on-no-match behavior, which would
   abort the script under pipefail when a release has no CRDs.

3. Post-flight Helm-annotation CRD check (was NVIDIA#600).
   Helm-managed CRDs that survive the per-release deletion loop went
   unnoticed until the next deploy.sh run rejected them. Add a
   verification pass at the end of undeploy.sh that re-enumerates CRDs
   filtered by meta.helm.sh/release-name + release-namespace
   annotations and surfaces any leftovers as a post-flight warning.
   CRDs already being deleted (non-null .metadata.deletionTimestamp)
   are excluded: the earlier `kubectl delete crd ... --wait=false`
   returns immediately, so a slow finalizer can leave the CRD listed
   though cleanup is normal. Only truly stuck (not-yet-terminating)
   CRDs are reported.

Unit test TestUndeployScript_TransientFailureWarnsAndContinues
generates a bundle, stubs kubectl to exit non-zero, sources each of
the three helpers, and asserts the helper returns 0 and emits the
expected Warning: on stderr.

Fixes: N/A
Related: NVIDIA#477 (introduced the per-component CRD cleanup pipeline),

Signed-off-by: Yuan Chen <[email protected]>