Summary

The expected-resources deployment validator stalls for the full Job
deadline (~8m) and reports an opaque status=other (instead of a clean
failed) whenever a recipe-declared component is absent or in a different
namespace than the recipe expects. This was observed validating a live
GB300 EKS cluster managed by an external platform (components deployed by the
platform's GitOps into different namespaces than the recipe assumes, plus one
component intentionally removed).

Symptoms

• expected-resources runs to ~8m24s and ends as status=other
  (phase: passed=3 failed=0), so the operator gets no actionable verdict.
• Per-component failures surface a misleading error:
  client rate limiter Wait returned an error: context deadline exceeded,
  which looks like a client bug but is really the per-check deadline firing on
  an in-flight GET.
• Healthy components are also delayed to the deadline (the check returns nothing
  useful until it is killed).

Root cause

1. Absent resources retry until the full readiness budget. A resource that
   does not exist (ErrCodeNotFound) is retried every AssertRetryInterval
   until ChainsawAssertTimeout (6m). With ChainsawMaxParallel = 4 worker
   slots, a few permanently-absent components occupy the pool for the full 6m
   and starve the healthy components queued behind them. The check then
   brushes the Job's activeDeadlineSeconds, the pod is killed, and a
   non-0/1/2 exit code maps to other.
2. The real reason is lost at the deadline. On context cancellation the
   retry loops return an ErrCodeInternal "context canceled" wrap (or the
   in-flight client-go rate-limiter error), masking the substantive assertion
   reason (e.g. "DaemonSet not found").
3. (Minor) client QPS is the client-go default (5 QPS / 10 burst) — too low
   for a one-shot enumeration across 18+ components, which throttles even the
   all-healthy sweep.

Proposed fixes

1. Bound only the absent (NotFound) case with a short
   AbsentResourceGracePeriod (~30s): a resource that does not exist will not
   appear by waiting out the readiness budget. A resource that exists but is
   not ready returns a shape-mismatch error (ErrCodeInternal), and a
   transient API failure returns ErrCodeUnavailable — both keep the full
   6m, so slow-but-healthy rollouts are not failed prematurely. (Verified:
   shape mismatch and transient errors are distinct codes from NotFound, so the
   gate is precise.)
2. Preserve the substantive assertion error at the deadline so the verdict
   is a clean failed with the real reason ("... not found"), not an opaque
   other driven by the context/rate-limiter artifact.
3. Raise the validator REST client limits (e.g. QPS 50 / Burst 100) so the
   healthy enumeration sweep is not throttled. apiserver APF still protects the
   server.

Evidence (prototyped + verified on a live GB300 EKS cluster)

Healthy components still pass; not-ready/slow-rollout components still get the
full 6m (no false-failure regression).

Affected code

• validators/chainsaw/inprocess.go — runAssertWithRetry / runErrorWithRetry
• validators/chainsaw/runner.go — assertRawResources
• pkg/defaults/timeouts.go — ChainsawAssertTimeout, ChainsawMaxParallel,
  AssertRetryInterval (+ proposed AbsentResourceGracePeriod)
• pkg/k8s/client/client.go / validators/context.go — REST client QPS/Burst

A working prototype of all three fixes (with unit tests) has been validated
end-to-end on a live GB300 EKS cluster; a PR can follow.