Summary

Three conformance validators probe a metrics endpoint exactly once and fail on the first attempt if it isn't ready yet. Add a short, bounded, fail-closed retry/poll to each so a direct aicr validate --phase conformance run against a freshly-deployed (not-yet-converged) cluster self-heals instead of spuriously failing.

Background

Surfaced during the uat-aws validation RCA. The UAT conformance failures (accelerator-metrics, ai-service-metrics, pod-autoscaling) were a timing race: conformance ran before the GPU stack / prometheus-adapter had converged. That is being fixed in the UAT harness by a post-helmfile apply readiness gate plus failFast (the deployment phase gates conformance), and re-running conformance against the converged cluster passes cleanly.

However, the underlying single-shot behavior in the validators themselves remains. A user running aicr validate --phase conformance directly against a still-converging cluster — without the UAT readiness gate, and without the deployment phase to front-run it — hits the same fast-fails. Note ai-service-metrics step 1 already polls (2m via AIServiceMetricsWaitTimeout); its siblings do not, so this is bringing them up to parity.

Call sites

• validators/conformance/pod_autoscaling_check.go step 1 — custom.metrics.k8s.io discovery GET (single-shot)
• validators/conformance/pod_autoscaling_check.go step 3 — external.metrics.k8s.io/.../dcgm_gpu_power_usage data GET (single-shot)
• validators/conformance/accelerator_metrics_check.go — DCGM exporter :9400/metrics GET (single-shot, no retry)

Proposed change

Wrap each probe in a bounded wait.PollUntilContextCancel (new pkg/defaults constant, e.g. MetricsAPIDiscoveryTimeout, reusing HPAPollInterval), failing closed after the budget — mirroring the existing ai-service-metrics step-1 retry. A genuinely-absent or broken component still fails after the window; only the convergence window is absorbed.

Hard constraint: do not convert any of these to Skip on missing-API/endpoint (that would be fail-open and mask a real broken adapter / DCGM exporter on clusters that do ship them).

Tests / coverage gate

pod_autoscaling_check.go and accelerator_metrics_check.go have no dedicated unit tests today, so new coverage is required to clear the project coverage gate and to exercise both transient-fail-then-ready and permanent-fail-closed paths. accelerator_metrics_check.go is testable via httptest (it takes a URL); the pod-autoscaling discovery uses the discovery REST client and needs a fake.

Acceptance criteria

• The three probes poll up to a bounded budget and pass once the endpoint becomes ready.
• Each still fails closed (non-Skip) when the endpoint never becomes ready / returns no data.
• Unit tests cover transient-then-ready and permanent-failure paths for all three.
• No regression in the UAT path (the readiness gate already covers it).

Priority / notes

Defense-in-depth only — not required to fix the UAT failures (the readiness gate + failFast already resolve those). Lower priority.