Summary
The pod-autoscaling conformance validator (CNCF #8b) hard-fails on DRA-based GPU clusters with [NOT_FOUND] no GPU custom metrics available (DCGM → Prometheus → adapter pipeline broken), even when the GPU-metrics pipeline and HPA path are healthy.
Root cause
The check's step 2 requires pod-scoped GPU custom metrics (custom.metrics.k8s.io/.../pods/*/gpu_utilization). prometheus-adapter's pod-association rules map those via the exported_pod/exported_namespace labels, which only appear when dcgm-exporter attributes a GPU to its consuming pod via the kubelet pod-resources API. That API surfaces device-plugin (nvidia.com/gpu) allocations; DRA-claimed GPUs (nvidia-dra-driver-gpu, K8s 1.34+) are not attributed, so the per-pod series carry no exported_pod label and the adapter returns empty.
External metrics (external.metrics.k8s.io, e.g. dcgm_gpu_power_usage) work fine because they're cluster-wide GPU aggregates needing no per-pod label — which is the asymmetry observed.
Verified on aicr3 (EKS H100, K8s 1.35, nvidia-dra-driver-gpu, dcgm-exporter 4.5.3): count(DCGM_FI_DEV_GPU_UTIL{exported_pod\!=""}) = [] while external metrics returned data and the HPA behavioral test was healthy.
This is not an AICR regression (the adapter rules date to #168, unchanged) — it's a dcgm-exporter DRA pod-attribution gap surfaced by a brittle check precondition.
Fix
Demote step 2 (pod-scoped GPU custom metrics) to best-effort: warn + record an evidence artifact and continue, rather than failing. The autoscaling capability is proven authoritatively by step 3 (external-metrics API) + step 4 (HPA behavioral scale-up/down test driven by dcgm_gpu_power_usage), which remain hard, fail-closed gates and need no per-pod attribution.
Validation
Re-ran conformance on aicr3 with the fix: 11/11 passed (pod-autoscaling now PASS via the external-metric HPA path); previously 10/11.
Out of scope (separate follow-up)
Restoring pod-scoped GPU custom metrics on DRA clusters (dcgm-exporter DRA-aware pod attribution) — useful for dashboards/per-pod HPAs, but not required for the conformance capability.
Summary
The
pod-autoscalingconformance validator (CNCF #8b) hard-fails on DRA-based GPU clusters with[NOT_FOUND] no GPU custom metrics available (DCGM → Prometheus → adapter pipeline broken), even when the GPU-metrics pipeline and HPA path are healthy.Root cause
The check's step 2 requires pod-scoped GPU custom metrics (
custom.metrics.k8s.io/.../pods/*/gpu_utilization). prometheus-adapter's pod-association rules map those via theexported_pod/exported_namespacelabels, which only appear whendcgm-exporterattributes a GPU to its consuming pod via the kubelet pod-resources API. That API surfaces device-plugin (nvidia.com/gpu) allocations; DRA-claimed GPUs (nvidia-dra-driver-gpu, K8s 1.34+) are not attributed, so the per-pod series carry noexported_podlabel and the adapter returns empty.External metrics (
external.metrics.k8s.io, e.g.dcgm_gpu_power_usage) work fine because they're cluster-wide GPU aggregates needing no per-pod label — which is the asymmetry observed.Verified on aicr3 (EKS H100, K8s 1.35,
nvidia-dra-driver-gpu, dcgm-exporter 4.5.3):count(DCGM_FI_DEV_GPU_UTIL{exported_pod\!=""}) = []while external metrics returned data and the HPA behavioral test was healthy.This is not an AICR regression (the adapter rules date to #168, unchanged) — it's a dcgm-exporter DRA pod-attribution gap surfaced by a brittle check precondition.
Fix
Demote step 2 (pod-scoped GPU custom metrics) to best-effort: warn + record an evidence artifact and continue, rather than failing. The autoscaling capability is proven authoritatively by step 3 (external-metrics API) + step 4 (HPA behavioral scale-up/down test driven by
dcgm_gpu_power_usage), which remain hard, fail-closed gates and need no per-pod attribution.Validation
Re-ran conformance on aicr3 with the fix: 11/11 passed (
pod-autoscalingnow PASS via the external-metric HPA path); previously 10/11.Out of scope (separate follow-up)
Restoring pod-scoped GPU custom metrics on DRA clusters (dcgm-exporter DRA-aware pod attribution) — useful for dashboards/per-pod HPAs, but not required for the conformance capability.