Summary

Add CNCF AI Conformance #8a (cluster_autoscaling) validation to both H100 GPU CI workflows
(inference and training). Validates the full metrics-driven autoscaling chain end-to-end:

DCGM metrics → Prometheus → prometheus-adapter (external metric) → HPA scales Deployment
  → pending GPU pods → Karpenter provisions KWOK nodes → pods schedule → consolidation

Also adds external metrics rules and workload-attributed custom metrics to prometheus-adapter,
enabling HPA-based GPU autoscaling from any namespace.

Fixes custom metrics validation to handle DCGM exporter pod-mapping (which relabels metrics
with the GPU workload's namespace when a GPU is in use) and adds retry logic for
prometheus-adapter metric discovery timing.

Adds kai-scheduler dynamo Queue CR creation before DynamoGraphDeployment in the inference
workflow (grove-operator sets kai.scheduler/queue=dynamo on pods, but the kai-scheduler
chart only creates default-parent-queue and default-queue).

Changes

New files:

• kwok/scripts/install-karpenter-kwok.sh — Builds Karpenter KWOK provider from source via ko, side-loads into kind, deploys via Helm with GPU instance types
• kwok/scripts/validate-cluster-autoscaling.sh — End-to-end cluster autoscaling validation script (external metrics → HPA → Karpenter → KWOK nodes → consolidation)
• kwok/manifests/karpenter/instance-types.json — GPU instance types for Karpenter KWOK provider (p5.48xlarge 8×GPU, g5.xlarge 1×GPU, g5.2xlarge 1×GPU)
• kwok/manifests/karpenter/nodepool.yaml — NodePool with GPU taint + KWOKNodeClass
• kwok/manifests/karpenter/hpa-gpu-scale-test.yaml — Deployment + HPA using external dcgm_gpu_power_usage metric

Modified files:

• recipes/components/prometheus-adapter/values.yaml — Add external metrics rules (dcgm_gpu_utilization, dcgm_gpu_memory_used, dcgm_gpu_power_usage) and workload-attributed custom metrics (workload_gpu_utilization, workload_gpu_memory_used)
• .github/workflows/gpu-h100-inference-test.yaml — Add cluster autoscaling step, Dynamo kai-scheduler queue, custom metrics namespace/retry fix, trigger paths
• .github/workflows/gpu-h100-training-test.yaml — Add cluster autoscaling step, trigger paths
• .settings.yaml — Add karpenter: v1.8.0 to testing_tools

How it works

1. Install: Karpenter KWOK provider built from source (ko build sigs.k8s.io/karpenter/kwok) and deployed into kind cluster
2. Configure: NodePool + KWOKNodeClass created with GPU taints; GPU instance types loaded via ConfigMap
3. Verify metrics: External metric dcgm_gpu_power_usage confirmed available via external.metrics.k8s.io API
4. Scale up: HPA reads real GPU power metric, scales Deployment beyond node capacity → overflow pods trigger Karpenter
5. Provision: Karpenter provisions simulated KWOK nodes with nvidia.com/gpu capacity
6. Schedule: All GPU pods schedule onto KWOK nodes
7. Scale down: After cleanup, Karpenter consolidates KWOK nodes back to zero

CI Results

Both H100 workflows pass all steps on commit 00cc5512:

Inference (24 steps): Dynamo deploy + inference, accelerator metrics, custom metrics (pod autoscaling), cluster autoscaling (Karpenter+KWOK), DRA GPU test, secure accelerator access, conformance evidence (54/54 resources)

Training (19 steps): Gang scheduling (2× H100 NVL), cluster autoscaling (Karpenter+KWOK), conformance evidence (39/39 resources)

Test plan

• H100 inference workflow: cluster autoscaling step passes
• H100 training workflow: cluster autoscaling step passes
• prometheus-adapter serves external metrics (dcgm_gpu_power_usage)
• HPA reads external metric and computes desiredReplicas > 1
• Karpenter provisions KWOK nodes with nvidia.com/gpu capacity
• All GPU pods schedule onto KWOK nodes
• Scale-down consolidation removes KWOK nodes
• Custom metrics validation handles DCGM pod-mapping namespaces
• Dynamo deployment works with kai-scheduler queue
• All other existing steps continue to pass (no regressions)