Bug Description

The ai-service-metrics conformance validator non-deterministically fails on clusters that segregate worker and system pods into different Security Groups (or otherwise gate cross-subnet pod-to-pod traffic on port 9090). Whether the validator passes is decided by where the kube-scheduler lands its orchestrator pod — which on a clean cluster is a tie-break, and on a re-used cluster is dominated by image-locality. The root cause is that the validator's Job spec sets tolerations: [{operator: Exists}] with no nodeAffinity or pod-affinity, so it has no preference for co-locating with the Prometheus pod it queries.

This is reproducible today on DGXC EKS GB200 clusters where customer-workload and system-workload ENIs use separate SGs with asymmetric ingress rules.

Impact

Medium (workaround exists — fix the cluster SG, or accept that a re-run on a fresh cluster may pass by luck)

Component

Validator

Regression?

No — this is the steady-state behavior across all aicr versions where ai-service-metrics has shipped. It only surfaces when the kube-scheduler's tie-break + image-locality history leaves the validator pod on a node whose ENI lives in a SG that can't reach Prometheus.

Steps to Reproduce

1. Provision a Kubernetes cluster with two ENI configs / two security groups, where the SG attached to the Prometheus pod's node does not accept ingress from the SG attached to other worker nodes. (DGXC EKS with separate customer/system ENI subnets is one example.)
2. Deploy a CUJ2 inference bundle (e.g. gb200-eks-ubuntu-inference-dynamo).
3. Run aicr validate --recipe recipe.yaml --phase conformance repeatedly.

On a fresh cluster, the first run may pass (if the validator pod lands on a system-ENI node by tie-break). Once the validator image is cached on a customer-ENI node, every subsequent run will land there via image-locality scoring and ai-service-metrics will fail.

Expected Behavior

The validator should deterministically succeed against the same cluster state, regardless of where the kube-scheduler decides to place the orchestrator pod. Either:

• The orchestrator pod is co-scheduled with the Prometheus pod (nodeAffinity / podAffinity), or
• The check is networking-tolerant by routing through the Prometheus Service (it already does — but it relies on the underlying pod-to-pod path being reachable, which in this cluster topology requires same-SG placement).

Actual Behavior

[SERVICE_UNAVAILABLE] Prometheus unreachable at http://kube-prometheus-prometheus.monitoring.svc:9090 — verify network connectivity
[SERVICE_UNAVAILABLE] failed to reach http://kube-prometheus-prometheus.monitoring.svc:9090/api/v1/query?query=DCGM_FI_DEV_GPU_UTIL:
Get "...": dial tcp 172.20.41.184:9090: i/o timeout

The validator pod lands on a customer-ENI node (e.g. nodeGroup=customer-cpu), gets a pod IP in the customer pod CIDR (100.65.0.0/17), and the SYN to Prometheus' pod IP (100.64.8.71, in system pod CIDR 100.64.8.0/22) is dropped by the system SG's ingress filter. The check times out at 5s and is marked failed.

Environment

• AICR version: 0.13.0-rc4 (release binary, also reproduced against built-from-source main)
• Install method: release binary
• Platform: eks (DGXC EKS, GB200 capacity block in us-east-1)
• Kubernetes version: v1.34.7-eks-40737a8
• OS: ubuntu 24.04 (kernel 6.14.0-1018-aws)
• GPU type: gb200 (p6e-gb200.36xlarge, 2 nodes)
• Workload intent: inference (dynamo platform)

Command / Request Used

aicr validate \
    --recipe recipe.yaml \
    --snapshot snapshot.yaml \
    --phase conformance \
    --output report.json

(The same failure occurs with --phase all, with or without --emit-attestation.)

Root Cause Analysis

Three independent facts compose to produce the failure:

1. Validator orchestrator tolerates every taint. The Job spec deployed by the validator includes tolerations: [{operator: Exists}] with an empty key. This is correct for the broader goal (run on whichever node has GPU access if needed), but it means there is no scheduling constraint that ties the pod to any particular zone, ENI config, or SG.

2. No affinity for the dependency. ai-service-metrics queries Prometheus over the cluster Service. The dial resolves the service ClusterIP, which then DNATs to the Prometheus pod IP. The orchestrator has no podAffinity toward the Prometheus pod or nodeAffinity toward the labels that identify a node hosting Prometheus.

3. Image locality is sticky. Once the validator image is pulled to a node, the default scheduler scores that node higher on subsequent runs. So whatever node "wins" the very first scheduling decision keeps winning.

The interaction with cluster network topology decides the outcome:

• On clusters with a single flat SG (or with symmetric cross-SG rules on port 9090), this doesn't matter — the validator can reach Prometheus from any node.
• On clusters with asymmetric SG rules (system SG doesn't accept ingress from customer SG), the validator fails iff it lands on a customer-SG node.

We confirmed the SG asymmetry on the affected cluster:

We also confirmed that kubectl cordon does not redirect the validator pod, because cordon adds node.kubernetes.io/unschedulable:NoSchedule, which the operator: Exists toleration covers.

Potential Solutions in AICR

In rough order of effort / safety:

1. Add podAffinity toward the Prometheus pod on the ai-service-metrics Job spec (preferred). Match on the label app.kubernetes.io/name=prometheus (or whatever the kube-prometheus-stack chart writes) with requiredDuringSchedulingIgnoredDuringExecution. This guarantees the validator runs on the same node as Prometheus, so the dial is loopback / same-SG and the SG topology becomes irrelevant. Right scope, no extra knobs, deterministic.

2. Add a soft nodeAffinity toward app.kubernetes.io/name=kube-prometheus-stack workload labels (or a configurable label key) as a fallback for clusters where Prometheus uses a StatefulSet with multiple replicas. Soft, preferred-not-required, so the pod still runs somewhere if Prometheus is gone.

3. Make orchestrator placement configurable. Today --node-selector and --toleration explicitly do not affect the orchestrator (only inner workloads). Add a separate flag pair (--orchestrator-node-selector, --orchestrator-toleration — or reuse the existing flags with a documented behavior change) so users can pin the validator to where it needs to be. Less safe / more failure modes than option 1 because it pushes the cluster-knowledge requirement onto the user.

4. Document the cluster-side prerequisite. Add an entry under docs/integrator/eks-dynamo-networking.md (or a new "multi-SG clusters" page) calling out that ai-service-metrics requires <customer SG> → <prometheus host SG> ingress on port 9090. Doc-only fix; doesn't change behavior; surfaces the constraint up front.

My preference is option 1, optionally with option 4 as a complement.

Additional Context

• Found while running CUJ2 against v0.13.0-rc4 on a GB200 EKS cluster. Full run report: see cuj2-findings-v0.13.0-rc4.md.
• The check failure surfaces in aicr evidence verify (informational exit 1, bundle valid; recorded validator results show failures), so users who push attestation bundles will see this in the rendered summary.