Summary

Three CNCF AI Conformance evidence artifacts capture tests that pass but do not actually exercise the capability they claim to prove. The evidence is internally consistent and was generated from real cluster runs, but the test design is too weak to be conformance-grade. Each needs a stronger test scenario and a re-run to regenerate the snapshot.

These were surfaced during a documentation audit of docs/conformance/cncf/v1.35/nim-eks/evidence/. The .md files are generated snapshots (**Generated:** …), so the fix lives in the test manifests/scripts under pkg/evidence/scripts/, not in the docs.

Findings

1. Cluster autoscaling — fixed-size ASG cannot demonstrate scaling

docs/conformance/cncf/v1.35/nim-eks/evidence/cluster-autoscaling.md

The GPU Auto Scaling Group is captured with MinSize: 2, MaxSize: 2, DesiredCapacity: 2. With min = max = desired the group is pinned at 2 nodes — it physically cannot scale up (already at max) or down (already at min). The evidence proves an ASG exists, not that GPU-aware autoscaling works.

Fix: provision the GPU ASG with min < desired < max (e.g. min 0–1 / max 4). Submit a pending GPU workload that does not fit current capacity → observe a new node join (scale-up); delete it → observe a node removed (scale-down). Capture the node-count transitions.

2. Gang scheduling — ample capacity never triggers the all-or-nothing barrier

docs/conformance/cncf/v1.35/nim-eks/evidence/gang-scheduling.md

A PodGroup{minMember: 2} is launched while the cluster has ample free GPUs, so both pods schedule regardless of whether gang semantics exist — an ordinary scheduler would behave identically. The all-or-nothing barrier is never exercised.

Fix: add a negative case. With fewer free GPUs than the group requires, confirm neither pod schedules (both Pending, not one Running/one Pending); then free capacity and confirm both transition to Running together. Assert both pods share the same PodGroup. The partial-capacity Pending state is the actual proof.

3. Robust operator — schema rejection mistaken for webhook validation

docs/conformance/cncf/v1.35/nim-eks/evidence/robust-operator.md

The "webhook rejection" test submits spec: {} and is rejected with spec.authSecret: Required value / spec.image: Required value. That is CRD OpenAPI schema validation enforced by the apiserver — it would reject spec: {} even with no validating webhook installed. It does not prove the validating admission webhook is operational.

Fix: submit a CR that is schema-valid (all required fields present) but violates a rule only the validating webhook enforces (a business-logic constraint), and show it is rejected with a webhook-specific message.

How to fix

1. Strengthen the test artifacts under pkg/evidence/scripts/ (gang-scheduling manifest, the autoscaling scenario, the operator webhook test case).
2. Re-run pkg/evidence/scripts/collect-evidence.sh against a real EKS/H100 conformance cluster to regenerate the three evidence snapshots.

Context

Identified alongside the documentation correctness sweep in #1528 (follow-up note: #1528 (comment)). The docs PR intentionally left these untouched because editing captured evidence snapshots would falsify recorded results.