Summary

Rewrites verify_pods() in kwok/scripts/validate-scheduling.sh to take one coherent kubectl get pods -o json snapshot per attempt, polls on a bounded scheduler-settle window when pods are still unscheduled, and derives every count, the pod distribution, and the failure diagnostic from the same snapshot — eliminating both the TOCTOU race (impossible math) and the post-sync-gate scheduler-latency flake.

Motivation / Context

verify_pods() had two flake classes producing false failures on KWOK Tier-1 lanes:

1. TOCTOU race across five separate kubectl get pods calls. total / pending / failed / running / unscheduled were sampled sequentially over several seconds (the unscheduled query alone takes hundreds of ms due to -o json | jq). Between calls, operator controllers (gpu-operator, dynamo, dra-driver, …) kept creating pods, so the snapshots disagreed:

Pod status: 20 total, 20 running, 0 pending (1 unscheduled), 0 failed
ERROR: Unscheduled pods:
       (kubectl returned "No resources found")
ERROR: Events for unscheduled pods:
  kube-system  coredns-…  FailedScheduling  untolerated taint(s)

unscheduled is a subset of pending, so pending=0 + unscheduled=1 cannot come from one state. The unfiltered event dump then surfaced kube-system noise from the intentional nfd-excluded=true:NoSchedule taint, misleading multiple prior reviews.

2. Post-sync-gate scheduler latency. Even after switching to a single coherent snapshot, two lanes failed with internally-consistent math but a real Pending pod that simply hadn't been bound yet:

• eks-inference (argocd-oci): cert-manager/cert-manager-786cc54854-kngvm — Pod status: 2 total, 0 running, 2 pending (1 unscheduled), 0 failed, zero FailedScheduling events
• oke-training (argocd-oci): skyhook/skyhook-operator-controller-manager-596b99b857-8shfm — Pod status: 12 total, 11 running, 1 pending (1 unscheduled), 0 failed, zero FailedScheduling events

Zero FailedScheduling events means the scheduler hadn't tried to schedule yet — pure latency between Argo CD reporting Synced+Healthy and the scheduler catching up. A single instantaneous snapshot would fail-fast on this transient.

Issue #1090 has the full evidence trail (jobs 78343519974, 78341110233 for the TOCTOU race; jobs 78353433292, 78353436030 for the post-sync latency).

Fixes: #1090
Related: #1061 (closed, sibling sync-gate race fixed by #1062 — distinct from this one)

Type of Change

• Bug fix (non-breaking change that fixes an issue)

Component(s) Affected

• Other: KWOK CI harness (kwok/scripts/validate-scheduling.sh)

Implementation Notes

Six changes inside verify_pods():

1. Single snapshot per attempt. One kubectl get pods --all-namespaces -o json per loop iteration; all five counts (total / running / pending / failed / unscheduled) plus pod distribution derived from it via jq. Within one iteration the math is guaranteed coherent — impossible-math signatures cannot reappear.

2. Bounded scheduler-settle poll loop. When unscheduled_pods > 0, the loop re-snapshots up to KWOK_VERIFY_PODS_DEADLINE seconds (default 60) at KWOK_VERIFY_PODS_INTERVAL spacing (default 5). Early-exits as soon as a snapshot shows unscheduled == 0 (or any Failed > 0, which is terminal). Declares failure only if the deadline elapses with unscheduled still positive — uses the FINAL snapshot for diagnostics so the named pod is the one that was still unscheduled at the deadline.

3. Diagnostics from the same snapshot. "Unscheduled pods:" listing and pod distribution come from the final snapshot — never prints "No resources found" when the count is positive.

4. Filtered FailedScheduling event dump. Piped through grep -vE "^(${exclude_ns})\s" so kube-system / local-path-storage noise from the intentional nfd-excluded taint stops surfacing as if it were the cause.

5. Fail-closed on kubectl error. Pre-existing 2>/dev/null | wc -l silently produced total_pods=0 on apiserver outage / auth failure / bad context and flowed through the "bundle may be empty" PASS branch. Now captures kubectl's stderr into a mktemp file and surfaces it in log_error before return 1.

6. (Drive-by, same shape) Rendering the failed-pod diagnostic from the same snapshot for consistency.

Tunables

Per-iteration cost: 1 kubectl call + 5 jq calls (~0.5s total). With default deadline, max ~13 iterations. Loop exits early on success, so the additional cost is paid only when scheduler latency is real — for hung pods that would have failed anyway, this adds the deadline as a fixed cost (worth it for stability).

What this PR does NOT change

• The intentional nfd-excluded=true:NoSchedule taint on the kind control-plane (still applied; only the diagnostic stopped surfacing its noise).
• The Argo CD sync-gate behavior from fix(kwok): require Succeeded operationState in argocd sync gate #1062 (separate race, separately fixed).
• Recipe overlays or any Go code.
• Genuine scheduling failures (missing toleration, no matching nodeSelector, insufficient resources): those persist through the full deadline and are reported with the same diagnostic.

Testing

# Static checks
shellcheck -x kwok/scripts/validate-scheduling.sh   # 0 errors

# Local smoke test with synthetic snapshot
# Covered: excluded namespaces (kube-system, local-path-storage, argocd),
#          Job-owned helm hooks, Failed pods, ContainerCreating pods.

# Project gate
unset GITLAB_TOKEN
make qualify    # 0 issues lint, 22/22 chainsaw, no vulnerabilities, license OK

End-to-end verification: the prior CI run on this PR (commit d702b9e9) hit the post-sync-gate scheduler latency on two argocd-oci lanes — exactly the case the new poll loop addresses. The current revision (e4827765) is the response to that finding. The next KWOK run on this PR is the natural integration test; if any lane still fails, the failure will name a specific pod that remained unscheduled for the full 60s deadline, which is a real bug not a flake.

Coverage: script-only change (no Go), so the per-package coverage gate does not apply per CLAUDE.md.

Risk Assessment

• Low — Single-function rewrite in a CI-only harness script. Success-path log unchanged on the happy path; failure-path output strictly more useful. The poll loop adds bounded latency (default 60s max) only when pods are not yet scheduled — already-scheduled bundles see zero added latency. Easy to revert (one file, one commit).

Rollout notes: None. Takes effect on first run after merge. No flag, no migration. The two new env vars default to the values used during development; operators only need to set them if they want a shorter/longer settle window.

Checklist

• Tests pass locally (make test with -race) — covered by make qualify
• Linter passes (make lint) — 0 issues
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality — N/A, no Go path; smoke-tested locally and end-to-end on the prior PR commit
• I updated docs if user-facing behavior changed — N/A, CI internals only
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)