Summary

kwok/scripts/validate-scheduling.sh::verify_pods() takes five sequential kubectl get pods snapshots and treats them as a coherent view of cluster state. They are not — controllers reconcile between calls, producing impossible Pod-status math and a misleading FailedScheduling diagnostic that has misdirected multiple recent investigations.

This is a CI harness race (TOCTOU), not a Go race and not a recipe regression. It explains the recent uptick in KWOK Tier-1 flakes across unrelated branches (#1053, #1086, #1046, and others).

Symptom (signature)

The job log shows arithmetic that cannot come from any single Kubernetes state:

Pod status: 20 total, 20 running, 0 pending (1 unscheduled), 0 failed
ERROR: Scheduling validation FAILED
ERROR: 1 pods could not be scheduled to nodes
ERROR: Unscheduled pods:
       (kubectl returns "No resources found")
ERROR: Events for unscheduled pods:
       kube-system  coredns-…  FailedScheduling  untolerated taint(s)
       kube-system  coredns-…  FailedScheduling  untolerated taint(s)
       local-path-storage  local-path-provisioner-…  FailedScheduling  untolerated taint(s)

unscheduled is a subset of pending, so pending=0 and unscheduled=1 are mutually exclusive in a single state snapshot. The diagnostic kubectl get pods --phase=Pending -o wide then returns "No resources found" because the bundle pod that was momentarily Pending has already been bound by the scheduler. The event dump that follows is unfiltered, so kube-system FailedScheduling noise (from the nfd-excluded=true:NoSchedule taint applied to the kind control-plane node) surfaces as if it were the cause — but those pods are explicitly excluded from the actual count.

Evidence

Two independent samples on different branches:

• Job 78343519974 (PR feat(recipes): add concrete GKE B200 service-bound overlays #1053, recipe aks-inference, deployer argocd-helm-oci, run 26589273745 attempt 1):
  20 total, 20 running, 0 pending (1 unscheduled), 0 failed — then "No resources found" on the diagnostic. Same SHA passed on rerun.
• Job 78341110233 (recipe + deployer differ): 2 total, 2 running, 1 pending (3 unscheduled) — a fresh nvidia-dra-driver pending pod appeared between snapshots. Same race class.

Issue #1061 fixed a sibling race (chainsaw sync gate firing too early — PR #1062). That fix tightened the gate but does not close the in-script sampling race in verify_pods(), which is still latent.

Root cause

kwok/scripts/validate-scheduling.sh:1283-1296 issues 5 separate kubectl get pods calls, each ~hundreds of ms apart due to -o json | jq overhead:

total_pods=$(kubectl get pods … | wc -l)                              # T0
pending_pods=$(kubectl get pods --phase=Pending … | wc -l)            # T1
failed_pods=$(kubectl get pods --phase=Failed … | wc -l)              # T2
running_pods=$(kubectl get pods --phase=Running … | wc -l)            # T3
unscheduled_pods=$(kubectl get pods --phase=Pending -o json | jq …)   # T4

During those several seconds, operators (gpu-operator, cert-manager, prometheus, dynamo, slurm, dra-driver, …) continue reconciling and creating new pods. A new Pending-unscheduled pod that appears between T1 and T4 makes pending=0, unscheduled=1 possible. By the time the diagnostic at line 1326-1327 re-queries, the scheduler has bound it — hence "No resources found".

The two earlier (incorrect) hypotheses that this same signature has provoked:

• "It's the nfd-excluded taint" — wrong: kube-system and local-path-storage are explicitly excluded from the unscheduled count at line 1281; only the event dump at line 1330 is unfiltered, which is why the noise appears in the log but cannot be the trigger.
• "It's the Argo CD sync gate firing too early" (KWOK argocd-* sync gate races ahead of child Application materialization (regression from #1050) #1061) — that was a real race that fix(kwok): require Succeeded operationState in argocd sync gate #1062 fixed, but it's distinct from this one. Tightening the sync gate just changed the timing without removing the in-script race.

Why this surfaces more often recently:

• refactor(kwok): migrate argocd-* and flux-oci sync gate from bash to chainsaw #1050/fix(kwok): require Succeeded operationState in argocd sync gate #1062 shortened the time between deploy-complete and verify_pods running, leaving less natural settle time before the race window opens.
• The bundle now includes more components (slurm, dynamo, grove, nvsentinel, kgateway, kai-scheduler), each with its own operator and reconcile cadence, widening the create-pod window between snapshots.

Fix

All three changes in kwok/scripts/validate-scheduling.sh:1283-1336:

1. Single snapshot: replace the 5 separate kubectl get pods calls with one kubectl get pods --all-namespaces -o json snapshot and derive total/running/pending/failed/unscheduled from that single coherent view with jq.
2. Reuse the snapshot for diagnostics: the "Unscheduled pods:" listing at line 1326-1327 must be rendered from the same snapshot, not a fresh kubectl get. This eliminates the "No resources found" diagnostic that has misled reviewers.
3. Filter the FailedScheduling event dump at line 1330 to the same namespace exclusions used by the count (kube-system, kube-node-lease, kube-public, local-path-storage, kwok-system, argocd, flux-system, aicr-registry) or to the exact set of unscheduled-pod names. This kills the nfd-excluded red herring that has provoked at least two wrong diagnoses.

Optional belt-and-suspenders (deferrable): if unscheduled_pods > 0 after a single snapshot, sleep ~5s and re-snapshot once before declaring failure. Covers the legitimate-but-transient case where the scheduler genuinely hasn't bound yet.

Estimated size: ~20 lines net, single commit, doc-only / script change. No Go, no chart, no recipe.

Reproduction

Race; not deterministic. Best signal:

# Triggers ~30-40% per attempt across recent KWOK runs:
gh run rerun <recent-failing-kwok-run-id> --failed

Or search the KWOK Cluster Validation workflow history for runs failing with the Pod status: X total, X running, 0 pending (Y unscheduled) signature where Y > 0.

Acceptance criteria

• verify_pods() derives all pod counts from one kubectl get pods -o json snapshot
• "Unscheduled pods:" diagnostic uses the same snapshot — never returns "No resources found" when count > 0
• FailedScheduling event dump filters by the same namespace exclusions as the count
• Add a unit-style chainsaw test or shellcheck-compatible smoke check, if feasible — non-blocking
• Three consecutive clean KWOK runs on main after merge (sanity check that the dominant flake class is gone)

Related

• KWOK argocd-* sync gate races ahead of child Application materialization (regression from #1050) #1061 (closed) — sibling sync-gate race, fixed by fix(kwok): require Succeeded operationState in argocd sync gate #1062. Different mechanism, same symptom class.
• PR feat(recipes): add concrete GKE B200 service-bound overlays #1053 KWOK Tier-1 failure (run 26589273745 attempt 1) — passed on retry, evidence captured above.