Summary

The KWOK argocd-* sync gate at tests/chainsaw/kwok/argocd-sync/chainsaw-test.yaml (introduced in #1050, merged at d31e1f5d) returns PASS within ~4–5 s — sometimes before Argo CD has finished its initial App-of-Apps sync and created the child Application resources. When the race is lost, the subsequent validate-scheduling.sh step runs against a half-deployed cluster and the percent-scheduled check trips on KIND-default kube-system pods.

Severity: intermittent flake, not deterministic break. Re-runs of the same SHA usually pass; the failure rate is low per-job but the 68-job workflow surfaces it at a meaningful rate.

Evidence

The cleanest race-condition signature: same commit, different outcome.

Pod-count snapshots at the scheduling-validation step (recipe eks-training, deployer argocd-helm-oci):

The 3 unscheduled pods on losing-the-race runs are the KIND-default coredns × 2 and local-path-provisioner × 1 — kwok/scripts/validate-scheduling.sh:1489 taints the kind-control-plane node with nfd-excluded=true:NoSchedule, and those kube-system pods don't carry that toleration. When the AICR bundle's pods haven't deployed yet, the only remaining pods in the cluster are those 3 stragglers, so the percent-scheduled check trips.

What this is NOT

• Not a deterministic break — same SHA can pass on retry.
• Not "every PR fails after refactor(kwok): migrate argocd-* and flux-oci sync gate from bash to chainsaw #1050." Several PRs have passed first try since the merge (e.g., renovate/busybox-1.37, feat/bcm-service-type first run, and the retry on this PR).
• The earlier feat/bcm-service-type 15:17→15:36 transition I cited compared different SHAs (59ab2ff4 → e324ef97); the pattern fits a race but isn't clean same-content evidence on that pair.

Root Cause

In tests/chainsaw/kwok/argocd-sync/chainsaw-test.yaml (step assert-all-applications-pass, line ~122):

1. The pass predicate doesn't require an Argo CD sync to have completed. Arms 1 and 2 of the JMESPath only check sync.status and health.status; only arms 3 and 4 require operationState.phase == 'Succeeded'. For the root App-of-Apps, the controller can briefly set sync.status=Synced (manifest matches cluster) before running the sync operation that creates the child Applications.

2. The selector has no minimum-count guard. assert says "every Application in argocd namespace must pass" — when only the root exists, "every" = 1, so the assertion passes once the root alone matches. Children that haven't been created yet aren't in the match set.

The old bash gate (wait_for_argocd_sync in validate-scheduling.sh, replaced by #1050) didn't have this race because its polling loop took long enough that by the time it re-listed Applications, the children had materialized and joined the pass-check. Replacing it with a fast chainsaw assertion removed the implicit wait without adding an explicit one.

Proposed Fix (minimum-invasive)

Add operationState.phase == 'Succeeded' to arms 1 and 2. All four arms then require Argo CD to have completed a sync operation, which for the App-of-Apps means children have been created and will be in the match set on subsequent polls.

# tests/chainsaw/kwok/argocd-sync/chainsaw-test.yaml ~line 122
(operationState.phase == 'Succeeded'
 && ((sync.status == 'Synced' && health.status == 'Healthy')
     || (sync.status == 'Synced' && health.status == 'Progressing')
     || (sync.status == 'OutOfSync' && health.status == 'Healthy')
     || (sync.status == 'Synced' && health.status == 'Degraded'))): true

Belt-and-suspenders option

Insert an intermediate step between assert-root-app-present and assert-all-applications-pass that waits for the root's status.resources list to be non-empty (or for at least N child Applications to exist). This makes the children-materialized invariant explicit instead of relying on operationState.phase propagation.

Impact / Why Fix It Even Though Re-runs Clear It

• A flake at the workflow level translates to roughly half the runs hitting at least one failing tier-1 job (68 jobs × low per-job failure rate). That's enough to burn CI minutes, slow reviewer iteration, and erode trust in red CI on legitimate breakage.
• The fix removes the race rather than masking it — it doesn't make the chainsaw assertion wait longer; it makes its pass predicate require a real terminal state.

Reproduction

Push any PR based on main ≥ d31e1f5d and watch any Tier 1: <recipe> (argocd-helm-oci) / (argocd-oci) job across multiple runs. Expect occasional failures with the signature "5 total pods, 3 unscheduled (coredns × 2, local-path-provisioner × 1)" that clear on retry of the same SHA.

Related

• Introduced by: refactor(kwok): migrate argocd-* and flux-oci sync gate from bash to chainsaw #1050 (refactor(kwok): migrate argocd-* and flux-oci sync gate from bash to chainsaw)
• Tracking: Refactor KWOK argocd-* sync gate from bash/jq to Chainsaw #962 (the migration this PR implemented)
• Observed on PR: feat(bundler): derive DRA chart-version annotation from resolved recipe (#973) #1033 (failed → retry passed on same SHA), feat/bcm-service-type (intermittent across SHAs)