You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The KWOK Argo CD sync gate tests/chainsaw/kwok/argocd-sync/chainsaw-test.yaml
(step assert-all-applications-pass) uses a name-less chainsaw assert, which
has exists-semantics — it passes the moment any one Application matches the
4-arm terminal-pass predicate, not when all do. This is the Argo CD half of #1288; PR #1290 fixed only the flux side (flux-sync, flux-git-sync) and left argocd-sync as a documented "Known follow-up" (see ADR-010 "Sync Gate:
All-Resources Semantics"). tests/chainsaw/kwok/argocd-git-sync/ inherits the
same step verbatim as a faithful sibling, so this covers argocd-oci, argocd-helm-oci, and argocd-git.
Surfaced while implementing the argocd-git lane (#963).
Impact
Blocking (cannot proceed)
Component
CLI (aicr)
Regression?
Unknown / first time using this feature
Steps to Reproduce
Expected Behavior
argocd-sync (and its sibling argocd-git-sync) fail when any Application
is outside the terminal-pass set at the deadline; zero Applications still fails.
The gate does not pass vacuously at apply-time (root must converge first).
The gate does not hang on KWOK-pinned Progressing apps whose sync op
never reaches Succeeded (e.g. kube-prometheus-stack).
Not flaky under selfHeal churn across repeated runs.
argocd-oci / argocd-helm-oci / argocd-git all stay green after the change
(budget a fidelity-iteration round per ADR-008; bump KWOK_ARGOCD_SYNC_TIMEOUT
to match KWOK_FLUX_SYNC_TIMEOUT=500s if the honest gate needs it).
Actual Behavior
Environment
AICR version (CLI aicr version, API image tag, or commit SHA):
GPU type (h100/h200/gb200/b200/a100/l40/rtx-pro-6000/other):
Workload intent (training/inference):
Command / Request Used
No response
Logs / Error Output
Additional Context
Why the naive fix is not enough (findings from a live attempt)
Porting the flux fix (existence assert + error:-polarity sweep) to argocd is not a drop-in. Two additional failure modes showed up on a live KWOK run of eks-training under argocd-git:
Apply-time race / vacuous pass. A name-less sweep can evaluate at the same
instant kubectl apply creates the root app-of-apps, when the only Application
present is nvidia-stack with empty status. The gate "passed" in 0.26s
with zero children synced. A staged fix (a namedassert that the root
reaches operationState.phase==Succeeded first, then the fleet sweep) closes
this.
operationState.phase==Succeeded hangs forever on health-gated apps. The KWOK argocd-* sync gate races ahead of child Application materialization (regression from #1050) #1061 predicate requires op==Succeeded. But on KWOK, kube-prometheus-stack
(and prometheus-adapter) are permanently Synced + Progressing: their Argo
sync operation stays op=Running with message: "waiting for healthy state of monitoring.coreos.com/Prometheus/kube-prometheus-prometheus".
The prometheus-operator can't actually run under KWOK (no kubelet) to make the Prometheus CR healthy, so op never reaches Succeeded and a strict gate
times out (observed: 10 min). Dropping op==Succeeded from the per-child sweep
(keeping it only on the named root step) fixes this, since sync.status==Synced
already implies a completed sync.
Prerequisites
Bug Description
The KWOK Argo CD sync gate
tests/chainsaw/kwok/argocd-sync/chainsaw-test.yaml(step
assert-all-applications-pass) uses a name-less chainsawassert, whichhas exists-semantics — it passes the moment any one Application matches the
4-arm terminal-pass predicate, not when all do. This is the Argo CD half of
#1288; PR #1290 fixed only the flux side (
flux-sync,flux-git-sync) and leftargocd-syncas a documented "Known follow-up" (see ADR-010 "Sync Gate:All-Resources Semantics").
tests/chainsaw/kwok/argocd-git-sync/inherits thesame step verbatim as a faithful sibling, so this covers
argocd-oci,argocd-helm-oci, andargocd-git.Surfaced while implementing the
argocd-gitlane (#963).Impact
Blocking (cannot proceed)
Component
CLI (aicr)
Regression?
Unknown / first time using this feature
Steps to Reproduce
Expected Behavior
argocd-sync(and its siblingargocd-git-sync) fail when any Applicationis outside the terminal-pass set at the deadline; zero Applications still fails.
Progressingapps whose syncopnever reaches
Succeeded(e.g.kube-prometheus-stack).argocd-oci/argocd-helm-oci/argocd-gitall stay green after the change(budget a fidelity-iteration round per ADR-008; bump
KWOK_ARGOCD_SYNC_TIMEOUTto match
KWOK_FLUX_SYNC_TIMEOUT=500s if the honest gate needs it).Actual Behavior
Environment
aicr version, API image tag, or commit SHA):Command / Request Used
No response
Logs / Error Output
Additional Context
Why the naive fix is not enough (findings from a live attempt)
Porting the flux fix (existence
assert+error:-polarity sweep) to argocd isnot a drop-in. Two additional failure modes showed up on a live KWOK run of
eks-trainingunderargocd-git:Apply-time race / vacuous pass. A name-less sweep can evaluate at the same
instant
kubectl applycreates the root app-of-apps, when the only Applicationpresent is
nvidia-stackwith emptystatus. The gate "passed" in 0.26swith zero children synced. A staged fix (a named
assertthat the rootreaches
operationState.phase==Succeededfirst, then the fleet sweep) closesthis.
operationState.phase==Succeededhangs forever on health-gated apps. TheKWOK argocd-* sync gate races ahead of child Application materialization (regression from #1050) #1061 predicate requires
op==Succeeded. But on KWOK,kube-prometheus-stack(and
prometheus-adapter) are permanentlySynced + Progressing: their Argosync operation stays
op=Runningwithmessage: "waiting for healthy state of monitoring.coreos.com/Prometheus/kube-prometheus-prometheus".The prometheus-operator can't actually run under KWOK (no kubelet) to make the
PrometheusCR healthy, soopnever reachesSucceededand a strict gatetimes out (observed: 10 min). Dropping
op==Succeededfrom the per-child sweep(keeping it only on the named root step) fixes this, since
sync.status==Syncedalready implies a completed sync.
References
ci(kwok): add Gitea-in-cluster for filesystem-bundle round-trip coverage #963 (argocd-git lane), Refactor KWOK argocd-* sync gate from bash/jq to Chainsaw #962 (chainsaw migration)
tests/chainsaw/kwok/{argocd-sync,argocd-git-sync}/chainsaw-test.yaml