Skip to content

[Bug]: argocd-sync gate uses exists-semantics, not all-Applications (argocd half of #1288) #1388

Description

@haarchri

Prerequisites

  • I searched existing issues and found no duplicates
  • I can reproduce this issue consistently
  • This is not a security vulnerability (use Security Advisories instead)

Bug Description

The KWOK Argo CD sync gate tests/chainsaw/kwok/argocd-sync/chainsaw-test.yaml
(step assert-all-applications-pass) uses a name-less chainsaw assert, which
has exists-semantics — it passes the moment any one Application matches the
4-arm terminal-pass predicate, not when all do. This is the Argo CD half of
#1288; PR #1290 fixed only the flux side (flux-sync, flux-git-sync) and left
argocd-sync as a documented "Known follow-up" (see ADR-010 "Sync Gate:
All-Resources Semantics"). tests/chainsaw/kwok/argocd-git-sync/ inherits the
same step verbatim as a faithful sibling, so this covers argocd-oci,
argocd-helm-oci, and argocd-git.

Surfaced while implementing the argocd-git lane (#963).

Impact

Blocking (cannot proceed)

Component

CLI (aicr)

Regression?

Unknown / first time using this feature

Steps to Reproduce

Expected Behavior

  • argocd-sync (and its sibling argocd-git-sync) fail when any Application
    is outside the terminal-pass set at the deadline; zero Applications still fails.
  • The gate does not pass vacuously at apply-time (root must converge first).
  • The gate does not hang on KWOK-pinned Progressing apps whose sync op
    never reaches Succeeded (e.g. kube-prometheus-stack).
  • Not flaky under selfHeal churn across repeated runs.
  • argocd-oci / argocd-helm-oci / argocd-git all stay green after the change
    (budget a fidelity-iteration round per ADR-008; bump KWOK_ARGOCD_SYNC_TIMEOUT
    to match KWOK_FLUX_SYNC_TIMEOUT=500s if the honest gate needs it).

Actual Behavior

Environment

  • AICR version (CLI aicr version, API image tag, or commit SHA):
  • Install method (release binary / build from source / container image):
  • Platform (eks/gke/aks/oke/kind/lke/bcm/other):
  • Kubernetes version:
  • OS (ubuntu/cos/other) + version:
  • Kernel version:
  • GPU type (h100/h200/gb200/b200/a100/l40/rtx-pro-6000/other):
  • Workload intent (training/inference):

Command / Request Used

No response

Logs / Error Output

Additional Context

Why the naive fix is not enough (findings from a live attempt)

Porting the flux fix (existence assert + error:-polarity sweep) to argocd is
not a drop-in. Two additional failure modes showed up on a live KWOK run of
eks-training under argocd-git:

  1. Apply-time race / vacuous pass. A name-less sweep can evaluate at the same
    instant kubectl apply creates the root app-of-apps, when the only Application
    present is nvidia-stack with empty status. The gate "passed" in 0.26s
    with zero children synced. A staged fix (a named assert that the root
    reaches operationState.phase==Succeeded first, then the fleet sweep) closes
    this.

  2. operationState.phase==Succeeded hangs forever on health-gated apps. The
    KWOK argocd-* sync gate races ahead of child Application materialization (regression from #1050) #1061 predicate requires op==Succeeded. But on KWOK, kube-prometheus-stack
    (and prometheus-adapter) are permanently Synced + Progressing: their Argo
    sync operation stays op=Running with
    message: "waiting for healthy state of monitoring.coreos.com/Prometheus/kube-prometheus-prometheus".
    The prometheus-operator can't actually run under KWOK (no kubelet) to make the
    Prometheus CR healthy, so op never reaches Succeeded and a strict gate
    times out (observed: 10 min). Dropping op==Succeeded from the per-child sweep
    (keeping it only on the named root step) fixes this, since sync.status==Synced
    already implies a completed sync.

kubectl get applications -A 
NAMESPACE   NAME                            SYNC STATUS   HEALTH STATUS
argocd      aws-ebs-csi-driver              Synced        Healthy
argocd      aws-efa                         Synced        Healthy
argocd      cert-manager                    Synced        Healthy
argocd      gpu-operator                    Synced        Healthy
argocd      k8s-ephemeral-storage-metrics   Synced        Healthy
argocd      kai-scheduler                   Synced        Healthy
argocd      kube-prometheus-stack           Synced        Progressing
argocd      nfd                             Synced        Healthy
argocd      nodewright-operator             Synced        Healthy
argocd      nvidia-dra-driver-gpu           Synced        Healthy
argocd      nvidia-stack                    Synced        Healthy
argocd      nvsentinel                      Synced        Healthy
argocd      prometheus-adapter              Synced        Progressing
argocd      prometheus-operator-crds        Synced        Healthy

References

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions