Skip to content

[Bug]: ci(kwok): chainsaw sync gates pass when ANY resource converges, not ALL #1288

Description

@haarchri

Prerequisites

  • I searched existing issues and found no duplicates
  • I can reproduce this issue consistently
  • This is not a security vulnerability (use Security Advisories instead)

Bug Description

The KWOK GitOps sync gates (tests/chainsaw/kwok/flux-sync,
tests/chainsaw/kwok/argocd-sync) use name-less chainsaw assert blocks to
express "ALL resources of kind X must reach a terminal-pass state". A
name-less chainsaw assert actually has exists-semantics: it succeeds as
soon as any resource of that kind matches the predicate. The gates
therefore pass while most of the fleet is still converging — or never
converges at all.

Impact

Blocking (cannot proceed)

Component

CLI (aicr)

Regression?

Yes, this worked before (please specify version below)

Steps to Reproduce

Run any flux-oci KWOK cell and watch the HelmReleases while the sync gate
runs:

$ kubectl get helmrelease -n flux-system
NAME                            AGE   READY   STATUS
aws-ebs-csi-driver              9s    True    Helm install succeeded ...
aws-efa                         9s    False   dependency 'flux-system/aws-ebs-csi-driver' is not ready
cert-manager                    9s    False   dependency 'flux-system/aws-efa' is not ready
... (10 more False)

The assert-all-helmreleases-ready step passes the moment the FIRST release
hits Ready=True — the remaining 12 are still False in the dependsOn
chain. Verified on eks-training: the gate even passes when one of those
releases is permanently failing (InstallFailed — see the companion
nvidia-dra-driver-gpu issue, which this bug masked).

verify_pods provides no backstop for this class: a failed Helm install
creates no pods, and zero pods means zero Pending pods.

Expected Behavior

Use the existence + error-polarity pattern the ArtifactGenerator step already
demonstrates:

  1. Keep an existence assert (resource kind + namespace, no status
    predicate) so zero resources fails instead of passing vacuously.
  2. Replace the terminal-pass assert with an error: op carrying the
    De Morgan negation of the predicate — chainsaw's error polls until
    NO resource matches the bad state, which is the actual "all resources
    converged" semantics.
  3. Set spec.timeouts.error alongside timeouts.assert and pass
    --error-timeout from the wait_for_*_sync shims in
    kwok/scripts/validate-scheduling.sh — chainsaw's error timeout is
    independent of the assert timeout and defaults to 30s, far below what a
    13-release dependsOn chain needs.

Acceptance criteria

  • flux-sync and argocd-sync fail while any HelmRelease/Application
    is outside the terminal-pass set at the deadline.
  • Zero-resource case still fails (existence preserved).
  • Drivers pass --error-timeout so KWOK_*_SYNC_TIMEOUT governs the
    inverted steps too.
  • Audit remaining KWOK chainsaw tests for name-less asserts intended as
    "all resources" sweeps.
  • Expect previously-green GitOps lanes to surface real failures once the
    gates are honest — budget a fidelity-iteration round per ADR-008.

Actual Behavior

Test Step Effect
tests/chainsaw/kwok/flux-sync/chainsaw-test.yaml assert-all-helmreleases-ready Passes on first Ready HelmRelease; InstallFailed releases invisible
tests/chainsaw/kwok/argocd-sync/chainsaw-test.yaml assert-all-applications-pass A single Application matching the 4-arm predicate satisfies the sweep. The operationState.phase == 'Succeeded' gate from #1061 narrows the race (root passes only after a completed sync materializes children) but does not close it — children stuck OutOfSync/Missing cannot fail the gate while any sibling matches

Note the ArtifactGenerator step in flux-sync is NOT affected — it already
uses inverted polarity (error: on the bad-state predicate), which is the
correct shape for an all-resources contract.

Environment

  • AICR version (CLI aicr version, API image tag, or commit SHA):
  • Install method (release binary / build from source / container image):
  • Platform (eks/gke/aks/oke/kind/lke/bcm/other):
  • Kubernetes version:
  • OS (ubuntu/cos/other) + version:
  • Kernel version:
  • GPU type (h100/h200/gb200/b200/a100/l40/rtx-pro-6000/other):
  • Workload intent (training/inference):

Command / Request Used

No response

Logs / Error Output

Additional Context

Metadata

Metadata

Assignees

Type

Fields

No fields configured for Bug.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions