Prerequisites
Bug Description
The KWOK GitOps sync gates (tests/chainsaw/kwok/flux-sync,
tests/chainsaw/kwok/argocd-sync) use name-less chainsaw assert blocks to
express "ALL resources of kind X must reach a terminal-pass state". A
name-less chainsaw assert actually has exists-semantics: it succeeds as
soon as any resource of that kind matches the predicate. The gates
therefore pass while most of the fleet is still converging — or never
converges at all.
Impact
Blocking (cannot proceed)
Component
CLI (aicr)
Regression?
Yes, this worked before (please specify version below)
Steps to Reproduce
Run any flux-oci KWOK cell and watch the HelmReleases while the sync gate
runs:
$ kubectl get helmrelease -n flux-system
NAME AGE READY STATUS
aws-ebs-csi-driver 9s True Helm install succeeded ...
aws-efa 9s False dependency 'flux-system/aws-ebs-csi-driver' is not ready
cert-manager 9s False dependency 'flux-system/aws-efa' is not ready
... (10 more False)
The assert-all-helmreleases-ready step passes the moment the FIRST release
hits Ready=True — the remaining 12 are still False in the dependsOn
chain. Verified on eks-training: the gate even passes when one of those
releases is permanently failing (InstallFailed — see the companion
nvidia-dra-driver-gpu issue, which this bug masked).
verify_pods provides no backstop for this class: a failed Helm install
creates no pods, and zero pods means zero Pending pods.
Expected Behavior
Use the existence + error-polarity pattern the ArtifactGenerator step already
demonstrates:
- Keep an existence
assert (resource kind + namespace, no status
predicate) so zero resources fails instead of passing vacuously.
- Replace the terminal-pass
assert with an error: op carrying the
De Morgan negation of the predicate — chainsaw's error polls until
NO resource matches the bad state, which is the actual "all resources
converged" semantics.
- Set
spec.timeouts.error alongside timeouts.assert and pass
--error-timeout from the wait_for_*_sync shims in
kwok/scripts/validate-scheduling.sh — chainsaw's error timeout is
independent of the assert timeout and defaults to 30s, far below what a
13-release dependsOn chain needs.
Acceptance criteria
Actual Behavior
| Test |
Step |
Effect |
tests/chainsaw/kwok/flux-sync/chainsaw-test.yaml |
assert-all-helmreleases-ready |
Passes on first Ready HelmRelease; InstallFailed releases invisible |
tests/chainsaw/kwok/argocd-sync/chainsaw-test.yaml |
assert-all-applications-pass |
A single Application matching the 4-arm predicate satisfies the sweep. The operationState.phase == 'Succeeded' gate from #1061 narrows the race (root passes only after a completed sync materializes children) but does not close it — children stuck OutOfSync/Missing cannot fail the gate while any sibling matches |
Note the ArtifactGenerator step in flux-sync is NOT affected — it already
uses inverted polarity (error: on the bad-state predicate), which is the
correct shape for an all-resources contract.
Environment
- AICR version (CLI
aicr version, API image tag, or commit SHA):
- Install method (release binary / build from source / container image):
- Platform (eks/gke/aks/oke/kind/lke/bcm/other):
- Kubernetes version:
- OS (ubuntu/cos/other) + version:
- Kernel version:
- GPU type (h100/h200/gb200/b200/a100/l40/rtx-pro-6000/other):
- Workload intent (training/inference):
Command / Request Used
No response
Logs / Error Output
Additional Context
Prerequisites
Bug Description
The KWOK GitOps sync gates (
tests/chainsaw/kwok/flux-sync,tests/chainsaw/kwok/argocd-sync) use name-less chainsawassertblocks toexpress "ALL resources of kind X must reach a terminal-pass state". A
name-less chainsaw
assertactually has exists-semantics: it succeeds assoon as any resource of that kind matches the predicate. The gates
therefore pass while most of the fleet is still converging — or never
converges at all.
Impact
Blocking (cannot proceed)
Component
CLI (aicr)
Regression?
Yes, this worked before (please specify version below)
Steps to Reproduce
Run any flux-oci KWOK cell and watch the HelmReleases while the sync gate
runs:
The
assert-all-helmreleases-readystep passes the moment the FIRST releasehits
Ready=True— the remaining 12 are stillFalsein thedependsOnchain. Verified on
eks-training: the gate even passes when one of thosereleases is permanently failing (
InstallFailed— see the companionnvidia-dra-driver-gpu issue, which this bug masked).
verify_podsprovides no backstop for this class: a failed Helm installcreates no pods, and zero pods means zero Pending pods.
Expected Behavior
Use the existence + error-polarity pattern the ArtifactGenerator step already
demonstrates:
assert(resource kind + namespace, no statuspredicate) so zero resources fails instead of passing vacuously.
assertwith anerror:op carrying theDe Morgan negation of the predicate — chainsaw's
errorpolls untilNO resource matches the bad state, which is the actual "all resources
converged" semantics.
spec.timeouts.erroralongsidetimeouts.assertand pass--error-timeoutfrom thewait_for_*_syncshims inkwok/scripts/validate-scheduling.sh— chainsaw's error timeout isindependent of the assert timeout and defaults to 30s, far below what a
13-release
dependsOnchain needs.Acceptance criteria
flux-syncandargocd-syncfail while any HelmRelease/Applicationis outside the terminal-pass set at the deadline.
--error-timeoutsoKWOK_*_SYNC_TIMEOUTgoverns theinverted steps too.
"all resources" sweeps.
gates are honest — budget a fidelity-iteration round per ADR-008.
Actual Behavior
tests/chainsaw/kwok/flux-sync/chainsaw-test.yamlassert-all-helmreleases-readytests/chainsaw/kwok/argocd-sync/chainsaw-test.yamlassert-all-applications-passoperationState.phase == 'Succeeded'gate from #1061 narrows the race (root passes only after a completed sync materializes children) but does not close it — children stuckOutOfSync/Missingcannot fail the gate while any sibling matchesNote the ArtifactGenerator step in
flux-syncis NOT affected — it alreadyuses inverted polarity (
error:on the bad-state predicate), which is thecorrect shape for an all-resources contract.
Environment
aicr version, API image tag, or commit SHA):Command / Request Used
No response
Logs / Error Output
Additional Context