Skip to content

Commit 6901715

Browse files
myasnikovdaniillexfrei
authored andcommitted
test(e2e): fail install test when HelmReleases are not Ready
The HR readiness gate in e2e-install-cozystack.bats was toothless in two layers: the backgrounded kubectl-wait fan-out discarded child exit codes (POSIX `wait` without args returns 0), and the final check echoed "Some HelmReleases failed to reconcile" without exiting non-zero — a regression from 1f24038, which replaced the unavailable bats `fail` helper with a bare echo. A permanently-failing platform HelmRelease (e.g. the backup-controller CRD rejection, #2822) shipped through green CI for weeks. Replace the fan-out with a single `kubectl wait hr --all -A` for trace visibility, gate on an outcome-based re-list (which also covers HRs created after the snapshot, with a 120s retry to absorb momentary drift-reconcile flaps), and dump the full Ready condition message per non-ready HR so the real error is visible in the test output. Refs #2822 Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 2363f2d commit 6901715

1 file changed

Lines changed: 18 additions & 3 deletions

File tree

‎hack/e2e-install-cozystack.bats‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,12 @@ EOF
109109
# signal for "all platform HRs have been emitted" without hard-coding the
110110
# expected list, so the 5s pad lets a few late-arrivals join the snapshot.
111111
sleep 5
112-
kubectl get hr -A | awk 'NR>1 {print "kubectl wait --timeout=15m --for=condition=ready -n "$1" hr/"$2" &"} END {print "wait"}' | sh -ex
112+
# Pacing only (|| true): a backgrounded fan-out's exit codes are discarded
113+
# by a bare POSIX `wait`, which let permanently-failing HRs ship green
114+
# (#2822). A single kubectl wait names every HR that timed out in the
115+
# trace; the authoritative pass/fail gate is the outcome-based re-list
116+
# below, which also covers HRs created after this snapshot.
117+
kubectl wait hr --all -A --timeout=15m --for=condition=ready || true
113118

114119
echo "Waiting for post-install-prep to complete"
115120
if ! wait $POST_PREP_PID; then
@@ -119,10 +124,20 @@ EOF
119124
fi
120125
cat /tmp/post-install-prep.log
121126

122-
# Fail the test if any HelmRelease is not Ready
123-
if kubectl get hr -A | grep -v " True " | grep -v NAME; then
127+
# Fail the test if any HelmRelease is not Ready. Re-list rather than trust
128+
# the wait above so late-created HRs are gated too; the brief retry absorbs
129+
# momentary Unknown flaps from helm-controller drift reconciles.
130+
if ! timeout 120 sh -ec 'while kubectl get hr -A --no-headers | grep -v " True " | grep -q .; do sleep 5; done'; then
124131
kubectl get hr -A
132+
# kubectl's STATUS column truncates long messages; dump the full Ready
133+
# condition per non-ready HR so the real error (e.g. a rejected CRD) is
134+
# visible in the test output instead of only inside the cozyreport.
135+
kubectl get hr -A --no-headers | grep -v " True " | while read -r ns name rest; do
136+
echo "--- Non-ready HelmRelease: $ns/$name" >&2
137+
kubectl get hr -n "$ns" "$name" -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason}: {.message}{"\n"}{end}' >&2
138+
done
125139
echo "Some HelmReleases failed to reconcile" >&2
140+
exit 1
126141
fi
127142
}
128143

0 commit comments

Comments
 (0)